I’m trying to get to a reason on this, but my point reach to a limit.
I’ve the feels that scraping the internet for public accessible data, like for example open and public music on Spotify wouldn’t be a crime, but the distribution would be. At the same token, this is seem as a crime, while Google does the same and nothing happens, even worse, if this get regulated, Google would have a huge advantage on anyone else.
So, my deeper question is: “Is copyright dead?”
Legally, check your local laws or just be sure to cover your ass with tor or a VPN with an anonymous core.
Ethically, just obey Wheaton’s Law: “Don’t be a dick.”
With web scraping, I can think of two ways Wheaton’s law applies:
Scrapers should blend into existing background web traffic. They should be slow enough to not overwhelm their servers. This requires babysitting any new scraper until one is sure it is tuned to be safe for the scraped site.
Any scraped content shouldn’t be re-hosted in a way that harms the original content creators. Sharing is lovely. Harming artists sucks. Finding the right balance between preservation and respect can take some thought, but it’s usually actually a pretty wide road.