Misbehaving scrappers overwhelming servers is just a demonstration of yet another fatal flaw in the server-centric model of networked communications.
The solution isn't to make it worse for humans (and irrelevantly slightly more expensive for commercially abusive scrappers), the solution is to move away from a model that clearly had issues even before rampant abuse became a thing.
And another note on rampant abuse and scrappers, scrapping does *not* need to be abusive. It is not synonymous with abusive behavior nor does it imply it. Providing a more reasonable way to get tarball'd copies of static sites or weekly db dumps (both of which could be torrented, like Wikipedia's) would prevent load even from those that are being considerate about it.
Much of the uses I have for scrapping are effectively working around bad administrator decisions, fuckups (both theirs and those of others such as infrastructure operators/providers') and protocol/design limitations (always-online with low latency assumptions are not good assumptions).
The solution isn't to make it worse for humans (and irrelevantly slightly more expensive for commercially abusive scrappers), the solution is to move away from a model that clearly had issues even before rampant abuse became a thing.
And another note on rampant abuse and scrappers, scrapping does *not* need to be abusive. It is not synonymous with abusive behavior nor does it imply it. Providing a more reasonable way to get tarball'd copies of static sites or weekly db dumps (both of which could be torrented, like Wikipedia's) would prevent load even from those that are being considerate about it.
Much of the uses I have for scrapping are effectively working around bad administrator decisions, fuckups (both theirs and those of others such as infrastructure operators/providers') and protocol/design limitations (always-online with low latency assumptions are not good assumptions).
1
