Live AWS keys in 75 throwaway repos, each made public for one of five windows from 60 seconds to 12 hours, every use logged. The keys were tripwires; the real question was who notices a private repo going public, and what they do once they’re in.
The most useful finding is the dull one: re-hiding the repo does nothing. One busy harvester kept re-validating the captured keys for a day after the repos went private again. Only rotating the key stops it.
This came out of building a monitor for exactly these repo-setting changes.
There’s hardly any cost to a bot operator, malicious , opportunistic or legitimate, to hit your end-point, so once they found a reason to hit it, hitting it a million more times costs cents.
Operators like Meta seem to make it a sport, trying to hit you with multiple parallel requests from multiple sources, across both IPv4 and IPv6 simultaneously, resulting in an effective DDoS for small and medium end point owners and increasing costs significantly for anyone trying fruitlessly to stay ahead of their onslaught.
The malicious traffic by contrast, attempts to sneak in a request with dynamic rate throttling as part of their attempts to stay hidden.
Between these two extremes are the opportunistic operators who hit the same 404 endpoint day after day, hour after hour, minute by minute, for weeks with specific blocks the only remedy.
There are plenty of legitimate bots that quietly go about their business, hitting you every couple of seconds, leaving you alone for long stretches, incrementally crawling, honouring the robots.txt file and generally acting the way a considerate adult might. They’ve been getting lower and lower in numbers over the years.
Source: I have logs.
I can’t imagine what the solution is to this madness.