Oddbean

>Out of curiosity, what UAs are they using? Tried SSL fingerprinting? You know why they'd be hitting your server, like did you check if DiscordBot or something is in your referrers, or someone linked to it from somewhere, or...? Nitter is a Twitter proxy and there is only a few left after Elon's antics trying to make it a walled garden. The current design of Nitter requires us to make a large number of "guest accounts" that are created during an onboarding process using an old Android version of the Twitter account. These guest accounts give us access to most API features that used to exist because the walled garden. Each one has about 499 requests out of them before getting rate limited and only last 30.5 days before expiring. As to why? Nitter is effectively the only way to scrape content from Twitter, the guest_account stuff can only be created 1 per IP per day, so a lot have to be generated via proxy service. All of the basic stuff like obvious bot user agents have been handled, these botnets never have a single IP make a request more than once every 7-11 seconds and always with a legitimate User Agent. Sometimes it looks like desktop windows Chrome sessions, sometimes Iphones, its all over the place no real pattern, same with the stuff being searched for. I think I may come up with a way of 403ing anything that doesn't have a referrer to specific endpoints. In theory they should hit the root page, search from there and get referred to another page.