Oddbean

@Salastil @fzorb @anime graf mays 🛰️🪐 > they're scraping half of twitter via my instance, Ha, it sounds a lot like what Boardreader was doing to FSE. They actually recorded browser sessions and played them back, big army of residential US proxies. I actually ended up writing a script that watched the logs and waited until some client had a suspiciously high proportion of requests hitting TWKN (watching behavior instead of source) and they would fire off a few hundred requests and then hop IPs. If I killed an IP, another one would arrive really quickly. Since they'd recorded browser sessions, it was hard to tell until they had already gotten some of the data already, but by the time they had hit several hundred requests for TWKN after the initial burst, it was too late to detect them. Maybe check if you see `devtools.boardreader.com` in your logs anywhere, ha. They weren't executing JavaScript (they couldn't) but I didn't wanna break all the clients by doing something like that. Nitter, on the other hand, is basically *just* a web UI, so you could go that route. Tack on some JS that adds a hash of the IP address plus a nonce to every link, this precludes a lot of proxy use and non-JS-executing scrapers because they'd have to know which place they're exiting from then do a hash. > One of the approaches brought up was to ban anything that wasn't containing a referrer from the site. That works sometimes, but they will pretty often spoof it or start spoofing it.