Oddbean new post about | logout
▲ ▼
laanwj | 6 days ago (raw) | root | parent | reply | flag 
 "is it a DoS or a CI runner" 😅 

often projects don't bother setting up caching and just re-download and check out the dependencies constantly

but yes blacklisting or just setting their rate limit to very very slow will send the right message, either way
▲ ▼
ChipTuner | 5 days ago (raw) | root | parent | reply | flag 
 If they weren't so clear about their crawling. I also just made repo urls public on my website Friday, so likely why crawlers picked it up. Its just been a steady stream of outgoing data XD

User Agent says "Amazbot" and "ClaudeBot/1.0; +claudebot@anthropic.com"

It's just this recursive path spam. I turned logging on briefly. Mind you there are no "icon" directories in these repos. 

Example: 
/cgit/vnuge/vnlib-core.git/tree/lib/Utils/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/icons/ubuntu-logo.png HTTP/1.1" 200 3121 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"