Oddbean new post about | logout
▲ ▼
nixCraft 🐧 | 1 years ago (raw) | export | reply | flag +1 
 Add OpenAI, Google, and Common Crawl to your robots.txt to block generative AI from stealing content and profiting from it. See https://www.cyberciti.biz/web-developer/block-openai-bard-bing-ai-crawler-bots-using-robots-txt-file/ for more info and firewall to block those and other AI bots too.

https://files.mastodon.social/media_attachments/files/111/150/376/103/641/549/original/ddfa3856f9b4263d.png
▲ ▼
Roadskater, Ph.D. | 1 years ago (raw) | root | parent | reply | flag 
 @425a31b8 And if one has control of the server, look into installing a port-blocking tool such as fail2ban that kicks in if it spots unwanted activity. I've seen too many bots that ignore the robots.txt file
▲ ▼
YamakaziTaiga | 1 years ago (raw) | root | parent | reply | flag 
 @425a31b8 So 🤔 not adding your two cents?
▲ ▼
Alexis Alzate | 1 years ago (raw) | root | parent | reply | flag 
 @425a31b8 I'm not sure bots give a shit about robots.txt