Oddbean new post about | logout
 robots.txt 1994–2023:

User-agent: *
Disallow:

robots.txt 2023–?:

User-agent: CCBot
User-agent: ChatGPT-User
User-agent: FacebookBot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: Omgilibot
Disallow: /

#web #ai #robotstxt