The unwritten rules of respecting robots.txt and not scooping up everything have already crumpled due to AI scraping.

https://www.businessinsider.com/meta-web-crawler-bots-robots-txt-ai-2024-8

Seems to be the trend that if you can access it it will be used. Encrypt and deny access might become the new reality, unfortunately.