robots.txt configuration
https://developer.mozilla.org/en-US/docs/Web/Security/Practical_implementation_guides/Robots_txt