I dunno what the web crawlers use. I made a bash script to scan logs and block them every 30 min