Why not just not charge people for a trivial amount of crawler traffic? If the backend is decently fast, it should basically be free to eat the load and not charge customers for it (and yes, I know the 0.5 QPS was for one site and it will add up; even so). My CDN doesn't charge me for getting DDoS'd because that traffic isn't really "my fault" and that could easily be orders of magnitude more traffic than all crawler traffic for all crawlers across every discourse instance.