Yep... And this is one of the reasons people tend to use a large number of relays. The reply times are quite "random" from the set of relays they have. Adding more relays just makes things work out of luck.  

 I think I saw an attempt at trying to optimize filters in the NIP PRs, but I don't know how far it went.