There is plenty of overhead to establishing a connection, making a REQ, then disconnecting. Even if the REQs are made strategically (and they certainly aren’t today on most clients) this still introduces a ton of additional latency. Small personal/community relays will also tend to be slower and unreliable, further increasing latency and performance issues. 

If as you say majority of users stay connected with a small number of larger relays (tends to be my belief), doesn’t the gossip/outbox just become a waste of resources unlikely to yield many new notes with terrible tradeoffs?

I could be wrong of course, but to me the long term outlook of public (anyone can read) nostr relays seem more like medium-large hubs not small personal/community data stores. We still get the advantage of having many in different locations with different policies run by different operators with different risk tolerances. Private relays (with NIP-42 auth) are different and can serve a completely different purpose.