There is plenty of overhead to establishing a connection, making a REQ, then disconnecting. Even if the REQs are made strategically (and they certainly aren’t today on most clients) this still introduces a ton of additional latency. Small personal/community relays will also tend to be slower and unreliable, further increasing latency and performance issues. 

If as you say majority of users stay connected with a small number of larger relays (tends to be my belief), doesn’t the gossip/outbox just become a waste of resources unlikely to yield many new notes with terrible tradeoffs?

I could be wrong of course, but to me the long term outlook of public (anyone can read) nostr relays seem more like medium-large hubs not small personal/community data stores. We still get the advantage of having many in different locations with different policies run by different operators with different risk tolerances. Private relays (with NIP-42 auth) are different and can serve a completely different purpose. 

 We already have better decentralization than the competition and we are lacking in performance, usability, features, etc. This is making our biggest issues worse to improve the benefit that is already unrivaled. 

 Yeah, I agree, I think authenticated relays are going to be where interesting new developments happen. Maybe clients should differentiate more strongly between public hubs/aggregators and private community/inbox relays.