What client are you using?

Local caching of your feed + outbox model should solve this and also be flexible enough to allow you to exchange censorship-resistance and speed for network usage.