For a lot of different reasons, gossip's code is not structured in a way that my (or your) mental model would suggest. Partly these are for performance or ease of coding (earlier), and partly because it developed along a certain path which created a lot of technical debt. Refactors are in order.

I've tried to fix this issue several times under the title of "load more" and each time I've gotten so tangled up and confused that I backed out and attempted to attack the problem from another angle (usually being able to apply some small set of safe steps first).  90% of what I need to do is understood by me, but it's hard to find a way to do it which doesn't become a mess as I start tripping over the 10% that I haven't completely realized from the outset was going to trip me up.

I'm well down a path of fixing this now which has a good chance of actually succeeding.  The general plan is like this:
1) Keep an 'anchor' time for the feed (just one, it resets when you switch feeds).
2) The first subscription to the feed you switched to starts a long-term subscription that loads events from the anchor forward, and continues to listen after the EOSE.
3) Start a short term subscription that loads until the anchor, limit some number of notes (call it N).
4) When you scroll to the bottom and press 'load more', then anchor shifts to the created_at of the oldest feed event, and then this subscription is rewritten to load the next N events (the long-term subscription doesn't care about the anchor shift as it doesn't change).
5) Whenever you shift to a different feed (inbox, a different list, My notes, etc) everything is reset and it starts over.  This means is goes back to the relays again and often loads things from them it already has.
6) I will then create some kind of caching (not of events which we already cache, but of knowledge about which time ranges have been loaded with which filters) so that it doesn't reload the same events from the relay over and over.

Once this is done, it should be easier to modify from that point than from the current point, so if I discover this is hard to do I can work backwards somewhat. 

 I get most of it. What you describe is closer to my mental model. 

 Ok I fixed gossip on the 'load_more' branch.  It still needs testing and some more changes before it gets merged in, but you can try it now if you'd like (no schema changes, safe to switch back and forth).