I smell an implied claim of unfeasibility ...

I'm fine with not getting events from people that are not followed by any of my follows. Any smartphone can store that list. If my follows reply to a note, yeah, load that, too etc.

I don't see CPU or RAM to be a limiting factor.