A working community notes system shouldn't depend on the number of followers you have.  One way this could work would be to have a community notes feed that you subscribe to.  A community note is provisionally added to an existing note, but is invisible in the feed unless you're viewing the community notes feed.  From there, when a nominal number of votes on the proposed note and percentage agreeing with the community note is met, it becomes visible in clients that support community notes.  Obviously once it becomes visible the user affected can direct their followers to vote against the note but they still *see* the note they're voting against and there's no guarantee that they will automatically follow that direction.

There are of course issues with bots that would need to be dealt with perhaps using account age or other factors to allow access to community notes and an ability to ban npubs that abuse the system (particularly if they bot creating notes that fill the feed).