Perhaps # of follows should be confirmed with each posting of an updated follow list. 

 Or maybe clients with local storage should notify on big unexpected changes. 

 Not a bad verification step. 
I think Nostur has something like this. 

 Good catch...  I have been updating my relay lists a lot lately.  But There are 2 relays that I am always connected to, Damus and Primal.