Nostr has no central entity... that's the whole point. We are not playing with a numbers here. :) 
If you rearrange your relays and some got removed, it may happen that some of your followers will lost you.