Sounds like you’re using a buggy Nostr client. Try a different or change which relays you’re using. Nobody but you can change your contact list. Maybe find the bug in the software you’re using and fix it. The vast majority of nostr apps are open source and welcome contributions.