Yep, unless you follow them, its going to block them again as quickly as the app sees the spam again. You can disable the spam filter on the security settings. You can also see which messages are deemed spam at the relay list screen, by clicking on one of the relays.

The app requires 5+ absolutely equal messages with over 50 chars (so GMs are good) to call an npub spammer. Usually that happens on public chats and some replies.