I had a similar problem when I was starting this keypair, but I think what was happening was, the old relay list kept overwriting the new one. I assume relays store that information somehow, so maybe there was a bad one that was screwing things up.

I didn't really do anything to stop it, it just gradually stopped breaking over time, so maybe the same will happen for you.