I guess the app you are using does not cache your follow list locally and solely relies on the data you receive from relays. In this case changing relays affects your follow list. You might be connected to relays that don't have your current follow list and serve you an outdated one.