Unless your using a client that supports and uses NIP-44 for DMs then no one can see who you're messaging.
One more question. Using Primal exclusively now. However, it appears to keep purging all of my Follows. It's happened 4x. Is this a client issue or something that could be manipulated in the protocol by a node?