That's exactly the issue. The other side is (most likely) using gmail or similar and all email conversations get scanned. 

Practically nobody uses email e2ee. 

Just use signal/simplex for conversations.