The main issue is that nip-04 leaks way too much data not only to the relay, but to the public in general. On top of that, the encryption procedures of NIP-04 are laughable. The lack of padding alone is a major problem. Imagine saying "Hi" on a DM and that gets converted to 4 encrypted chars. Besides letting everyone know your message is small, how hard is to break the encryption of a 4 char cipher text? Not that hard. 

If you repeat GMs around, now you have 100s or 1000s of 4 char encrypted messages. How hard would it be to recover your shared key knowing all those little messages? Not that hard. 

With enough shared keys, how hard would it be to figure out somebody's nsec? It gets in the realm  of "possible" with today's available computing power.