In this way, the DM goes from my phone to his phone without anyone seeing it. But if people get access to his private key AND the DM event,  they can decrypt it.

And that is the same design of Signal and others: You have to have both the seed/key and the local database to decrypt.