Guess:
Decryption tricked me once in ndk because sender and receiver pubkey namings can be confusing.

You always need to decrypt with providing the recipient pubkey(p-tag) and the PEER pubkey. 
This "peer pubkey" could be your user (your user sent a message) OR another user (p-tag refers to YOUR user as recipient and someone sent your user a message).

I had to check which is which to get it right.