You are not sharing the nsec. You will be sharing just the sum (nsec+npub) of each conversation. In NIP-44 its impossible to calculate the original nsec just with the conversation key. 

Clients can then encrypt and decrypt messages but they can never sign for them. That's the role of the bunker.