Yes, the only way to see if the private zap is to you or for you is if the signer is able to decrypt it.
In amber there's a way to reject a event for ever.

https://github.com/nostr-protocol/nips/blob/master/55.md#L291

https://github.com/greenart7c3/Amber/blob/master/app%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgreenart7c3%2Fnostrsigner%2FSignerProvider.kt#L272