Your nsec wouldn’t be compromised. The only risk is to the corresponding ActivityPub ID on the other end.