No, gift wrap DMs get sent to both the sender and the receiver, so you don't need to keep track of all the ephemeral keys