If I understand this right, the relay would be able to see the public IP of every client and correlate that to the group IDs that they are downloading?  In the worst case of a note being published to a single malicious relay, that relay would know the public IP of every client that downloaded the note and know that they where a member of the group (or at least have high confidence of it)?