If I understand this right, the relay would be able to see the public IP of every client and correlate that to the group IDs that they are downloading?  In the worst case of a note being published to a single malicious relay, that relay would know the public IP of every client that downloaded the note and know that they where a member of the group (or at least have high confidence of it)? 

 Yes, but VPNs or trusted relay proxies would mitigate this 

 What @hodlbod said. There are a few other tricks you could also use to obfuscate the size of the group though.