A private relay would be required here to properly enable AUTH and trust that the relay is authing against a configured ACL for your group. In that case yeah don't see the reason for encryption since TLS between client/relay and group messages are private to that relay behind auth.  

 And then you can add/remove people at relay-level, which works quickly and smoothly, and doesn't require a fake-admin-npub as a crutch. 

 We just need relay admin UIs?? 

 We already have a good one, that is OS.
https://github.com/orgs/relaytools/repositories 

 That's what he's running here: https://relay.tools/