Nobody has an excuse to be randomly violent toward anyone else. Yet violent people will still behave that way. It’s beside the point. Freedom of association is a good thing. Choosing to exclude trans people is an asshole way of exercising your freedom of association. This isn’t a question of first principles. Freedom is good. Using freedom to be a bigot is bad. It’s not nuanced.
(I disagree with your point but I appreciate you replying and engaging in good faith. I meant to start with that).
I was thinking of that women's boxing match at the Olympics. They let a man fight, and he just absolutely demolished the women. The silver medalist quit after fighting him for 40 seconds. She fell to her knees and yelled "this is unjust", and forfeited after being punched twice. She's an Olympian boxer and she said she had never been hit that hard in her life. Why was this allowed to happen? Because we keep listening to the wrong people who claim it's virtuous to let this happen. It's not.