I don't think it's my right to try to exert compulsory influence over my neighbors.

And I don't think "well they'll do it to you" is a valid moral argument.