It's a valid point, people should be good and use their freedom for good.
The divide comes from a variance of ideas about social and moral virtues and harms. You accuse them of being harmful, they accuse you of the same.
What I don't get is why do we all have to live together? I think any group should be allowed to collectivize. If we were smart we would help any self identifying groups to relocate or secede so they can all be together and govern themselves and leave us alone.
I think voluntary separation is the obvious solution to all world conflict. It's just that abusive narcissists won't let it happen.