I'm all for everyone being able to speak no matter how wrong they are, no matter how hateful they are, and no matter how evil I believe they are.  On the other hand, I have the right to not listen, read, or watch them and should have the right to turn them off for me.  They shouldn't be able to force me to listen any more than I should be able to silence them. 

 Right to free speech AND the right to free association

They can say whatever they want and we can walk away and talk to people we prefer speaking with. 

 Correction: "They [should] say whatever they want and we [should have the right to] walk away and talk to people we prefer speaking with."

Also what is your free speech based on?  Power?  God? 

 Fully agreed.  

 When someone is silenced, it not only affects the speaker, it also infringes on the rights of those who want to listen.