The hardest but most necessary pill to swallow if you're a libertarian is that a vast majority of the people do not have any desire of being free -- because freedom cannot be detached from responsibility.
In my opinion, libertarians waste too much time theorizing why liberty is good, hoping to convince the people who will never be convinced. Instead, libertarians should worry a lot more about how to effectively protect themselves and their freedom from the self-subjugated and those who rule them.