One must be prepared for the consequences of displaying flags or symbols or speech while in society.  Carrying a nazi flag is also perfectly within your rights, but low self-control, low IQ people could definitely do you harm.  Sometimes it’s better to be the grey man.  Being in the right won’t matter if you are injured or dead.