The right to free speech in the United States is restricted by court cases, specifically if words will cause imminent lawless action. The phrase often attributed to this is "You can't shout fire in a crowded theatre," and though it's not the most perfect example to use, it suffices.
But I know that we ain't talking about legal rights, but rather moral rights and the personal value one holds of free speech.
Limiting speech is an unfortunate and difficult reality. One cannot slander, libel, or essentially cause damage to someone through false remarks. I think most people can see how this acts as a benefit to society. But then how do you handle someone saying, "Fuck it, don't limit speech!" because that notion allows people to run rampant with lies. I suppose one could say that a free society will course correct itself, but I don't find this to be true.
So, with this in mind, I think it's clear that there must exist some limit on free speech. And these limits should include hate speech. People are entitled to opinions, racist ones as well as accepting ones, but when these opinions become words, and specifically words which endanger someone's security, then they must be stopped. Having a neo-nazi conference, or a nazi protest where participants are yelling slurs wouldn't directly endanger a counterprotestor's security because it's safe to assume that there would be police in the area. But if it was at a bar and some dude was calling a Jew a kyke etc, and enticing people to attack him based on his Jewishness, then that should be limited.
But that's my opinion.