Freedom of Speech is not some magical thing: like all freedoms in politics, there’s got to be a justification for it. In the case of the US – on paper, at least – individual liberties are defined in terms of, “other than the things the state says you cannot do, you’re free.” So, because the state has not legislated that I cannot dye my hair blue, I can dye my hair blue. Freedom of speech is specifically called out, though, as a positive freedom. It’s not that “because the state has not told you what you can’t talk about, you can talk about anything else” – it’s specifically stated:
Congress shall make no law respecting an establishment of religion, or prohibiting the free exercise thereof; or abridging the freedom of speech, or of the press; or the right of the people peaceably to assemble, and to petition the Government for a redress of grievances.