The First Amendment to the constitution says that "Congress shall pass no law abridging freedom of speech". This wording is important. It doesn't say that congress shall pass laws protecting our speech, but that congress shall not abridge it. "Free speech" is not a right given to us by government. Instead, "free speech" is a right we have, regardless of government.
The forces that want to restrict your speech include others than just government. For example, cartoonists around the world draw pictures of Jesus and Buddha, but do not draw pictures of Mohamed, because they are afraid of being murdered by Islamic fundamentalists. South Park depicted Jesus as addicted to Internet porn, and Buddha with a cocaine habit, but the censors forced them to cover up a totally innocuous picture of Mohamed. It's a free-speech issue, but not one that involves government.
In oppressive countries like Russia, the threats to speech rarely come directly from the government. It's not the police arresting people for speech. Instead, it's the local young thugs beating up journalists, with the police looking the other way.
In the United States, society has gone overboard on "political correctness" that silences speech. A good example in cybersec community is when the "Ada Initiative" got that talk canceled at "BSidesSF" last year, for obscure reasons that come down to the Ada Initiative hating that speech. That sort of thing is very much a "free speech" issue, even though the official government wasn't involved. Any time those in a position of power restrict speech, purely for its political content, this impinges on our values of "free speech".
The "responsible-disclosure" debate is about "free-speech", where some try to use the hammer of "ethical behavior" to control speech. Last night I tweeted a line of code from the OpenSSL source code that demonstrates a hilariously funny bug. I was attacked for my speech from OpenSSL defenders who want me to quietly submit bug patches rather than making OpenSSL look bad on Twitter.
That's why so many of us oppose the idea of "responsible-disclosure" -- the principle of "free-speech" means "free-disclosure". If you've found a vulnerability, keep it secret, sell it, notify the vendor, notify the press, do whatever the heck you want to do with it. Only "free-disclosure" advances security -- "responsible" disclosure that tries to control the process holds back security.
Such debates do circle back to government. For example, in the Andrew 'weev' Auernheimer case, the government cited Andrew's behavior (notifying the press) as an example of irresponsible behavior, because it didn't fit within the white-hat security norms of "responsible-disclosure". Andrew was sent to jail for speech that embarrassed the powerful -- and it was your anti-free-speech arguments of "responsible-disclosure" that helped put him there.
Certainly, it's technically inaccurate to cite "First Amendment" rights universally, as that's only a restriction on government. But the "free speech" is distinct: you can certainly cite your "right to free speech" in cases that have nothing to do with government.
Update: I shoulda just started this post by citing the Wikipedia entry on rights: "Rights are legal, social, or ethical principles of freedom". In other words, it's perfectly valid to use the word "right" in contexts other than "legal".
Update: I mention South Park because the XKCD mentions "when your show gets canceled". If your show gets canceled because nobody watches it, then that's certainly not a free-speech issue. But, when your show gets canceled because of threats from Islamists, then it certainly is a free-speech issue.