Is it OK to block political speech?

I've been talking to folks a lot these past couple of days about the potential legalities around blocking unwanted spam from non-profit, political or advocacy senders. From what I understand, this is pretty likely to be legal. The first amendment limits government action as it relates to restricting speech. But the first amendment doesn't apply to private parties; there is no constitutional "right" that private party number one must accept a message from private party number two. That seems cut and dry. CAN-SPAM certainly doesn't touch on it; it doesn't say a spam filterer can't block certain kinds of messages.