Every once in a while somebody drops into comments espousing a conspiracy theory around how blocklists are “designed to extract money from email senders” or some other crazy thing about how dare blocklists not block mail from Gmail given that they’re receiving unsolicited emails from Gmail subscribers.
The folks saying this, I don’t really think they understand how it all works. So allow me to explain.
Blocklists Were Created To Stop Spam
Way back when I got my start in email in the late 1990s, setting up my employer’s first mail server and connecting it to the internet, it was truly a wild west for spam, and not in a good way. Mailboxes were being filled with unsolicited garbage, and a number of spammers were just absolutely pathologically near-evil, if not perhaps actually evil, sending millions of unsolicited messages, with no care or limits. And a lot of it was just literally disgusting. Think of any gross thing involving gore or porn and yeah, some jerk was sending pictures of it, and advertising for paid access to more of it, via spam.
If you weren’t around in email then, you really can’t know how email was just getting overrun with this garbage. It was bad.
We needed a way to make this stuff stop. To reduce it or eliminate it so that we could actually use email for what it was made for, to communicate. To send and receive wanted email messages.
To do that, people started to list and block mail from servers that were shoveling spam. Either because they were the source of the spam, or because they were relaying it (wittingly or unwittingly) for the spammers. The blocking was done by IP address. Thus was born the first foray into IP reputation, keeping a list of servers that were transmitting spam, by IP address, to prevent them from delivering mail until the spam stopped.
The distribution of these lists of spam spewing servers was handled by way of blocklists, technically called DNSBLs (DNS-based BlockLists or Blacklists). The first one was MAPS (Mail Abuse Prevention System) RBL (Realtime Blackhole List), which is why some people call blocklists RBLs. But there were many more after this first one (and I’ve run at least a couple of them along the way myself). There came to be so many that I started a website, now called Blocklist Resource, where I tracked many of them and used to share data on their efficacy.
Blocklists were important and necessary. Email as a usable communication medium would have died without them.
Many Blocklists Have Shutdown
And that’s why my Blocklist Resource website has now become a sort of blocklist graveyard, mostly denoting when any given blocklist has shut down. Most recently: NixSPAM and SORBS, two blocklists that had been around for many years. I document the shutdown so that it helps to remind people to remove dead blocklists from their mail server spam filtering (because leaving a dead blocklist in place can often result in unexpected and unwanted blocking of all inbound mail). And so that any internet users who run into a friend, company or mailbox provider still using a dead blocklist can share a link to my site showing some “proof” that a blocklist truly has shut down.
Why did those blocklists shut down? Reasons vary; sometimes it's due to legal threats or concern over liability, but more often, the publisher simply decided to move on, that it wasn't worth the effort to continue. Blocklists were very, very important and necessary back in the day, but in this modern age, most internet service providers and mailbox providers have moved beyond simple IP reputation-related rejections for their spam filtering.
IP Reputation Has Limits
This is both why so many blocklists have shut down, and why the remaining blocklists don’t just “block Gmail” because Gmail (like any other large mailbox provider) emits some non-zero amount of unwanted email.
Blocking based on the IP address of a mail server is a very rough and often inexact target. A single mail server with a single IP address can serve millions of email messages daily for many thousands or even millions of users. “Shared IP address” is a deliverability and email technology term that describes this – it indicates that the server being referred to as having a “shared IP address” handles mail (usually outbound mail) for multiple users or customers. This applies to both email service providers (ESPs), the marketing automation platforms that serve bulk mail, and mailbox providers like Gmail. Most email infrastructure is “shared” because servers and IP addresses are expensive.
Thus, IP reputation has limits. If you’re trying to block (only) spam, you usually care enough to not cause the blocking of legitimate mail. It undermines your spam stopping goal, and it makes people mad. If a blocklisting issue results in all email from any user of Gmail being rejected somewhere important, a whole bunch of those Gmail users are going to be REALLY angry and come screaming for the blocklist operator’s head. It’s happened before.
A few blocklist operators don’t understand this or don’t care, but most do. And that’s why blocklists don’t just block all of Gmail’s sending IP addresses. It’s not a conspiracy, it’s good sense.
Smart spam fitlerers, and the mailbox providers they often work for, have realized that just blocking based on IP address isn’t good enough. That’s a big part of why “domain reputation” began to become a whole thing. Gmail’s inbound filtering is actually quite focused on domain reputation, and is pretty good at figuring out when bad guys, using shared IP addresses, are sending unwanted spam. They’re so good at it that they often block JUST the bad guys, letting through the good mail from the same server.
IP reputation still matters, especially for the worst of the worst – computers infected with malware and controlled to be part of a botnet, for example – but domain reputation plays a big and important part of spam filtering nowadays.
Some IP-based blocklist operators get this and have pivoted to either restricting what they list or evolving into a more nuanced and sophisticated purveyor of reputation data, like Spamhaus.
Yes, Some Blocklists are Dumb
Or are run by jerks. I’ve seen blocklist operators who listed a whole email service provider because they had a personal beef with an employee, and another who posts weird misogynistic rants about women. Some use circular logic, failing to track or save data to defend actions or listings, unable to defend against challenges, interviewing very poorly when investigated. Blocklist operators have been sanctioned, called into court, or worse: One supposedly fled the country they were living in to avoid blocklist-related legal repercussions.
Blocklist operators are people, and some people are idiots.
And anybody can publish a blocklist. It’s like a blog; just because you publish it, doesn’t mean that anybody reads it. That is VERY IMPORTANT to keep in mind for blocklists. Translate it thusly: Just because your IP address is listed on Blocklist X, does not mean that you will have deliverability problems.
People get this bit wrong all the time. Tier 1 support reps for too many companies respond to tickets “but you’re on the Purple Turtle blocklist, that must be what’s going wrong!” when the Purple Turtle blocklist is NOT being used by Microsoft, Google, Yahoo, or anybody else notable or important to filter mail. There is just no connection.
SOMETIMES you can draw a correlation, in that because most blocklistings are driven by sending unsolicited mail to spamtrap addresses, that a blocklisting is a “canary in a coal mine” indicator that a marketing sender has a list hygiene problem. But it is NOT universal.
Maybe a dozen blocklists out there – probably fewer than that – are broadly used. If you find that you have your sending IP address listed on a blocklist, be sure to correlate it to actual deliverability data to look for real issues (only). Are there bounces referencing the blocklist issue? If not, there’s a good chance that you have no issue and that you’d be wasting time to continue looking into it.
And of course, not all blocklist operators are bad. They’re people, and some people are good and wise. So when a well-known reputation provider blocklist tells you that you have an issue with list hygiene and permission, you’d be wise to take the warning seriously and investigate, instead of firing up the megaphone to start yelling loudly about how they’re part of some conspiracy.
Every once in a while somebody drops into comments espousing a conspiracy theory around how blocklists are “designed to extract money from email senders” or some other crazy thing about how dare blocklists not block mail from Gmail given that they’re receiving unsolicited emails from Gmail subscribers.
The folks saying this, I don’t really think they understand how it all works. So allow me to explain.
Blocklists Were Created To Stop Spam
Way back when I got my start in email in the late 1990s, setting up my employer’s first mail server and connecting it to the internet, it was truly a wild west for spam, and not in a good way. Mailboxes were being filled with unsolicited garbage, and a number of spammers were just absolutely pathologically near-evil, if not perhaps actually evil, sending millions of unsolicited messages, with no care or limits. And a lot of it was just literally disgusting. Think of any gross thing involving gore or porn and yeah, some jerk was sending pictures of it, and advertising for paid access to more of it, via spam.If you weren’t around in email then, you really can’t know how email was just getting overrun with this garbage. It was bad.
We needed a way to make this stuff stop. To reduce it or eliminate it so that we could actually use email for what it was made for, to communicate. To send and receive wanted email messages.
To do that, people started to list and block mail from servers that were shoveling spam. Either because they were the source of the spam, or because they were relaying it (wittingly or unwittingly) for the spammers. The blocking was done by IP address. Thus was born the first foray into IP reputation, keeping a list of servers that were transmitting spam, by IP address, to prevent them from delivering mail until the spam stopped.
The distribution of these lists of spam spewing servers was handled by way of blocklists, technically called DNSBLs (DNS-based BlockLists or Blacklists). The first one was MAPS (Mail Abuse Prevention System) RBL (Realtime Blackhole List), which is why some people call blocklists RBLs. But there were many more after this first one (and I’ve run at least a couple of them along the way myself). There came to be so many that I started a website, now called Blocklist Resource, where I tracked many of them and used to share data on their efficacy.
Blocklists were important and necessary. Email as a usable communication medium would have died without them.
Many Blocklists Have Shutdown
And that’s why my Blocklist Resource website has now become a sort of blocklist graveyard, mostly denoting when any given blocklist has shut down. Most recently: NixSPAM and SORBS, two blocklists that had been around for many years. I document the shutdown so that it helps to remind people to remove dead blocklists from their mail server spam filtering (because leaving a dead blocklist in place can often result in unexpected and unwanted blocking of all inbound mail). And so that any internet users who run into a friend, company or mailbox provider still using a dead blocklist can share a link to my site showing some “proof” that a blocklist truly has shut down.I’ve tracked the shutdown of more than 30 blocklist and blocklist organizations.
IP Reputation Has Limits
This is both why so many blocklists have shut down, and why the remaining blocklists don’t just “block Gmail” because Gmail (like any other large mailbox provider) emits some non-zero amount of unwanted email.Blocking based on the IP address of a mail server is a very rough and often inexact target. A single mail server with a single IP address can serve millions of email messages daily for many thousands or even millions of users. “Shared IP address” is a deliverability and email technology term that describes this – it indicates that the server being referred to as having a “shared IP address” handles mail (usually outbound mail) for multiple users or customers. This applies to both email service providers (ESPs), the marketing automation platforms that serve bulk mail, and mailbox providers like Gmail. Most email infrastructure is “shared” because servers and IP addresses are expensive.
Thus, IP reputation has limits. If you’re trying to block (only) spam, you usually care enough to not cause the blocking of legitimate mail. It undermines your spam stopping goal, and it makes people mad. If a blocklisting issue results in all email from any user of Gmail being rejected somewhere important, a whole bunch of those Gmail users are going to be REALLY angry and come screaming for the blocklist operator’s head. It’s happened before.
A few blocklist operators don’t understand this or don’t care, but most do. And that’s why blocklists don’t just block all of Gmail’s sending IP addresses. It’s not a conspiracy, it’s good sense.
Smart spam fitlerers, and the mailbox providers they often work for, have realized that just blocking based on IP address isn’t good enough. That’s a big part of why “domain reputation” began to become a whole thing. Gmail’s inbound filtering is actually quite focused on domain reputation, and is pretty good at figuring out when bad guys, using shared IP addresses, are sending unwanted spam. They’re so good at it that they often block JUST the bad guys, letting through the good mail from the same server.
IP reputation still matters, especially for the worst of the worst – computers infected with malware and controlled to be part of a botnet, for example – but domain reputation plays a big and important part of spam filtering nowadays.
Some IP-based blocklist operators get this and have pivoted to either restricting what they list or evolving into a more nuanced and sophisticated purveyor of reputation data, like Spamhaus.
Yes, Some Blocklists are Dumb
Or are run by jerks. I’ve seen blocklist operators who listed a whole email service provider because they had a personal beef with an employee, and another who posts weird misogynistic rants about women. Some use circular logic, failing to track or save data to defend actions or listings, unable to defend against challenges, interviewing very poorly when investigated. Blocklist operators have been sanctioned, called into court, or worse: One supposedly fled the country they were living in to avoid blocklist-related legal repercussions.Blocklist operators are people, and some people are idiots.
And anybody can publish a blocklist. It’s like a blog; just because you publish it, doesn’t mean that anybody reads it. That is VERY IMPORTANT to keep in mind for blocklists. Translate it thusly: Just because your IP address is listed on Blocklist X, does not mean that you will have deliverability problems.
People get this bit wrong all the time. Tier 1 support reps for too many companies respond to tickets “but you’re on the Purple Turtle blocklist, that must be what’s going wrong!” when the Purple Turtle blocklist is NOT being used by Microsoft, Google, Yahoo, or anybody else notable or important to filter mail. There is just no connection.
SOMETIMES you can draw a correlation, in that because most blocklistings are driven by sending unsolicited mail to spamtrap addresses, that a blocklisting is a “canary in a coal mine” indicator that a marketing sender has a list hygiene problem. But it is NOT universal.
Maybe a dozen blocklists out there – probably fewer than that – are broadly used. If you find that you have your sending IP address listed on a blocklist, be sure to correlate it to actual deliverability data to look for real issues (only). Are there bounces referencing the blocklist issue? If not, there’s a good chance that you have no issue and that you’d be wasting time to continue looking into it.
And of course, not all blocklist operators are bad. They’re people, and some people are good and wise. So when a well-known reputation provider blocklist tells you that you have an issue with list hygiene and permission, you’d be wise to take the warning seriously and investigate, instead of firing up the megaphone to start yelling loudly about how they’re part of some conspiracy.
Comments
Post a Comment
Comments policy: Al is always right. Kidding, mostly. Be polite, please and thank you.