Richi Jennings breaks it down: Peter Brockman, and open questions on C/R success rate determination methodology. As Richi puts it,
"Statistics aside, asking C/R users if they're happy isn't the be-all and end-all of anti-spam research. C/R users may indeed be happy -- happily unaware that their spam filter is sending spam by replying to innocent third parties who's addresses have been forged by spammers."
Spot on.
Justin Mason's take on it is accurate and insightful, as well:
"Now, here’s the first problem. The “Spam Index” therefore considers a false negative as about as important as a false positive. However, in real terms, if a user’s legit mail is lost by a spam filter, that’s a much bigger failure than letting some more spam through. When measuring filters, you have to consider false positives as much more serious! (In fact, when we test SpamAssassin, we consider FPs to be 50 times more costly than a false negative.)"
Justin hits the nail on the head. Part of the problem a number of anti-spam "researchers" have in common is discounting the damage done (or even inaccurately counting FPs) by doing things like relating the number of "hits" a blacklist or spam filter gets and assuming that the more hits you get, the better.
Then add in the, um, awesomeness of C/R, in that you're bouncing unwanted spam back to unrelated parties who were forged in from lines. C/R is a good way to block spam, by bouncing it off your bad filter and in to somebody else's inbox. That's like keeping criminals away from you by helping them break into your neighbor's home. Yuck.
Happy Friday from...the Baron!
I've apparently been dubbed "the Baron of Blacklists" for "waxing lyrically" on the subject of DNSBLs. If you're wondering what that's all about, Melinda Krueger published some information about blacklists in a recent Email Diva column. A long time subscriber myself, I thought it would be helpful to provide some more detail and clarification. So, I dropped her an email, which landed in a follow-up Diva column with my blessing. Neat!
Of course, to see what the Baron of Blacklists will be waxing lyrically about next, head on over to my other site, DNSBL Resource.
Of course, to see what the Baron of Blacklists will be waxing lyrically about next, head on over to my other site, DNSBL Resource.
Where was the consumer?
My friend Neil Schwartzman asked me a question during the FTC Spam Summit a couple of weeks ago. He asked me, “Where's the consumer?”
Neil, executive director of CAUCE (the Coalition for Unsolicited Commercial Email) in North America, had a point. The whole point of this exercise is figuring out how to answer the question, how do we protect the consumer? Problem is, there were a lot of consumer groups completely unrepresented at the event. It's great that they got Consumer Reports and Consumer Action to participate. In particular, Consumer Reports teased us with an upcoming review of spam filtering applications. Good stuff!
But, there was still a glaring omission: Where were the consumer groups actually focused on dealing with the spam problem? Where were the blacklists? How come CAUCE wasn't on a panel?
These are the groups actively fighting behind the scenes to preserve email. Working across countries, across boundaries, to solve the spam problem. The blacklists work hard to identify bad actors (often at significant personal legal liability), enabling receiving sites to more easily reject unwanted mail. Not everybody agrees with their methodology, and not everybody agrees with their goals. That's OK-- the same can be said of just about anybody else who was represented at the event. That doesn't mean they don't deserve a seat at the table.
That seat is important, for two simple reasons. One, so they can educate the rest of us of their point of view and all the valuable information they have. Two, so we can educate them. Put everybody in as room, get them to listen to each other, and something rubs off in both directions – usually for the better.
By not including CAUCE, or any of the blacklist groups like Spamhaus, SURBL, NJABL, PSBL, etc., in any of the panel discussions, we all lost out on that opportunity.
I'm very disappointed.
Neil, executive director of CAUCE (the Coalition for Unsolicited Commercial Email) in North America, had a point. The whole point of this exercise is figuring out how to answer the question, how do we protect the consumer? Problem is, there were a lot of consumer groups completely unrepresented at the event. It's great that they got Consumer Reports and Consumer Action to participate. In particular, Consumer Reports teased us with an upcoming review of spam filtering applications. Good stuff!
But, there was still a glaring omission: Where were the consumer groups actually focused on dealing with the spam problem? Where were the blacklists? How come CAUCE wasn't on a panel?
These are the groups actively fighting behind the scenes to preserve email. Working across countries, across boundaries, to solve the spam problem. The blacklists work hard to identify bad actors (often at significant personal legal liability), enabling receiving sites to more easily reject unwanted mail. Not everybody agrees with their methodology, and not everybody agrees with their goals. That's OK-- the same can be said of just about anybody else who was represented at the event. That doesn't mean they don't deserve a seat at the table.
That seat is important, for two simple reasons. One, so they can educate the rest of us of their point of view and all the valuable information they have. Two, so we can educate them. Put everybody in as room, get them to listen to each other, and something rubs off in both directions – usually for the better.
By not including CAUCE, or any of the blacklist groups like Spamhaus, SURBL, NJABL, PSBL, etc., in any of the panel discussions, we all lost out on that opportunity.
I'm very disappointed.
Blacklist notifications? Think again.
Infacta's "Messaging Times" posted a generally good article today on what you should be doing to minimize blacklistings. Except...
The article posits that "blacklist agents" should "contact senders that were reported prior to listing them with a plain-English explanation of what was reported and give them an opportunity to respond appropriately prior to being blacklisted. This process should be clear with instructions that are easy to follow."
Whoa. This is untenable on every possible level. Why?
Next, the article mentions "email authentication systems" referring to things like Goodmail and Sender Score Certified. These are actually email certification services, not authentication systems. You can choose to participate in a certification system, but it's not required on any level to get your mail delivered. Email authentication systems are actually things like SPF (Sender Policy Framework), Sender ID, DomainKeys, and DKIM. These all make it easier for receivers to identify senders and help their efforts to improve their ability to discern the good mail apart from the bad mail. They don't cost anything. SPF and Sender ID are things you set up in your DNS and can be done in about five minutes if you're technically inclined. DK/DKIM require support at the mail server sending side. Sometimes this is free, sometimes it might require an upgrade. This is like upgrading any piece of software, though, and it it's part of some conspiracy to make you pay to have to send email. (I think in the future you'll find just about every free or commercial mail server software will support DK or DKIM.)
And finally, the article asks the question, "Since when did the world "free" become a bad word?" The answer is: It didn't. It's not. The vast majority of spam content filters don't do anything so simplistic as to filter or block a message just because it contains the word "free." Don't be afraid to use the word "free." If you're not sending spam, it's not likely to get you blocked.
The article posits that "blacklist agents" should "contact senders that were reported prior to listing them with a plain-English explanation of what was reported and give them an opportunity to respond appropriately prior to being blacklisted. This process should be clear with instructions that are easy to follow."
Whoa. This is untenable on every possible level. Why?
- The vast majority of this spam is coming from forged addresses, overseas IPs, or infected machines (or all of the above). Notification to the listee is far from trivial and it will send bogus notifications to the wrong person 99% of the time. It is not worth it just to notify the 1% person who is actually reading his postmaster/abuse mailbox and speaks English.
- It just doesn't scale. Consider: My tiny random site receives, on average, ten thousand spams each day. Of the (approximately) 807,998 spams I've received since March 10, they came to me from 532,958 unique IP addresses. You expect me to send out over five hundred thousand notifications? Now explode that out exponentially to the real levels that blacklists deal with (which reveal my volumes to be puny).
- Smart senders check their bounces. The default configuration for blacklist usage includes a clear message with every bounce containing a link to a site or reference code with more information. This is notification. Do your due diligence and you'll notice a blacklisting within minutes or hours of it taking place. In most cases it is then easily and simply resolved.
- Smart senders periodically check blacklists to see if their IP addresses are listed. Any good email service provider (ESP) offer this service. Sites like DNS Stuff and Open RBL make it easy to check a bunch of lists at once.
- Good email actually doesn't get blacklisted very often. Sure, there are badly run blacklists out there (and I catalog both good and bad ones over on www.dnsbl.com), but most lists are not run by bad guys and are not out to attack people sending regular opt-in mail. If you are regularly ending up on lists like Spamhaus, NJABL, CBL, etc., then you're probably doing something wrong. If you're regularly getting blocked at Yahoo, Hotmail, or AOL, then you're probably doing something wrong. Fix your list. Stop trying to blur the lines of permission. Stop mailing to bounced email addresses repeatedly. Confirm new signups. Re opt-in your existing lists. Be proactive. It's not up to some external third party to tell you that you screwed up; if you let it go and got bitten by a blacklisting, you've usually got nobody to blame but yourself. The real problem is whatever caused the blacklisting, not the lack of a notification.
Next, the article mentions "email authentication systems" referring to things like Goodmail and Sender Score Certified. These are actually email certification services, not authentication systems. You can choose to participate in a certification system, but it's not required on any level to get your mail delivered. Email authentication systems are actually things like SPF (Sender Policy Framework), Sender ID, DomainKeys, and DKIM. These all make it easier for receivers to identify senders and help their efforts to improve their ability to discern the good mail apart from the bad mail. They don't cost anything. SPF and Sender ID are things you set up in your DNS and can be done in about five minutes if you're technically inclined. DK/DKIM require support at the mail server sending side. Sometimes this is free, sometimes it might require an upgrade. This is like upgrading any piece of software, though, and it it's part of some conspiracy to make you pay to have to send email. (I think in the future you'll find just about every free or commercial mail server software will support DK or DKIM.)
And finally, the article asks the question, "Since when did the world "free" become a bad word?" The answer is: It didn't. It's not. The vast majority of spam content filters don't do anything so simplistic as to filter or block a message just because it contains the word "free." Don't be afraid to use the word "free." If you're not sending spam, it's not likely to get you blocked.
Ask Al: My email address is being used in spam!
Gerald writes, "Help! I need to call the spam police and I don't know where to turn. My email address has been used to SEND spam. I know this only because an email sent under my name was undeliverable, and so the 'undeliverable' email report was sent to me. The subject line or sender's name was 'Free online secrets.' What can I do?"
Gerald, thanks for writing. Unfortunately, there's really not a ton you can do about this. There's no central spam police to report things to, nobody you who'll jump in and chase down those who forge your domain. Well, there is the FTC, but good luck getting this issue onto their radar – their resources are limited to the point that they really are only going after the biggest, baddest couple of bad guys at any given time. (And who's to say that yours is even in the US.)
But, if I were in your shoes, here's what I would be doing.
I realize it's very annoying, and I wish I had better answers for you. Thankfully, your online reputation isn't likely to be tarnished over this issue, especially not in the longer term.
Gerald, thanks for writing. Unfortunately, there's really not a ton you can do about this. There's no central spam police to report things to, nobody you who'll jump in and chase down those who forge your domain. Well, there is the FTC, but good luck getting this issue onto their radar – their resources are limited to the point that they really are only going after the biggest, baddest couple of bad guys at any given time. (And who's to say that yours is even in the US.)
But, if I were in your shoes, here's what I would be doing.
- Make sure there's really something significant going on here. Lots of spam has variable from lines. Some of it purposely tries to look like it's coming "from" you "to" you. It could just be that your copy had you on the from line. That alone wouldn't mean millions of other random joes got mail from you. One bounce back alone wouldn't be a concern. Getting dozens, hundreds, thousands? Then it would be safe to say that this is taking place on a wider scale. If not, I wouldn't bother with the rest of this (except authentication).
- Contact your ISP and let them know what's happening. Give them one example of the spam, and explain that you are being “joe jobbed” and that you're not responsible for the mail in any way. You don't condone it, you don't want it. I would do this pro-actively to ensure some over-zealous ISP doesn't take down your site after receiving spam complaints and making the false assumption that you must be up to something nefarious.
- After things have calmed down, look up your domain in the SURBL and URIBL "URI" blacklists. If you find that your domain is listed, contact them and ask to be removed, via the process they list on their sites. Like you did with the ISP, explain that you were the victim of a joe job, and that you don't send spam. They will likely remove you. If they don't, any mail you send to any site using SpamAssassin or other filters that check these lists will likely junk your mail if your domain or URL is mentioned in the body of messages.
- If you have the money to spare, you can hire lawyers and consultants to track the source of the forgery, figure out who to sue, and sue the offender. I'm happy to recommend someone who can help, but I would warn you that it's going to be expensive, and unlikely to be rewarding. My recommendation would be not to bother.
- For the long term: authenticate your mail. We're not quite there yet, but we're moving in the right direction. The big ISPs are just starting to pay attention to email authentication. For example, if you published the right kind of SPF or Sender ID record in DNS, Hotmail would automatically have discarded all of the forged spam attempts aimed at its user base. SPF and Sender ID records are a simple bit of text added to your domain name service record, and don't usually require any sort of additional infrastructure on your part. For more on SPF look here and here. (Regular readers may note that the authentication drum is something that I've been banging on for quite a while now.)
I realize it's very annoying, and I wish I had better answers for you. Thankfully, your online reputation isn't likely to be tarnished over this issue, especially not in the longer term.
Topics:
ask al,
authentication,
forgery
Blink: 32 new spams.
Hi from DC. I'm taking a break from the FTC Spam Summit 2007 to swap laptop batteries and check email.
Just as I got back to my hotel room, I got a page from a monitoring script I had set up. One of my spamtrap mailboxes was almost full and needed housecleaning. I logged in, and with the push of a few buttons, I emptied out the account (hey, there's always more spam) and turned the monitoring back on.
For the 30-60 seconds it took to empty out the mailbox's trash folder, I received 32 new spams. Click delete all, empty it, go back to the inbox and bulk folders, and I had 6+26 new messages. Man, I get a lot of spam.
A lot of discussion surrounding harvesting is taking place this time around. I am strongly anti-harvesting and it's clearly a bad practice. So, great. But harvesters are fairly easy to catch, and Project Honeypot seems to be spending significant effort going after them, so I wonder if this is something that really needs to be discussed in so much detail. Harvesting bad, check. What's next?
That's not to say that you shouldn't still protect your email addresses when putting them out on the web. On a whim, I had set up a special email account with a tagged address that I put only on one website back in May. After a couple days it started getting spam, and from May 26th through today, that address has received 189 spams. Man, what a pain.
But, as Suresh Ramasubramanian of Outblaze, and others have pointed out, keeping your address off the web doesn't prevent you from getting spam. You mail a friend, that friend's computer gets infected with malware, and that malware scoops all the email addresses it can find out of your friend's address book, and suddenly you're getting pharma spam served via botnets.
Just as I got back to my hotel room, I got a page from a monitoring script I had set up. One of my spamtrap mailboxes was almost full and needed housecleaning. I logged in, and with the push of a few buttons, I emptied out the account (hey, there's always more spam) and turned the monitoring back on.
For the 30-60 seconds it took to empty out the mailbox's trash folder, I received 32 new spams. Click delete all, empty it, go back to the inbox and bulk folders, and I had 6+26 new messages. Man, I get a lot of spam.
A lot of discussion surrounding harvesting is taking place this time around. I am strongly anti-harvesting and it's clearly a bad practice. So, great. But harvesters are fairly easy to catch, and Project Honeypot seems to be spending significant effort going after them, so I wonder if this is something that really needs to be discussed in so much detail. Harvesting bad, check. What's next?
That's not to say that you shouldn't still protect your email addresses when putting them out on the web. On a whim, I had set up a special email account with a tagged address that I put only on one website back in May. After a couple days it started getting spam, and from May 26th through today, that address has received 189 spams. Man, what a pain.
But, as Suresh Ramasubramanian of Outblaze, and others have pointed out, keeping your address off the web doesn't prevent you from getting spam. You mail a friend, that friend's computer gets infected with malware, and that malware scoops all the email addresses it can find out of your friend's address book, and suddenly you're getting pharma spam served via botnets.
Topics:
harvesting,
rant,
spam,
suresh ramasubramanian
Blogger listed on Spamhaus blacklist
It would seem that this SBL listing means that if you have a blog at http://(something).blogspot.com, your mail is going to be blocked by any site that checks the IP addresses of URLs found in messages, to see if those IP addresses are blacklisted.
Read more about it here.
I don't necessarily have an opinion on this at the moment. The devil's in the details, and I'm short on details. Generally speaking, I do want Spamhaus (and other blacklists) to bring the smack down on the bad guys. And if Google is (even unintentionally) being one of the bad guys by not doing enough to prevent spammers from using Blogger blogs as landing pages for spam, then that's a bad thing.
Read more about it here.
I don't necessarily have an opinion on this at the moment. The devil's in the details, and I'm short on details. Generally speaking, I do want Spamhaus (and other blacklists) to bring the smack down on the bad guys. And if Google is (even unintentionally) being one of the bad guys by not doing enough to prevent spammers from using Blogger blogs as landing pages for spam, then that's a bad thing.
TQMCUBE Blacklist Status
The TQMCUBE Blacklist seems to have been abandoned, and/or the creator and admins are missing in action. Over on DNSBL.com, I've collected all the information I have on the topic.