My wife and I were lucky enough to be able to purchase a new car earlier this year. It's a nice car and we love it. But ever since then, seemingly once a month or so, I get a survey request related to automobiles and the automotive industry. Some from known entities, some from unknown entities. A number of them are coming from third parties that I didn't specifically hand my email address to.
The anti-spammer in me tells me that these emails are spam. Somebody I don't recognize is sending me list mail or bulk mail, to an address that I did not give to them.
But when talking with clients or potential clients I have had a lot of them try to tell me that this kind of mail is expected and that it's not spam.
For the moment, forget about who's right or wrong here.
Consider this: Just about all of those surveys have gone to my Gmail spam folder, including the most recent one. Why? Poor sender reputation, I think. Why? I would guess that perhaps I am not the only one questioning why I'm receiving mail from somebody I didn't give my email address to. I didn't report this mail as spam, but it sure looks like enough other people are reporting this sender's mail as spam and thus, making it near impossible for them to reliably get to the inbox.
That's the practical consideration. Not whether or not you think what you're doing is legal or expected or common or necessary; what recipients think is given greater weight. Enough weight that it can bog down your sender reputation.
That's why a mail stream or marketing program or survey program probably just doesn't work without clear cut permission. Regardless of what you think is right or wrong, your opinion (and my opinion) is only a tiny part of the equation.
Is Yahoo.com a wireless domain?
First published in 2005, the FCC's Wireless Domains list was intended to be a list of domains associated with mobile devices (cell phones, pagers, etc.) and that senders of commercial messages were to avoid those messages unless appropriate consent was obtained for each recipient. It sounds like a simple "don't spam me" list, but the form of consent referenced "must include the subscriber’s signature, which may be in digital or electronic form as allowed under the federal E-Sign Act and state counterparts" and the FCC has said that the burden of proof to resolve any complaint rests squarely on the sender, so the net was that most email service providers prohibited their clients from sending to those domains, unless the client implements verifiable consent compatible with the E-Sign Act or similar. (A longer discussion on what constitutes appropriate consent might make sense here but I don't have the time to dive deep and my focus today is more on the Yahoo.com domain landing on the list, see below.)
Email industry insiders noted that the domain yahoo.com had landed on the FCC wireless domains list sometime in the past few days, meaning that if this procedure were to be followed, email services providers would have thirty days at max before they would be forced to restrict their clients from sending mail to subscribers at Yahoo Mail's primary domain name.
That's potentially a big deal! Thankfully, it seems to be recognized as an error and is being addressed. I know that both the FCC and Yahoo have been notified of this and my understanding is that it is likely to be resolved very soon, meaning that it probably won't be necessary for a bunch of senders to suddenly stop sending mail to their yahoo.com subscribers. Whew!
(Update: Yahoo.com has been removed from the FCC Wireless Domains list.)
Email industry insiders noted that the domain yahoo.com had landed on the FCC wireless domains list sometime in the past few days, meaning that if this procedure were to be followed, email services providers would have thirty days at max before they would be forced to restrict their clients from sending mail to subscribers at Yahoo Mail's primary domain name.
That's potentially a big deal! Thankfully, it seems to be recognized as an error and is being addressed. I know that both the FCC and Yahoo have been notified of this and my understanding is that it is likely to be resolved very soon, meaning that it probably won't be necessary for a bunch of senders to suddenly stop sending mail to their yahoo.com subscribers. Whew!
(Update: Yahoo.com has been removed from the FCC Wireless Domains list.)
Interesting SBLs is back
Ever seen the @InterestingSBLs twitter account? It's kind of interesting and occasionally entertaining. It highlights various SBL entries that its anonymous author finds "interesting" by whatever criteria that may be. Because it's an ESP? An ESP's client? A Fortune 500 company? Not sure, but all have appeared there. My own employer has occasionally been called out on it, as have others. Some representatives of some companies have gotten really upset over being mentioned by that Twitter account, but not me. To me, it's really just a synopsis of a public record. And good companies occasionally have Spamhaus issues, too-- not just bad companies. It tells me it's something "interesting" to go look at, not that so-and-so is a scumbag spammer. If you or your company gets mentioned there, take a deep breath and look into it.
There are often big gaps between when @InterestingSBLs posts, but he or she seems to have been active as recently as just over a week ago.
If that doesn't interest you, there's always the Spamhaus SBL "Latest Entries" page, showing you what has been recently entered or recently removed into Spamhaus's main blacklist. This can be pretty interesting. I once knew an alleged spammer who spent most of his day hitting "refresh" on this page every few minutes, looking to find that partners in (alleged) crime may have been caught in the Spamhaus cross hairs.
Keep in mind that all Spamhaus SBL entries are effectively public information. Spamhaus does not password protect or otherwise obviously restrict access to the listing information available on their website. (I'm not necessarily making a case for whether or not they should be public or not, just noting how it is today.)
There are often big gaps between when @InterestingSBLs posts, but he or she seems to have been active as recently as just over a week ago.
If that doesn't interest you, there's always the Spamhaus SBL "Latest Entries" page, showing you what has been recently entered or recently removed into Spamhaus's main blacklist. This can be pretty interesting. I once knew an alleged spammer who spent most of his day hitting "refresh" on this page every few minutes, looking to find that partners in (alleged) crime may have been caught in the Spamhaus cross hairs.
Keep in mind that all Spamhaus SBL entries are effectively public information. Spamhaus does not password protect or otherwise obviously restrict access to the listing information available on their website. (I'm not necessarily making a case for whether or not they should be public or not, just noting how it is today.)
Purchased lists? DOA.
MailChimp's John Foreman explains the usual reasons why sending to purchased lists is a bad idea, but then he adds on one important practical fact: They perform very poorly. Click on over to the MailChimp blog to learn more.
Pre-order Spam Nation by Brian Krebs
From Brian Krebs: "The backdrop of the story is a long-running turf war between two of the largest sponsors of spam. A true-crime tale of political corruption and ill-fated alliances, tragedy, murder and betrayal, this book explains how the conditions that gave rise to this pernicious industry still remain and are grooming a new class of cybercriminals.
But Spam Nation isn’t just about junk email; most of the entrepreneurs building and managing large-scale spam operations are involved in virtually every aspect of cybercrime for which there is a classification, including malware development, denial-of-service attacks, identity theft, credit card fraud, money laundering, commercial data breaches and extortion."
I'll be ordering my copy soon!
But Spam Nation isn’t just about junk email; most of the entrepreneurs building and managing large-scale spam operations are involved in virtually every aspect of cybercrime for which there is a classification, including malware development, denial-of-service attacks, identity theft, credit card fraud, money laundering, commercial data breaches and extortion."
I'll be ordering my copy soon!
Does Gmail use Spamhaus blacklists?
Probably, implies Return Path based on a correlation between a typical Spamhaus blacklisting and drop in inbox delivery rates at Gmail. I think it's safe to assume that Google does use Spamhaus data for some sort of reputation calculation impacting Gmail deliverability.
Ask Al: Should I add a DMARC record to fix the Yahoo issue?
A friendly representative of a company who helps small businesses sell products asked: "We're having problems forwarding mail from our customers back to our users due to the new Yahoo and AOL restrictive DMARC policy. If we add a DMARC record for our own domain name, would that help address the Yahoo/AOL bouncing issue? Would that explain to the ISPs that we're not spoofing when we forward on that mail?"
No, this wouldn't fix your issue. It's probably not a bad idea for you to implement a DMARC record for your domain, especially if the domain is one you use for email marketing or online retail and want to make it harder for bad guys to spoof it. (But be sure you learn more about DMARC before proceeding; I would recommend partnering with somebody like Return Path or Agari to use their tools and benefit from their expertise with regard to anti-phishing/spoofing and DMARC.)
The reason this wouldn't fix your issue is because the Yahoo and AOL DMARC policies affect only mail that has a Yahoo or AOL domain in the from address. Also, they have the potential to affect all/any mail with a Yahoo or AOL domain in the from address. What other domain you might have in the message or message headers has no bearing on that fact. Whatever DMARC policy setting you publish wouldn't override whatever policy setting the owner of those domains may have published. In other words, if it's AOL.com in the from address, it's always going to be the AOL.com policy that applies, no matter what.
The real fix for the issue is to figure out how to get it so only your own domain name shows up in the from address. That might necessitate a change in your message flow process. It might make you have to reconsider whether or not you forward on messages through your system at all. Or you might have to rewrite headers, if you still want to be able to forward on that mail.
No, this wouldn't fix your issue. It's probably not a bad idea for you to implement a DMARC record for your domain, especially if the domain is one you use for email marketing or online retail and want to make it harder for bad guys to spoof it. (But be sure you learn more about DMARC before proceeding; I would recommend partnering with somebody like Return Path or Agari to use their tools and benefit from their expertise with regard to anti-phishing/spoofing and DMARC.)
The reason this wouldn't fix your issue is because the Yahoo and AOL DMARC policies affect only mail that has a Yahoo or AOL domain in the from address. Also, they have the potential to affect all/any mail with a Yahoo or AOL domain in the from address. What other domain you might have in the message or message headers has no bearing on that fact. Whatever DMARC policy setting you publish wouldn't override whatever policy setting the owner of those domains may have published. In other words, if it's AOL.com in the from address, it's always going to be the AOL.com policy that applies, no matter what.
The real fix for the issue is to figure out how to get it so only your own domain name shows up in the from address. That might necessitate a change in your message flow process. It might make you have to reconsider whether or not you forward on messages through your system at all. Or you might have to rewrite headers, if you still want to be able to forward on that mail.
Need to contact Live.com/Hotmail?
On a mailing list I subscribe to, someone recently asked for assistance with getting mail delivered to Microsoft's Live.com and Hotmail.com domains. Apparently the poster works for an internet provider that had a compromised account or two, and Microsoft was blocking their mail as a result.
A kind soul posted this reminder in response:
"The most efficient and effective way to address any deliverability problem is by submitting the issue to our dedicated deliverability support team. Senders can do this by filling out a form with detailed information necessary to diagnose the problem. The form can be found here: https://support.microsoft.com/en-us/getsupport?oaspworkflow=start_1.0.0.0&wfname=capsub&productkey=edfsmsbl3&ccsid=. Going directly to deliverability support will ensure that we have the right information to investigate the cause and recommend the right solutions quickly."
Those of us who have been around a while already have that URL bookmarked, but I thought I would share it here for new folks who might not already know it.
(June 2015 update: Link has been updated to most current version.)
A kind soul posted this reminder in response:
"The most efficient and effective way to address any deliverability problem is by submitting the issue to our dedicated deliverability support team. Senders can do this by filling out a form with detailed information necessary to diagnose the problem. The form can be found here: https://support.microsoft.com/en-us/getsupport?oaspworkflow=start_1.0.0.0&wfname=capsub&productkey=edfsmsbl3&ccsid=. Going directly to deliverability support will ensure that we have the right information to investigate the cause and recommend the right solutions quickly."
Those of us who have been around a while already have that URL bookmarked, but I thought I would share it here for new folks who might not already know it.
(June 2015 update: Link has been updated to most current version.)
Check out this neat thing: Email Privacy Tester
Smart web developer and systems admin Mike Cardwell put together something neat that I think you'll want to check out: Email Privacy Tester. Plug in your email address, and his system will send you an email message. That message will contain a whole bunch of different types of active content in various different wrappers and encodings, to see which of them are triggered by your MUA (mail user agent; aka email client). I did a quick test to a Gmail account and I already see one thing that I don't like, something Gmail executes automatically without asking me.
DMARC does only one thing (but pretty well)
DMARC (specifically, DMARC with a p=reject policy) does one thing: It makes it very difficult for somebody to use your domain in a from address without your permission. It doesn't stop spam from open relays, open proxies, snowshoeing network blocks, nor does it prevent people from ignoring CAN-SPAM and continuing to mail you after you have unsubscribed.
It doesn't do all those other things; but the one thing it does, it seems to do it very well.
It has limitations; yes. It doesn't specifically do anything the cousin domain problem (deceptively-registered domains with similar, but not exactly the same names). It makes mailing list management more complex. And it makes it so for the most part, users of email service providers (ESPs) cannot use Yahoo or AOL addresses as from addresses (because Yahoo and AOL publish strict DMARC policies).
Is anybody saying that DMARC is a magic bullet? I don't think so, but a better question is -- Does DMARC help? Yes, says Yahoo.
But spammers eventually moved on to forging one of Yahoo's domains, so it was a pointless exercise, others have said. Pointless exercise? I don't agree with that at all. It's true that Yahoo only published a strict policy for one of their domains; it's almost like they decided to lock only one door first, either because it was the most abused, or perhaps because you have to start somewhere (and maybe want to see how that goes before adding others).
But if I were a Yahoo or AOL, or a financial institution, or well known retail brand, would I think it is a win; a gain, quite likely a long term one, if I could keep bad guys away from using my domain in the from address of their bad mail? Yes, I would. DMARC helps. And I think when you look at who is implementing DMARC, I suspect that I'm not the only one who feels that way.
This reminds me so much of the days when I first got involved in spam fighting. Open relaying mail servers were a big bad problem. Spammers exploited them regularly and with great vigor. I was so tired of dealing with spam from open relaying mail servers that I started to block it, an effort that ended up becoming the MAPS Relay Spam Stopper, a coordinated third party blacklist that easily allowed mail system administrators to block mail from open relaying mail servers.
Spammers cried about being blocked. But not just spammers; there were also a non-zero number of people who claimed that I was part of some big conspiracy to wreck email. People called my employer, trying to get me fired. A number of people threatened to sue. A few high-profile savvy internet users like John Gilmore, explained that they needed to be able to legitimately relay mail for other people, and that blocking mail from those servers was akin to censorship.
Somehow, we survived all of that. Eventually, people seemed to accept this change. We got to a point where, while you do still see open relays out there from to time, there's a general consensus that it's a bad way to configure a mail server. But for us to get there, a number of us had to first band together and take action; personally implementing policies wherein we block mail from misconfigured servers used to shovel abusive, unwanted mail, while people who didn't understand why it mattered gave us hell for at the time.
I find a lot of parallel there; my point, other than I'm old and 1998 was a long time ago, is that I have a feeling that this will be a non-issue in a few years, and even if sites don't broadly implement DMARC with a p=reject policy, enough of them will do so that you're truly going to have to just deal with it.
It doesn't do all those other things; but the one thing it does, it seems to do it very well.
It has limitations; yes. It doesn't specifically do anything the cousin domain problem (deceptively-registered domains with similar, but not exactly the same names). It makes mailing list management more complex. And it makes it so for the most part, users of email service providers (ESPs) cannot use Yahoo or AOL addresses as from addresses (because Yahoo and AOL publish strict DMARC policies).
Is anybody saying that DMARC is a magic bullet? I don't think so, but a better question is -- Does DMARC help? Yes, says Yahoo.
But spammers eventually moved on to forging one of Yahoo's domains, so it was a pointless exercise, others have said. Pointless exercise? I don't agree with that at all. It's true that Yahoo only published a strict policy for one of their domains; it's almost like they decided to lock only one door first, either because it was the most abused, or perhaps because you have to start somewhere (and maybe want to see how that goes before adding others).
But if I were a Yahoo or AOL, or a financial institution, or well known retail brand, would I think it is a win; a gain, quite likely a long term one, if I could keep bad guys away from using my domain in the from address of their bad mail? Yes, I would. DMARC helps. And I think when you look at who is implementing DMARC, I suspect that I'm not the only one who feels that way.
This reminds me so much of the days when I first got involved in spam fighting. Open relaying mail servers were a big bad problem. Spammers exploited them regularly and with great vigor. I was so tired of dealing with spam from open relaying mail servers that I started to block it, an effort that ended up becoming the MAPS Relay Spam Stopper, a coordinated third party blacklist that easily allowed mail system administrators to block mail from open relaying mail servers.
Spammers cried about being blocked. But not just spammers; there were also a non-zero number of people who claimed that I was part of some big conspiracy to wreck email. People called my employer, trying to get me fired. A number of people threatened to sue. A few high-profile savvy internet users like John Gilmore, explained that they needed to be able to legitimately relay mail for other people, and that blocking mail from those servers was akin to censorship.
Somehow, we survived all of that. Eventually, people seemed to accept this change. We got to a point where, while you do still see open relays out there from to time, there's a general consensus that it's a bad way to configure a mail server. But for us to get there, a number of us had to first band together and take action; personally implementing policies wherein we block mail from misconfigured servers used to shovel abusive, unwanted mail, while people who didn't understand why it mattered gave us hell for at the time.
I find a lot of parallel there; my point, other than I'm old and 1998 was a long time ago, is that I have a feeling that this will be a non-issue in a few years, and even if sites don't broadly implement DMARC with a p=reject policy, enough of them will do so that you're truly going to have to just deal with it.
OpenDKIM & SpamAssassin Gotchas on Ubuntu 12.04
Allow me to share with you my rough notes compiled during my recent configuration of email and DKIM on a new Ubuntu VPS installation. Hopefully these helpful hints will help the next poor soul trying to get DKIM up and running on the first try:
(Originally posted to my personal blog, but I think it makes more sense here since it's talking about email, spam filtering and authentication.)
- SpamAssassin seems to want to fail DKIM keys from Gmail or Google Apps with an error of T_DKIM_INVALID. There's some noise online about this perhaps being an issue with clock synchronization, but it is more likely that you don't have one or more perl modules installed necessary for SpamAssassin to properly decode the DKIM key. The fix turned out to be simple; installing these packages solved that issue:
sudo apt-get install libmail-dkim-perl
sudo apt-get install libcrypt-openssl-random-perl
sudo apt-get install libcrypt-openssl-rsa-perl
(Thank you, Henrik Schack, for the tip!) - If you are new to OpenDKIM's Authentication Results header, you're going to be confused by this. You'll see a lot of DKIM as having passed, but with a reason of "1024-bit key; insecure key." This made me start poking around, looking at file permissions for my various keys (I set up signing for multiple domains). I assumed I had done something wrong, but I couldn't find any issue no matter where I looked. It turns out that it is not really an issue at all. What the error message actually means is that the domain that send you the message isn't using DNSSEC. Long term? Sure, yeah, everybody should look at DNSSEC, but one thing at a time.
- Here's what nobody tells you if you're DKIM signing multiple domains on the same server, using OpenDKIM: It is possible to interpret the opendkim.conf configuration file in a way that would lead you to add multiple sections starting with "domain," setting a selector for each and linking to different keys. Truth be told, OpenDKIM will only honor the LAST one of these sections, signing mail for only one of your domains. It won't generate any sort of error message, either, so it can be frustrating to understand what is happening. Remember, if you want to set up signing for multiple domains, look at how to configure the SigningTable and KeyTable settings in OpenDKIM.
(Originally posted to my personal blog, but I think it makes more sense here since it's talking about email, spam filtering and authentication.)
106 miles to Chicago
If you've ever set up catch-all domain names, that accept mail destined to any address, or set up your MTA in a way where all mail is accepted before checking to see if the recipient is valid, then you've probably seen emails containing this quote: "It’s 106 miles to Chicago, we got a full tank of gas, half a pack of cigarettes, it’s dark… and we’re wearing sunglasses."
The movie isn't new to me, but this is the first time I've run across the quote in this context. Apparently, for a while now, some spammer has been trying to find open relaying mail servers and seems to use this quote in almost every open relay attempt. I guess they just got around to trying it some place where I would notice the attempts.
Blast from the past: Challenge/Response Filtering?
A few weeks ago, somebody on the Mailop list asked about an email they received. Turns out, it was a Challenge/Response spam filter. You know, one of those emails from a robot that says "click here to prove you're not a robot, otherwise I'm not going to deliver your email on to the intended recipient." That doesn't work very well -- do you think Amazon has people waiting to click on that link each time they send out an emailed order receipt? Yeah, yeah, supposedly you (the user with the C/R spam filter) can whitelist that stuff, but you're going to miss something.
Challenge/Response filtering seems to have mostly disappeared; this was the first time I had heard about it in many years. But it's not the first time I've blogged about it; look all the way back to this MediaPost article from 2006, and you can see me raise still-extant concerns relating to how C/R works.
Click here to jump in the time machine and read what else I've written about C/R in the past.
Challenge/Response filtering seems to have mostly disappeared; this was the first time I had heard about it in many years. But it's not the first time I've blogged about it; look all the way back to this MediaPost article from 2006, and you can see me raise still-extant concerns relating to how C/R works.
Click here to jump in the time machine and read what else I've written about C/R in the past.
Signing outbound list mail with OpenDKIM
For my server running the discussion mailing lists that I host, I'm running OpenDKIM, OpenDMARC and Postfix on Ubuntu. I run a custom mailing list manager that I wrote myself; I keep telling myself that eventually I'll polish the code enough to share it with the world.
I wanted all posts to the mailing list to be signed with the DKIM signature of my mailing list domain name. Any DKIM signature the message was initially signed with when submitted to the list is rendered invalid by the header changes my mailing list software makes before distributing the posted message to all subscribers. Thus, mail would have been distributed to subscribers either with no signature or with a now failing signature. I feel strongly enough about email authentication to be dissatisfied by that; I want all mailing list messages to clearly demonstrate to the world that my server, my domain is responsible for the list mail.
To do this, I would in theory want to add OpenDKIM's "SenderMacro" directive to opendkim.conf. Except, support for SenderMacro is not compiled into OpenDKIM by default. No big deal, right? My brain shouldn't have atrophied too much from when I had a day job as a unix system administrator; I can just recompile it. But, it turns out, compiling OpenDKIM from source on Ubuntu is a hugely frustrating pain in the ass, with a ton of dependencies that I had to solve for manually, and package names that correspond to the absent library names in almost no way whatsoever. And even after I got it all compiled, there still ended up being a path issue with locating libraries at runtime that I just was not able to figure out. In theory I know how to fix that, but in practice, it just didn't work no matter what I did.
But then I stumbled across a much simpler way to do this: The "SenderHeaders" directive. The OpenDKIM README has a helpful section on mailing lists, where it explains that you can add this directive and set it to "Sender, From" to tell OpenDKIM, when deciding whether or not to sign the mail, look first at the domain in the sender header, then if that doesn't match the list of domains you sign, look at the From address domain to find a match.
My mailing list manager software doesn't use a sender header. I could easily add one, but I don't really like how that displays in the user interface of certain MUAs. Thankfully, the OpenDKIM documentation suggests an alternative header: "List-Post." Most mailing list manager software, including mine, includes a header designating what address you send to if you want to post a message to the mailing list. (The header looks like this: "List-Post: <mailto:LISTNAME@example.com>.")
I decided to give that a try and updated the SenderHeaders directive to read "SenderHeaders List-Post,Sender,From" and it worked! OpenDKIM looks at the List-Post header, sees a domain there that I can sign mail for, and it signs the mail with the key for that domain. The net result, all messages to the mailing list a signed with my DKIM key automatically.
I wanted all posts to the mailing list to be signed with the DKIM signature of my mailing list domain name. Any DKIM signature the message was initially signed with when submitted to the list is rendered invalid by the header changes my mailing list software makes before distributing the posted message to all subscribers. Thus, mail would have been distributed to subscribers either with no signature or with a now failing signature. I feel strongly enough about email authentication to be dissatisfied by that; I want all mailing list messages to clearly demonstrate to the world that my server, my domain is responsible for the list mail.
To do this, I would in theory want to add OpenDKIM's "SenderMacro" directive to opendkim.conf. Except, support for SenderMacro is not compiled into OpenDKIM by default. No big deal, right? My brain shouldn't have atrophied too much from when I had a day job as a unix system administrator; I can just recompile it. But, it turns out, compiling OpenDKIM from source on Ubuntu is a hugely frustrating pain in the ass, with a ton of dependencies that I had to solve for manually, and package names that correspond to the absent library names in almost no way whatsoever. And even after I got it all compiled, there still ended up being a path issue with locating libraries at runtime that I just was not able to figure out. In theory I know how to fix that, but in practice, it just didn't work no matter what I did.
But then I stumbled across a much simpler way to do this: The "SenderHeaders" directive. The OpenDKIM README has a helpful section on mailing lists, where it explains that you can add this directive and set it to "Sender, From" to tell OpenDKIM, when deciding whether or not to sign the mail, look first at the domain in the sender header, then if that doesn't match the list of domains you sign, look at the From address domain to find a match.
My mailing list manager software doesn't use a sender header. I could easily add one, but I don't really like how that displays in the user interface of certain MUAs. Thankfully, the OpenDKIM documentation suggests an alternative header: "List-Post." Most mailing list manager software, including mine, includes a header designating what address you send to if you want to post a message to the mailing list. (The header looks like this: "List-Post: <mailto:LISTNAME@example.com>.")
I decided to give that a try and updated the SenderHeaders directive to read "SenderHeaders List-Post,Sender,From" and it worked! OpenDKIM looks at the List-Post header, sees a domain there that I can sign mail for, and it signs the mail with the key for that domain. The net result, all messages to the mailing list a signed with my DKIM key automatically.
Yahoo on Yahoo's new DMARC Policy
What impact has Yahoo's new "p=reject" DMARC policy had on spam claiming to be from Yahoo.com? "DMARC has reduced spam purported to come from yahoo.com accounts by over 90%," says Alex Stamos, Yahoo's Vice President of Information Security, in his recent testimony before Congress.
H/T: Return Path's Ken Takahashi.
(Note: Alex's title may be Chief Information Security Officer; I've seen him referred to both ways. I decided to use the title referenced in the testimony.)
H/T: Return Path's Ken Takahashi.
(Note: Alex's title may be Chief Information Security Officer; I've seen him referred to both ways. I decided to use the title referenced in the testimony.)
The Current State of TLS over SMTP?
Michael Adkins, Mail Integrity Engineer at Facebook summarizes it thusly:
"STARTTLS encryption is widely supported and has achieved critical mass despite some issues with certificate management. A system deploying STARTTLS support for the first time can expect more than half of its outbound email to be encrypted. Also, the majority of deployments provide Perfect Forward Secrecy. We see two high priority areas for improvement. First, we encourage the industry to work together to develop better tools for preventing mismatched certificates. Second, we encourage everyone to deploy support for opportunistic encryption via STARTTLS."
TL;DR? Turn on opportunistic TLS, and if your results are like Facebook's results, at least half of the time your mail will be encrypted in transit. This is a very good thing, and adoption is only going to grow, especially when you've got a big site like Facebook who sends a lot of mail, helping to gently nudge folks in the right direction.
Want to learn more? Steve Atkins explains how to Protect your email with TLS over at Word to the Wise.
"STARTTLS encryption is widely supported and has achieved critical mass despite some issues with certificate management. A system deploying STARTTLS support for the first time can expect more than half of its outbound email to be encrypted. Also, the majority of deployments provide Perfect Forward Secrecy. We see two high priority areas for improvement. First, we encourage the industry to work together to develop better tools for preventing mismatched certificates. Second, we encourage everyone to deploy support for opportunistic encryption via STARTTLS."
TL;DR? Turn on opportunistic TLS, and if your results are like Facebook's results, at least half of the time your mail will be encrypted in transit. This is a very good thing, and adoption is only going to grow, especially when you've got a big site like Facebook who sends a lot of mail, helping to gently nudge folks in the right direction.
Want to learn more? Steve Atkins explains how to Protect your email with TLS over at Word to the Wise.
Yahoo Groups rewriting from addresses to handle DMARC policy
As reported on the Mailop list today, Yahoo has modified their Yahoo Groups discussion group service to "play nice" with posts from subscribers at domains behind a restrictive DMARC policy. Google recently implemented very similar changes to its Google Groups service. Both are believed to have done so in response to Yahoo and AOL having recently both implemented strict "p=reject" DMARC policies, affecting Yahoo users' and AOL users' ability to participate in some mailing list discussion forums.
Yahoo is modifying the original from address header to make the sending email address the Yahoo Group address and moving the original from address header to an X-Original-From header.
Yahoo Groups, Google Groups, mailman, and others services and software packages have been updating their handling of from addresses recently, in response to the challenges posed by restrictive DMARC policies that prevent off-network, unauthenticated uses of certain domain names in a from address. I've updated my own mailing list manager software similarly, and I'm sure you'll be seeing more of the same from other mailing list services and software packages in the near future.
Operators of discussion mailing lists and similar services pretty much have two possible choices for how to respond to this type of issue: Modify their software to write the from address to cooperate with this policy change, or lock Yahoo and AOL users out of participating in these various services and discussion groups; this course of action that doesn't seem widely accepted or adopted. Most list managers want their subscribers to be able to continue to participate, and most commercial service providers seem to not want to deny participation to a large swath of internet users. And certainly, one concern I've hear is, what happens when the next large ISP or webmail provider implements a similar policy?
May 15, 2014 update: Yahoo explains their changes to Yahoo Groups here.
(May 13, 2014 author note: I had an unfinished, half-written blog post in my draft queue trying to answer the question "What of Yahoo Groups?" I was basically going to say that it wouldn't surprise me to see Yahoo Groups update their service similar to how Google has done. Guess there's now no need to finish that one up.)
Yahoo is modifying the original from address header to make the sending email address the Yahoo Group address and moving the original from address header to an X-Original-From header.
Yahoo Groups, Google Groups, mailman, and others services and software packages have been updating their handling of from addresses recently, in response to the challenges posed by restrictive DMARC policies that prevent off-network, unauthenticated uses of certain domain names in a from address. I've updated my own mailing list manager software similarly, and I'm sure you'll be seeing more of the same from other mailing list services and software packages in the near future.
Operators of discussion mailing lists and similar services pretty much have two possible choices for how to respond to this type of issue: Modify their software to write the from address to cooperate with this policy change, or lock Yahoo and AOL users out of participating in these various services and discussion groups; this course of action that doesn't seem widely accepted or adopted. Most list managers want their subscribers to be able to continue to participate, and most commercial service providers seem to not want to deny participation to a large swath of internet users. And certainly, one concern I've hear is, what happens when the next large ISP or webmail provider implements a similar policy?
May 15, 2014 update: Yahoo explains their changes to Yahoo Groups here.
(May 13, 2014 author note: I had an unfinished, half-written blog post in my draft queue trying to answer the question "What of Yahoo Groups?" I was basically going to say that it wouldn't surprise me to see Yahoo Groups update their service similar to how Google has done. Guess there's now no need to finish that one up.)
Gmail’s message to email marketers: Focus on engagement
Campaign Monitor's Andrew Bonar reports on the takeaways from a recent presentation and Q&A session where Gmail's Sri Somanchi chatted with members of the ESPC (Email Sender and Provider Coalition).
How popular is Yahoo Mail?
That's a question I get asked from time to time, so I figured this would be a good place to link to numbers.
As reported on CNET today: "The company said it has 110 [million] daily active email users, though that figure includes users on every device, including desktop computers. Yahoo would not break down the figures for mobile, but said its number of users has grown 15 percent over the last two quarters."
As reported on CNET today: "The company said it has 110 [million] daily active email users, though that figure includes users on every device, including desktop computers. Yahoo would not break down the figures for mobile, but said its number of users has grown 15 percent over the last two quarters."
Google Groups rewriting from addresses to handle DMARC policy
Now that Yahoo and AOL have both implemented "p=reject" DMARC policies, Google has modified their Google Groups discussion group service to "play nice" with posts from subscribers at domains behind a restrictive DMARC policy.
If Google took no action with regard to Google Groups, whenever an AOL or Yahoo user tried to post to a Google Group, their post would be rejected by any ISP that rejects DMARC policy, including Comcast, Gmail, Yahoo and others.
The action they've taken looks like this: IF the post was submitted by a user at a domain that uses a restrictive ("p=reject") DMARC policy, THEN rewrite the from address so that the message is from "the list" instead of the person, AND add a reply-to header containing the original poster's email address.
The good: When you hit reply, your reply will go to the original poster, regardless of whether or not the from header was rewritten. Alternately, hit reply-all to reply to both the person and the list. The Google Groups user experience is essentially unchanged.
The bad: Some folks are saying this violates RFC 5322, which they claim says that the from address should (only) be the author of the message. It's not actually that strict-- it also says the from address can be the "system responsible" for the message. It also goes on to say that the from address "should not" be any address that doesn't belong to the message author. "Should not" has a specific definition in IETF parlance-- it allows for operational considerations to override initial guidance. Meaning, they admit there might be a reason you need to do something other than what they recommend.
The ugly: See how they're including the original poster's email address inside of the friendly from when they rewrite the email headers? I'd strongly recommend against this type of thing. It doesn't seem right to include an email address in a place where it can't be machine validated, and it potentially opens up subscribers to confusion down the road.
If Google took no action with regard to Google Groups, whenever an AOL or Yahoo user tried to post to a Google Group, their post would be rejected by any ISP that rejects DMARC policy, including Comcast, Gmail, Yahoo and others.
The action they've taken looks like this: IF the post was submitted by a user at a domain that uses a restrictive ("p=reject") DMARC policy, THEN rewrite the from address so that the message is from "the list" instead of the person, AND add a reply-to header containing the original poster's email address.
The good: When you hit reply, your reply will go to the original poster, regardless of whether or not the from header was rewritten. Alternately, hit reply-all to reply to both the person and the list. The Google Groups user experience is essentially unchanged.
The bad: Some folks are saying this violates RFC 5322, which they claim says that the from address should (only) be the author of the message. It's not actually that strict-- it also says the from address can be the "system responsible" for the message. It also goes on to say that the from address "should not" be any address that doesn't belong to the message author. "Should not" has a specific definition in IETF parlance-- it allows for operational considerations to override initial guidance. Meaning, they admit there might be a reason you need to do something other than what they recommend.
The ugly: See how they're including the original poster's email address inside of the friendly from when they rewrite the email headers? I'd strongly recommend against this type of thing. It doesn't seem right to include an email address in a place where it can't be machine validated, and it potentially opens up subscribers to confusion down the road.
AOL Adopts New DMARC Policy
Today, AOL announced that they, too, have adopted a "p=reject" DMARC policy. The same considerations previously mentioned as applying to Yahoo Mail users now apply to AOL users as well.
In today's AOL postmaster blog post, Vishwanath Subramanian offers some solid advice on how to deal with this change:
Beats the heck out of competing advice that said "just kick all the Yahoo users to the curb" when Yahoo implemented this change. If I were walking that line, I'd now have to kick out all AOL users. And then maybe Hotmail and Gmail users, too, if and when those other two big webmail providers followed suit.
In today's AOL postmaster blog post, Vishwanath Subramanian offers some solid advice on how to deal with this change:
In almost all cases, we recommend that you switch to sending mail from your own domain. You may also consider using AOL SMTP directly.
For mailing lists, also known as listservs, we recommend configuring reply behavior to fill the From line with the mailing list's address rather the sender's and put the actual user / sender address into the Reply-To: line. Please also note that current "auto unsubscribe" logic based upon bounces might be too rigid until this change has been in place for a while.
For website operators with 'share from email' functionality, please consider using an email address from your own domain as the From address and populate the Reply-To: line with the address of the person sharing.Solid advice. Especially the guidance about mailing lists; it roughly mirrors my prior advice.
Beats the heck out of competing advice that said "just kick all the Yahoo users to the curb" when Yahoo implemented this change. If I were walking that line, I'd now have to kick out all AOL users. And then maybe Hotmail and Gmail users, too, if and when those other two big webmail providers followed suit.
Yahoo DMARC Policy Change Roundup
Surprise! Or was it? I've been warning for a while now that DMARC doesn't play nice with mailing lists. But really nobody, not even me, thought that a big ISP like Yahoo was going to publish a "p=reject" DMARC policy. Nonetheless, they did publish such a policy in early April, and depending on who you ask, either panic and chaos has ensued since, or we're in the first stages of a new "this is how it is" era of mail.
Here's a roundup of posts from me (and a few other folks) on the topic of Yahoo's recent DMARC policy change.
Here's a roundup of posts from me (and a few other folks) on the topic of Yahoo's recent DMARC policy change.
- DMARC: Please Be Careful! June 10, 2013. My high level overview of operational considerations for domain owners when implementing a DMARC policy.
- Up in arms about Yahoo's DMARC Policy? You're not alone. April 8, 2014. What are the three areas of impact caused by this change?
- Run an email discussion list? Here's how to deal with DMARC. April 9, 2014. My suggestions on how to modify a mailing list manager to "play nice" with Yahoo's DMARC policy.
- Who uses a Yahoo from address? April 10, 2014. Andrew Barrett reports on his sampling of a live mail stream to identify groups affected by the Yahoo policy change.
- How OnlineGroups.net used the Yahoo! DMARC crisis to make a better Mailing List Manager. April 10, 2014. Dan Randow explains how OnlineGroups.net stepped up to the challenge of making their platform work within the new confines established by Yahoo's policy change.
- Yahoo DMARC Policy: Why they did it. April 11, 2014. My best guess as to what drove Yahoo to make this policy change.
- Yahoo Statement on new DMARC Policy. April 12, 2014. A bit of information from Yahoo on why they made this change and their recommendations for what senders may want to do to maximize interoperability with the new policy.
- Ask Al: Is my personal domain affected by DMARC? April 14, 2014. Explaining the potential impact on a user's personal domain name when an ISP uses a "p=reject" DMARC policy.
On April 7th, Laura Atkins of Word to the Wise posted "a brief DMARC primer" to help explain the technical concepts related to Yahoo's recent policy change and what this could mean for you.
Ask Al: Is my personal domain affected by DMARC?
All this talk about Yahoo's recent DMARC policy change got a friend to ask me about her domain name and whether or not this change has any impact on her.
Ellen asked me, "Does this mean anyone with a personal domain sending through an ISP who implements DMARC with a p=reject policy is going to have problems if they try to send mail to any recipient who checks DMARC?"
Ellen asked me, "Does this mean anyone with a personal domain sending through an ISP who implements DMARC with a p=reject policy is going to have problems if they try to send mail to any recipient who checks DMARC?"
Yahoo Statement on new DMARC policy
Yesterday, Yahoo posted "an Update on our DMARC Policy to Protect Our Users." They've also posted "Yahoo DMARC Policy Change - What Should Senders Do."
(H/T: WTTW)
(H/T: WTTW)
Yahoo DMARC Policy: Why they did it.
How dare Yahoo update their DMARC policy without warning the internet community of the potential fallout from doing so. At least, that's what some other folks have said. My take on it is more prosaic. I figure it's your domain name, you're free to do whatever you want with it. Initially, Yahoo made no statement, leaving us interested folks with nothing but our own speculation about why they've implemented this policy change. (They did later post a limited DMARC Help page and then also a more detailed statement explaining the change.) Here's my speculation.
How OnlineGroups.net used the Yahoo! DMARC crisis to make a better Mailing List Manager
Yahoo's recent DMARC policy change didn't just break somebody's church list. It also caused problems for every single discussion group hosted by OnlineGroups.net. Chief Wrangler Dan Randow and his team didn't take that sitting down. They didn't cry, shake their fists at the heavens, or order t-shirts that said "YAHOO BROKE MY MAILING LISTS AND ALL I GOT WAS THIS LOUSY T-SHIRT." Instead, they quickly came up with and executed a plan, implementing product changes within two days to make their collaboration platform compatible with Yahoo's DMARC domain policy. What did they do and how did they do it? Click on through to learn more about it.
Who uses a Yahoo from address?
In the next chapter in the story of Yahoo's recent DMARC policy change, Andrew Barrett shares a snapshot of what percentage of an example email service provider's clients send mail via the ESP using a Yahoo.com from address.
Run an email discussion list? Here's how to deal with DMARC
Yahoo's recent DMARC policy changes have made it so that Yahoo subscribers will now have trouble participating in old fashioned LISTSERV-style discussion lists. When a Yahoo user posts to your discussion list, very few subscribers will receive that message, because any ISP that respects DMARC policy will bounce that message. (And I believe that at least half of the top ten mailbox providers in the US now respect DMARC policy.)
Up in arms about Yahoo's DMARC Policy? You're not alone.
A few days ago, Yahoo updated their DMARC policy setting to "p=reject." What this means is, mail containing a Yahoo from address is basically no longer considered legitimate if it doesn't contain an authentication signature or if it didn't come from properly identified Yahoo infrastructure. (I'm oversimplifying things there, but bear with me; I think it's close enough for this discussion. Read more about it over at Word to the Wise.)
This effectively restricts Yahoo Mail users so that they can only send from their Yahoo email address when using the Yahoo Mail web user interface. For a big segment of regular joes, this may not ever be an issue. But for some people, this is a profoundly significant new restriction on what you can do with a Yahoo email address. Indeed, this change "brings the pain" for some, as Andrew Barrett explains over on the E-mail Skinny blog.
This effectively restricts Yahoo Mail users so that they can only send from their Yahoo email address when using the Yahoo Mail web user interface. For a big segment of regular joes, this may not ever be an issue. But for some people, this is a profoundly significant new restriction on what you can do with a Yahoo email address. Indeed, this change "brings the pain" for some, as Andrew Barrett explains over on the E-mail Skinny blog.
Payday Loans: Not Even Necessary
I have no problem helping a client address deliverability issues, even if their industry or politics encompass something I don't personally approve of. My friend Mickey Chandler and I (who have very different political affiliations) have worked capably together to help address deliverability and compliance issues for various political senders on both sides of the US political spectrum.
But payday lending holds a special place in my (dark) heart.
But payday lending holds a special place in my (dark) heart.
Masking WHOIS Information: No Abuse.net for you
The WHOIS process and protocol isn't just some nerd thing that goes back a hundred years; it's a valuable public directory for savvy internet users to be able to identify who owns a given domain name. Spam and security investigators find it a valuable tool -- even if sometimes bad guys submit bogus details, commonality of information across domains allows them to paint a clearer picture of who is behind a bad act or how broad that bad act may be.
SpamAssassin 3.4.0 Released
The Apache Software Foundation just announced the release of SpamAssassin 3.4.0 via a message posted to the SpamAssassin announcements email list by project chair Kevin A. McGrail. Release notes menion that "this is a major release. It introduces over two years of bug fixes and
features since the release of SpamAssassin 3.3.2 on June 16, 2011." To learn more, head on over to their website.
features since the release of SpamAssassin 3.3.2 on June 16, 2011." To learn more, head on over to their website.
Gmail Oops
As reported over on the iDownload blog (and as mentioned to me by others), Gmail had a significant oops lately. Apparently, there was some system glitch that resulted in some actions (such as "delete" and "report spam" being applied to a message other than the one a user selected. Perhaps not all Gmail users were affected; I didn't receive the notification that other Gmail users have reported receiving.
In case you're curious, here's what the notification said:
Important Notice
You may have been impacted by a recent issue in Gmail that inadvertently caused some actions (e.g. delete, report spam) taken while viewing a message to be applied to a different message. The issue occurred between January 15 and January 22 and is now fixed.
We encourage you to check your Trash and Spam folders before February 14, 2014 for any items you did not intend to delete or mark as spam and move them back to your inbox. We apologize for any inconvenience.
In case you're curious, here's what the notification said:
Important Notice
You may have been impacted by a recent issue in Gmail that inadvertently caused some actions (e.g. delete, report spam) taken while viewing a message to be applied to a different message. The issue occurred between January 15 and January 22 and is now fixed.
We encourage you to check your Trash and Spam folders before February 14, 2014 for any items you did not intend to delete or mark as spam and move them back to your inbox. We apologize for any inconvenience.
Gmail: Reach the people you know more easily
On January 9th, Google announced that your Google+ contacts (people in your G+ circles) will now automatically show up as contacts inside of Gmail.
This isn't really a good thing, from my perspective. A lot of (OK, just about all of) my G+ connections are acquaintances or people who know me from the industry (many of whom I do not know personally). Making the ability to email in Gmail automatically visible to all of those folks feels like something akin a privacy violation. I don't really want to end up on somebody's huge CC list of forwarding some joke or political complaint. I think if folks want to contact me, they can go to my website and actually click on the contact me link.
I think this is auto contact enabling is a bad idea. If you agree and want to disable this new functionality, Consumerist explains how to do so: Go into "Settings" in your Gmail account, and under the "General" tab you will find an "Email via Google+" entry where you can change the setting so that "no one" of your Google+ contacts is automatically given the ability to send you email from inside of Gmail.
This isn't really a good thing, from my perspective. A lot of (OK, just about all of) my G+ connections are acquaintances or people who know me from the industry (many of whom I do not know personally). Making the ability to email in Gmail automatically visible to all of those folks feels like something akin a privacy violation. I don't really want to end up on somebody's huge CC list of forwarding some joke or political complaint. I think if folks want to contact me, they can go to my website and actually click on the contact me link.
I think this is auto contact enabling is a bad idea. If you agree and want to disable this new functionality, Consumerist explains how to do so: Go into "Settings" in your Gmail account, and under the "General" tab you will find an "Email via Google+" entry where you can change the setting so that "no one" of your Google+ contacts is automatically given the ability to send you email from inside of Gmail.