Google Postmaster Tools: Domain vs. IP address Data Thresholds

Today's guest post is from Brian Curry, Manager of Deliverability, for Merkle Inc.. Take it away, Brian!

Since July of 2015, Google rolled out a shiny new tool for the Deliverability community to poke around and nerd out. We all know Google hasn’t always been the most transparent to Senders, so while we play with our new Gmail toy, we start to notice small things that perhaps give us clues into Google’s mind.

Consensus has been that volume that needs to exceed about 500/day on an IP address to warrant Google displaying reputation data for that IP address. Domain Reputation seemed to follow the same criteria (as far as we knew). Until a smaller volume sender showed something different for Domain Reputation thresholds.

Filtering to show the last 120 days of data...

We see the data point move on the Domain Rep graph to 3/16 with a High Rep, but the IP graph does not budge…interesting. What the below graph suggests is Domains have a lower volume threshold to display data.

More details for this given scenario:

  • This Sender has a dedicated IP and Domain for their use only.
  • No spikes in other Postmaster Graphs during this timeframe.
  • 12/29 - only 95 Gmail users were sent 
  • 1/12 - we saw 5,400 Gmail users sent.
  • 3/16 - the day the Domain Rep graph shoots up, only 65 Gmail users were sent. 

Notice how the IP address does not have a data point on 12/29 0r 3/16? This makes sense, the IP sent volume below 500 daily, except on 1/12. The Domain data points existing on 12/29, 1/12 and 3/16 is the intriguing part.

Perhaps the Domain volume threshold is lower? Anything above 50/day? Maybe this confirms how much importance Gmail places on the Domain (DKIM signed domains)? Engagement play a part there…maybe?



So, what does this all mean?

Domains are looked at differently. Google has long said that they have thousands of signals to determine where to deliver an email. I suspect signals for a Domain will continue to carry a heavier weight.

Let’s not try to game Gmail’s system, why even try? Instead let’s understand the importance of reputation on the sending Domain. Showing Gmail that your Domain is sending wanted mail is the only true way to have a strong influence on Gmail’s filters.

Back to nerding out...








Google Postmaster Tools: Not receiving data?

A few different folks have reported to me that when accessing Google Postmaster Tools, they were seeing this message being displayed instead of data:

No data to display at this time. Please come back later. Postmaster Tools requires that your domain satisfies certain criteria before data is available for this chart. Refer to the help page for more information.

What criteria? The most common one is that you must have enough sending volume to merit data being displayed. In one case, the deliverability consultant knew that the client was sending 50,000-200,000 messages to Gmail subscribers daily, so that can't be it.

It turned out, if you enter your domain name with any capital letters, Google Postmaster Tools won't display any data. In other words, if I enter my domain name into GPT as SpamResource.com, instead of as spamresource.com, no data will display. The fix is to go back in, remove your domain name, and re-add it, in all lower case.

The various deliverability consultants that have run into this have graciously given permission for me to share this here, so that other folks running into this same issue can more easily figure out how to address it. Thank you all!

B2B Spam is Dumb and You're Dumb and This Other Guy is Dumb, too

Every so often somebody approaches me to ask me if B2B spam is OK now, because they get B2B spam at their work address. Everybody gets it, they presume, thus it must now be acceptable. Or it was always acceptable and those deliverability guys were just trying to mislead them before.

Nope, B2B spam is still unwanted garbage. Google and Microsoft are the biggest outsource hosts of mail for corporate domains nowadays and they'll tell you flat out that they have no intention of purposely allowing unsolicited mail through, because they know their users don't like it. Sure, you get B2B spam. I get both B2B and B2C spam. That B2B spam doesn't stop because 1. you're not complaining about it enough and 2. it's often sent in such small volumes that spam filters haven't gotten enough data points to understand that it's unwanted.

For number one, I don't blame you. Reporting spam correctly is hard, and spammers try hard to make it harder and more confusing. But I promise you, if you take every B2B spam you get for a week, look up the return-path domain in the email headers, look up that domain in abuse.net, and then send a complaint to the address found in abuse.net, you'll get noticed, and the spam volume will probably go down.

You won't stop the "Sloan / Data Champions" idiots who are trying to sell you B2B lists, because they're based in India, have been doing bad stuff for years, and don't care. Spam filters are getting better at blocking those guys, though. If you want to have some fun with somebody like that, reply and tell them you're interested. Then they'll respond to you from their real domain name, the one they weren't mentioning in the original spam (because they don't want it to get blocked), and then block that domain.

For number two, it's sort of a self-solving problem in the long term. Companies sending more and more B2B spam are going to garner more and more complaints. Technological loopholes, like jumping IP addresses and domains, are getting smaller and smaller. Stuff like DKIM and DMARC pushes all senders to have to use reliable, stable sender identifiers, making it easier for ISPs to calculate aggregate stats across the mail they're sending. I think right now Google and Microsoft are thinking about how to make their platforms better catch B2B spam, and have been thinking about this for a while. Why? Because spam ruins a user's enjoyment of an email platform. Many users blame the email platform for the spam they let through.

And it surely doesn't help when some newly minted, self-proclaimed email marketing expert stumbles on the scene and decides that he's got it all figured out. This isn't even a phenomenon unique to the United States. Over on the Mainsleaze blog, Finnish anti-spam expert Atro Tossavainen highlights one such example from his neck of the woods.

"Hardcore Business Man" Filip Poutintsev is sure that spam isn't a problem. He tries to debunk the myths of spam in a hilariously ignorant post. Top highlights:

  • Spam is bad because it's not wanted. He says most advertising is unwanted. Sure, but legitimate advertising sponsors something, or is done under an agreement. It covers the cost for something. This is completely untrue with spam.
  • It wastes your time when you have to check spam in your mailbox. He says TV ads are worse. Not true for the 4.9 million of us in the USA who 'cut the cord' in 2015. Ever had a mailbox overrun with spam? I have. Do you want to have to dig through a hundred spams for tactical flashlights and medicare scams to find an email from a loved one? Why is that OK for those people to inject that stuff into your mailbox without permission?
  • If you don't want TV or radio ads you can just turn of TV or radio. You can't hide from ads, he says. Is that true? I don't think so. But where they are, they're sponsoring something. Spam is not similarly paying its way.
  • Spam overloads email traffic. His rebuttal to this one is too laughable to bother quoting. He's not a network engineer. Ask Microsoft how much network traffic, processing power and disk space is wasted by spam. Don't ask this guy.

Anybody who's been fighting spam for a while knows that this guy is ignoring the cost shifting argument. Why spam isn't wanted is because it wastes our time and our servers' time and space, and we didn't ask for it, and it's not sponsoring something we asked for. If I tune into Saturday Night Live to watch that Prince tribute, there's a very clear, even though implied, agreement that I'm allowing those commercials onto my TV as they're subsidizing my ability to watch that television program.

If you send me a spam email, you're injecting something into my view without my agreement and without my permission. (And the 'but it's just one argument' fails because it's never just one. What you do today the next three guys want to do tomorrow, and the next nine guys the next day. It literally impedes the ability to deal with email in a timely manner.)

Great minds think alike: More on this topic from Laura Atkins of Word to the Wise.

Now Hiring: Deliverability Coordinator @ BlueHornet

Here's a good opportunity from the good folks at email service provider BlueHornet. They're looking to hire a Deliverability Coordinator. My contact there shared the following info: "The position will be local to San Diego, CA.  The Deliverability Coordinator will be responsible for compiling data for customer presentations and pulling reports for delivery-related analysis.  The person in this position will also be responsible for creating DKIM and SPF entries, as well as assisting customers with enabling the DNS entries that are necessary for good delivery.  This individual may also assist in the day-to-day coordination for agency and reseller accounts, but the primary responsibility will be to provide support to the deliverability team.  PowerPoint experience is required.  Familiarity with SQL, DNS, and email filtering systems is a plus, but training will be provided on all technical aspects of this position." Apply here.

Microsoft Outlook.com / Hotmail Deliverability Troubleshooting

Here's my top five tips for deliverability success when sending to Microsoft’s Outlook.com / live.com / Hotmail email platform:

  1. Publish SPF records and sign mail with DKIM authentication. Sender Policy Framework (SPF) is a simple DNS record that conveys what IP addresses are allowed to transmit mail for your domain. It's easy to configure and there's absolutely no reason you shouldn’t have already implemented SPF. Domain Keys Identified Mail (DKIM) is a little bit trickier. Your (or your ESP’s) outbound mail server needs to support DKIM directly; it involves signing outbound mail with a cryptographic key. Microsoft’s Outlook.com webmail platform checks for DKIM and SPF, and you’ll want to have both in place, one to back up the other. I’ve been occasionally seeing odd DKIM failures when sending to Outlook.com. Very low numbers, but in that scenario, mail is still getting delivered fine, because SPF is in place and acting as a backup.
  2. Be sure to ramp-up your sending IP address. Microsoft explains: “IPs not previously used to send email typically don't have any reputation built up in our systems. As a result, emails from new IPs are more likely to experience deliverability issues. Once the IP has built a reputation for not sending spam, Outlook.com will typically allow for a better email delivery experience.” Meaning: Don't expect to send to 30 million Hotmail subscribers on day one. Build up that volume over time on a new IP address. Assuming it's wanted mail, it'll probably take three to four weeks until you can get up to full speed. If you're sending unwanted mail, life probably isn't going to get easier.
  3. Sign up for SNDS. Microsoft’s Smart Network Data Service (SNDS) is a reputation portal that helps you to understand deliverability-related reputation issues. After you’ve signed up, and requested access to view data about your sending IP address(es) you can get some idea if you’re hitting spamtraps, and whether or not your reputation is red, yellow or green. If you’re seeing a consistently “red” (bad) sending reputation, it could be that you’re getting too many complaints (if you see numbers suggesting an elevated spam complaint rate) or it could be related to Windows “SRD” (Sender Reputation Data).  How SRD works is that Microsoft queries a small sampling of your subscribers, allowing them to provide more direct feedback regarding your email messages. Does the recipient feel the message is spam? If enough of those people say “yes” then your deliverability reputation is really likely to suffer. To avoid that, don’t spam, don’t buy lists and be very transparent to subscribers. If need be, cut back to mailing only engaged subscribers, after dumping any questionable data that may not be opt-in.
  4. Don’t get tagged for Namespace Mining. Microsoft explains: “This is the practice of verifying email addresses without sending (or attempting to send) emails to those addresses. This method is commonly used by malicious senders to generate lists of valid e-mail addresses that they can send spam, phishing emails or malware. Microsoft does not allow this behavior and takes action on IPs that engage in it.” I’ve seen this happen to senders where I’m fairly certain they’re not engaged in such a practice, and I think in that scenario, the sender has such a horribly dirty list, with so many invalid addresses, that they’ve tripped this trigger at the ISP. Solution: Don’t mail really, really old lists that you found in the back of some filing cabinet. Attempt to mail enough invalid addresses, and you’ll run into this issue. It also goes without saying that Microsoft is explicitly saying that SMTP-based address validation is not allowed.
  5. Get Certified. Return Path runs a paid Certification whitelist, and Microsoft is listed (by both Microsoft and Return Path) as a user of this whitelist. If you pass the vetting process and your sending IP address gets certified, you’ll be able to bypass some filtering at Microsoft's Outlook.com and other ISPs. Do note that this isn’t a free pass to spam; if you garner too many spam complaints (or negative SRD votes) you could find your certification suspended.
If you're experiencing blocking when sending to outlook.com / live.com / hotmail.com subscribers, here's where you need to go to request that you be unblocked.

Microsoft operates a feedback loop, called the Junk Mail Reporting Program (JMRP). If you're working with an email service provider (ESP), they've likely already signed up for the JMRP and you're benefitting from it directly without having to take any action. If you're sending from your own server, and not using an ESP, then you'll likely want to enroll your sending IP addresses in JMRP. You'll need to specify an email address that Microsoft can send complaints to, and you are expected to unsubscribe people who complain. Also note that if you send lots of mail, you're going to get lots of complaints. Even if everything is on the up-and-up. Point being, don't send the complaints to some personal mailbox where the receptionist has to check it and manually process things. It'll fill up quick and possibly crash your mail server. Feedback loops (FBLs) typically need automation and a technically-minded process review.

Keep in mind that you don't need to bother with Sender ID. Dated deliverability guidance abounds out on the Internet; you'll still find reference guides that talk about Sender ID. SPF has since taken its place.

Microsoft's Office 365 platform operates its own unblocking request form. It has been suggested that at some point in the future these platforms may merge, and thus some of these support / request mechanisms may merge, but that does not seem to be the case as of April, 2016.

(Special thanks to Luke Martinez and Laura Atkins for helping to clarify the proper place to go to request unblocking.)

I'm blocking all mail from .top

I'm blocking all mail from the new .top TLD, because I'm getting absolutely pummeled with spam from a spammer or small group of spammers rotating through .top domains, trying to hide who they really are. You might want to avoid the .top TLD for email purposes right now; since the only samples I can see are bad things, you won't be in very good company.

UnsubCentral: Anybody home?

I emailed your support address five days ago and haven't heard back. Are you out there? Please get back to me. Thanks!

Gmail: Top 5 Deliverability Do's and Don'ts

Want solid inbox delivery at Gmail? Here's what you need to do (and not do) in 2016:
  1. DO enable DKIM and SPF authentication. Sender Policy Framework (SPF) is a simple DNS record that conveys what IP addresses are allowed to transmit mail for your domain. It's easy to configure and there's absolutely no reason you shouldn’t have already implemented SPF. Domain Keys Identified Mail (DKIM) is a little bit trickier. Your (or your ESP’s) outbound mail server needs to support DKIM directly; it involves signing outbound mail with a cryptographic key. You don’t see a message’s signature by default; it’s stored in a hidden header. But the receiving ISP (in this case, Gmail) uses that to ensure that the mail was truly from the domain name it claims to be from. (That’s a simplistic definition that any handful of email geeks will take issue with, but I’m going to stand behind that and say “that’s what it boils down to.”) Mail that lacks SPF and DKIM authentication is going to be more closely scrutinized by various filtering steps at Gmail. It’s not the only determining factor in inbox placement by any means, but it definitely matters.
  2. DO enable TLS. Back in November, Google warned that they were going to eventually highlight mail sent over unencrypted SMTP connections. They started doing so in February, adding a scary little "unlocked" icon to messages sent over SMTP connections that don't utilize Transport Layer Security. The reality of the situation is that Google is pushing email senders to make it harder for bad guys and governments to monitor email in transit. I don't think sniffing unencrypted marketing mail is that useful to either bad guys or governments, so I question this focus. But if you don't get rid of that little "unlocked" icon, your subscribers will probably start to wonder about you. Best to jump on board and just do it, for perhaps no real reason other than Google wants you to do it. Enabling TLS is typically pretty simple-- it's already supported by almost all mail server software and typically just needs to be enabled. The only likely downside is that the encryption adds overhead to both the mail server software and the connections it makes to other servers. What that means is that mail can end up being delivered a bit more slowly with TLS enabled.
  3. DON'T focus on "Promotions" vs "Updates" tab placement. Google wants to place marketing mail in the "Promotions" tab and trying to outsmart them is something you'll possibly get away with in the short term, but not in the long term. Google likely notices mis-categorized mail and any workaround will probably stop working after their next system update. Instead, focus on sending wanted mail and subscribers are going to read it regardless of what tab it shows up in. (And keep in mind, whichever tab it is under, it is actually in the inbox.)
  4. DON'T buy or rent lists. Google is very sensitive to feedback from Gmail subscribers. Send mail to a purchased list, and enough people will mark that mail as spam, and Gmail's filters will pick up on that and then all your mail goes to spam. Cleaning up after you fall into this trap takes time and is not fun. There is nobody we can call at Google to get them to give you a do-over and reconsider your lack of inbox placement. Sure, we all know people at Google, but they aren't willing to take those calls, because they think the filters are working as advertised and you're getting the deliverability you deserve.
  5. DO focus on engagement. Gmail's filtering is heavily engagement-driven. (You will occasionally find a marketer who swears otherwise, but they're full of beans. I am not.)  If you're struggling to get solid inbox placement, a common fix is to adjust who you're sending to so that you are mailing only subscribers who have opened or clicked in the past eighteen months. Do that for your next three-to-four weeks of mailings and that will likely result in you getting solid inbox placement, and should keep your mail out of the spam folder. (If your mail truly is wanted by your subscribers.)  Then you have to decide what to do about the rest of your subscriber base, the people who aren't responding. There's a potential for strategy there that goes beyond the scope of this simple blog post. Talk to your friendly neighborhood deliverability consultant to figure out how best to handle this.
And here's a bonus tip: DON'T focus only on a technological solution for something that is probably a permission problem. Somewhere around 99% of the time, the reason your mail is going to the spam folder is not going to be because you or your ESP missed something or didn't fully follow Gmail's Bulk Sender Guidelines. There isn't some magic header or code you need to add that will fix a deliverability problem caused by low engagement or a lack of permission. Sure, occasionally an ESP or email platform will mess something up, and you should definitely work with them to check to see that SPF, DKIM, bounce processing, and all that other stuff are working properly. But I just want to set the expectation appropriately -- it is pretty rare that a technical failure is the source of a deliverability issue.

(ETA: Of course, saying this last bit got me into trouble -- the very next client issue I looked at, a DNS issue was causing odd DKIM failures. But I still maintain that this the exception, not the rule.)

Want to learn more about Google's Gmail filters? Laura Atkins from Word to the Wise has got you covered.