Microsoft: 451 4.7.500 Server busy delays on Wednesday 10/11

Today, Wednesday, October 11th, many senders are reporting trouble (specifically, delayed mail) when attempting to send email to Microsoft-hosted subscribers. Here's more on the issue from Steve Atkins from Word to the Wise. 

I'm keeping a close eye on my Office365 admin dashboard to watch for updates on this issue. Microsoft has logged this as "incident EX680695" and those with O365 admin access can go to Home -> Health -> Exchange Online -> Health to see the current status and click on the incident/advisories there to find the latest details. Here's a direct link to that incident information, which likely only works if you have appropriate access.

Many (but not all) senders were impacted, seeing "451 4.7.500 Server busy" delivery delays when sending to email domains hosted by Office 365 / Exchange Online. Many of the senders reporting trouble were located in the EU; suggesting that the issue may be EU-specific, or specific to an IP range primarily used by those in the EU.

Microsoft lists the "start time" for this issue as 4:00 pm central time on 10/9, but that doesn't quite jibe with most of the complaints I saw. Most folks seemed to suggest that the issue started overnight at 3:00 am or 4:00 am US central time on 10/11. Microsoft is listing the "end time" for this issue as 1:24 pm US central time.

The final status update for this incident is as follows:

Title: Some users may encounter delays sending and receiving external email messages in Exchange Online

User impact: Users may have encountered delays sending and receiving external email messages in Exchange Online.

More info: Impact was specific to Simple Mail Transfer Protocol (SMTP). Most users would have experienced up to a 1-hour delay; however, in some cases, users may have encountered delays of up to 24 hours. Additionally, the email sender may have received a Non-Delivery Report (NDR) with a "451 4.7.500 Server busy" error message, and their attempt to retry delivery of the email may have been delivered up to 24 hours later.

Final status: We’ve completed our deployment of the change, which was a targeted reversion of the code issue causing impact. Users should now be experiencing mitigation, though some residual delays may still occur over the next hour as the change completes.

Scope of impact: Impact was specific to some users who were served through the affected infrastructure. The issue affected users Worldwide and only for a portion of IP addresses, therefore not all users experienced impact.

Start time: Monday, October 9, 2023 at 4:00 PM CDT

End time: Wednesday, October 11, 2023 at 12:00 PM CDT

Preliminary root cause: A recent service update that was applied to a section of infrastructure responsible for enforcing IP address anti-spam rules contained a code issue, which was resulting in the delays with SMTP email delivery.

Next Steps:

- We're further reviewing the recent update to understand how the code issue was introduced and to understand what prevented it from being detected in our update testing and validation procedures, which will allow us to prevent similar issues in future updates.

We’ll publish a Post-Incident Report for this event within five business days.

As this issue appears to be resolved (or resolving), I'll make no further updates to this post.