A maintenance bug took Microsoft 365 down for four hours

On Thursday, SharePoint, OneDrive, and Teams stopped working for a large slice of Microsoft's customers for the better part of four hours. The cause wasn't an attack or a hardware failure. A piece of Microsoft's own automation mis-scoped a routine maintenance job and pulled network routes it was never supposed to touch. If you write automation that changes production — and if you run agents, you now do — the blast radius of that automation is the thing to look at this week.

What actually happened

Per BleepingComputer's reporting, the incident (tracked as MO1437424) began at 10:44 AM ET on July 23 and wasn't fully resolved until mid-afternoon — a rollback started at 1:45 PM ET and completed at 2:26 PM ET, with services fully recovered by 3:41 PM. Customers routed through Microsoft's West US Azure region took the brunt of it. SharePoint Online accounted for the majority of complaints, with OneDrive, Teams, the Microsoft 365 Admin Center, Power Automate, Copilot Chat, Loop, Power BI, Windows 365, and Defender all degraded.

Microsoft's explanation is the interesting part: a bug in its automated maintenance request conversion system "incorrectly marked additional network devices as part of the maintenance event." The system did exactly what it was told. It was told the wrong scope. IP routes came off more devices than intended, and traffic between the West US datacenter and the wide-area network stopped flowing.

Why blast radius matters for your business

There are two lessons here and most write-ups only cover the first one.

The first is dependency. You cannot patch Microsoft's automation. What you can do is know, in advance, which four hours of your operation die with SharePoint — and have the paper-thin fallback ready. For most of our clients that means a phone number that isn't Teams, an order queue that isn't a SharePoint list, and someone who knows where the export lives.

The second is the one that should sting a little. This is your bug too. Every deploy script, every scheduled job, every cleanup task, every agent you've given write access is a system that computes a scope and then acts on it. Microsoft's converter was correct in every step except which machines counted. That failure mode doesn't care how big your company is.

So scope defensively. Have the automation print what it's about to touch and hard-fail when the count exceeds a sane ceiling — a job that expects nine devices and finds nine hundred should stop, not proceed. Make changes reversible before you make them fast; Microsoft's saving grace was a rollback that took 41 minutes, not a heroic debugging session. Stage by blast radius, not by convenience. And log the computed target list, because "what did it actually select?" is the only question that matters at 11 AM on a Thursday.

Sources: BleepingComputer, Microsoft 365 Status.