7.2 C
New York
Tuesday, November 17, 2026

Microsoft outage traced to July software defect

Must read

Trends News Stories Editorial
Trends News Stories Editorialhttps://trendsnewsstories.com/
Editorial desk at Trends News Stories (trendsnewsstories.com), an independent news site based in New Delhi, India. We publish original trending-news explainers across entertainment, sports, tech, business and health. Facts are checked against official sources and trusted outlets; we credit sources, note what is unconfirmed, and correct errors when we find them.

Microsoft has identified a months-old software defect as a central cause of the global outage that shut Outlook, Exchange and other Microsoft 365 services for about a day last week. The company issued an initial report on Monday that named three factors behind the failure: a latent defect in its code, a surge in traffic and reduced server capacity. The Microsoft outage began on Thursday, 12 November, and Microsoft said service was restored by Saturday.

Charles Lamanna, executive vice president of Copilot, Agents, and Platform at Microsoft, set out the findings in a post on X on Monday. He said a latent bug or defect from July caused the Microsoft outage by making Microsoft's servers become overwhelmed and unresponsive. The company said the blackout was not the result of a cybersecurity breach, and it issued an apology to customers. Microsoft plans to publish a final report with more detail later this week.

The July defect behind the Microsoft outage

The defect behind the Microsoft outage was the result of a fix prepared specifically for Microsoft 365 networking gateway servers, and it did not apply to broader Windows deployments. Deployment of the change began in July, and it spread around the world over several weeks. Microsoft said that for more than three months the change was deployed worldwide without impact. The problem was linked to lock contention inside an internal process, which made machines unavailable under certain load conditions.

Microsoft said its normal load testing did not detect the latent issue because the traffic and the mix of server types used in its test labs differed from production. The contention issue was worse on machines with more cores and newer hardware, while load testing tended to focus on weaker hardware with fewer cores. The company admitted that the environment used to test the change did not match the actual infrastructure it would run on. That gap in testing is one reason the Microsoft outage went undetected for months.

How the Microsoft outage spread

About 15 hours before the Microsoft outage, Teams failed over from an alternative networking gateway to the primary gateway that also supports Exchange Online. That increased the load on the primary gateway and changed its traffic patterns. At the same time, nearly 14 percent of server capacity for the networking gateway was out of service because of an unrelated issue from the previous day, and Microsoft did not give details of that issue.

Microsoft said the increased load and the reduced server capacity caused the Microsoft outage, as the network gateway encountered the latent bug from July. Requests built up until servers stopped responding, and traffic management systems routed users to other endpoints, which made the problem more likely to occur on the remaining working servers. Microsoft said this caused a cascading failure from server to server, as the same amount of load was spread across fewer servers. In less than two hours the defect was encountered across the fleet, and at its lowest point more than 80 percent of servers were unresponsive.

Why recovery from the outage took so long

Recovery from the Microsoft outage took longer than usual because servers brought back online were at once overwhelmed by waiting traffic, which triggered the bug again and made them unresponsive. Microsoft said its normally automated recovery processes were not designed to restore service from such a broad unresponsive state. Since the main code defect was a latent bug from several months earlier, the company's normal rollback, flighting and feature flags were unable to quickly revert the change. Engineers then developed an update that returned the code path to the previously known good state.

Microsoft Threat Intelligence also detected a credential phishing campaign that used the disruption as cover. Attackers sent emails disguised as Microsoft 365 service health notices, referring to incident identifier EX1550543, and falsely warned recipients that held emails would be deleted within 24 hours unless they clicked a restoration link. The final post-incident review is expected to give more detail on the causes and on the steps Microsoft will take to stop a similar failure.

More articles

Latest article