Microsoft outage: A cascading failure that started with an undetected bug
Following a roughly day-long global outage of many Microsoft services last week, Microsoft on Monday issued a preliminary post-incident report that identified multiple causes for the issues, including the admission that the environment used to test the underlying problematic software update did not match the actual infrastructure it would run on. The report, a summary of which was posted on social media by Microsoft EVP Charles Lamanna, identified some of the factors that cascaded to cause the outage. “A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions,” the report said, adding, “this defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers.” Its deployment began in July, and, the report said, “for more than three months, the change was deployed worldwide without impact.” Microsoft admitted that, although it did test the change before deployment, its test environment didn’t match reality. “Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs,” it said. “The contention issue is worsened on machines with more cores and …