You will not believe the trick-or-treat trick Microsoft just pulled!

[rysiek@mstdn.social]

> We have confirmed that an inadvertent configuration change was the trigger event for this issue.

Folks, it's all okay! It was just an inadvertent configuration change [that happened to bring loads of services for millions of users down]. No biggie.

#Azure #Microsoft #Outage

[rysiek@mstdn.social]

Look, this is the level of professionalism we should expect of Microsoft at this point.

They had a separate global outage of Office 365 (or whatever it's called this week) *this month* already:
https://www.bleepingcomputer.com/news/microsoft/microsoft-365-outage-blocks-access-to-teams-exchange-online/

And then there was the one from February this year:
https://www.upi.com/Top_News/US/2025/03/01/microsoft-outages/3061740867107/

Not to mention smaller ones, like the one in US-East that prevented admins from accessing the admin center:
https://www.bleepingcomputer.com/news/microsoft/microsoft-investigates-outage-affecting-microsoft-365-admin-center/

#Microsoft #Azure #Outage

[philip@mastodon.mallegolhansen.com]

@rysiek Next time it’ll be an advertent change, just to keep it fresh.

[rysiek@mstdn.social]

I wonder if firing 15.000+ employees just this year had any impact here.
https://www.windowscentral.com/microsoft/microsoft-continues-layoffs-with-42-more-roles-cut-for-the-fifth-month-in-a-row

#Microsoft #Azure #Outage

[ericlawton@kolektiva.social]

@rysiek

"or whatever it's called this week":

They should at least reduce the "365" by one, for every outage.

@Szescstopni

[msabourin@pouet.chapril.org]

@rysiek @mms who knew multi-cloud meant down when any cloud is down.

[xinit@mastodon.coffee]

@pfm @rysiek If your downtime isn't from the DNS region, it's just sparkling latency.

[rysiek@mstdn.social]

While Azure was down and you could not use your Office 365 for 8 (eight) hours yesterday, Microsoft gloated about record profits:
https://www.theguardian.com/technology/2025/oct/29/microsoft-earnings-azure-outage-xbox

After all, Office 365 price has increased at least twice over the last 12 months:
https://www.tomsguide.com/computing/microsoft-365-gets-massive-45-percent-price-hike-and-its-all-to-do-with-ai-tools
https://techcommunity.microsoft.com/blog/publicsectorblog/upcoming-changes-office-365-g1-price-increase-effective-march-2025/4385970

Price hikes were due to "AI", obviously. It's what users crave.

Your Microsoft Tax dollars at work!

#Microsoft #Azure #Outage

[rysiek@mstdn.social]

Two price hikes, three major outages…

Microsoft will have to hike the price again just to keep the ratio at 1:1!

[rysiek@mstdn.social]

This might be a good time to consider deploying, contributing to or otherwise supporting @nextcloud:
https://nextcloud.com/

And before anyone says that Nextcloud's UI/UX is lacking: of course it is! Nextcloud has several orders of magnitude less money to throw at UI/UX.

But guess what:
1. this is fixable if they get more resources to work with;
2. every single Nextcloud instance I know of or use (there are many) stayed up and running yesterday.

#Microsoft #Azure #Outage

[rysiek@mstdn.social]

Cascading failure. It cascaded.

> As unhealthy nodes dropped out of the global pool, traffic distribution across healthy nodes became imbalanced, amplifying the impact and causing intermittent availability even for regions that were partially healthy.
https://azure.status.microsoft/en-us/status/history/#incident-history-collapse-YKYN-BWZ

#Microsoft #Azure #Outage

[rysiek@mstdn.social]

> An inadvertent tenant configuration change within Azure Front Door (AFD) triggered a widespread service disruption…

> The trigger was traced to a faulty tenant configuration deployment process. Our protection mechanisms, to validate and block any erroneous deployments, failed due to a software defect which allowed the deployment to bypass safety validations.

Am I reading this right? Global – pardon, "non-regional" – #outage of #Microsoft #Azure was caused by a tenant changing their config? 👀

[rysiek@mstdn.social]

Can't have a snow day without cloud. 🌨️

[rysiek@mstdn.social]

These massive "cloud" outages will continue to happen, inevitably, because they are "normal accidents":
https://en.wikipedia.org/wiki/Normal_Accidents

Any system that:
- is complex,
- is tightly coupled, and
- has catastrophic potential

…can be expected to experience catastrophic failures. AWS, Google Cloud, #Microsoft #Azure check all these boxes.

Fun fact: Internet was specifically designed to *not* be like that. It was designed to be loosely coupled.

But that's not as good for maximizing shareholder value!

[spytfyre@mastodon.scot]

@rysiek software defect
I think it was any tenant change by my reading not one single idiot. They're not pinning this on tenants users but it is written that way so you can misinterpret it that way and blame Microsoft less
Software defect is there to point at that they didn't blame anyone they blamed the software
They're shying away from blaming skynet so it doesn't fire them next

[svelmoe@hachyderm.io]

@rysiek I know I'm only an average 'in the trench' type developer, and my interests have always been code over servers/hosting........

But I cannot express how much I miss the "old days" of normal web-servers and where focus was on delivery of (customer/user) value and optimisations and not just bloating everything with external services sold into a cloak of 'critical infrastructure' and 'pass/sass' cloud stuff.

[mkj@social.mkj.earth]

No matter how you slice it, as I see it, it boils down to: Microsoft's software did insufficient input validation on untrusted data (specifically in this case configuration changes for one customer, it seems), and this led to widespread outages affecting many, many, *many* customers in many different ways.

Security done properly should not rely on "blocking erroneous" stuff, but should verify that everything is good and only *if* that succeeds allow any of it through.

@spytfyre @rysiek