Network security company CrowdStrike pushed a minor update to its Falcon Sensor product around 11pm Chicago time yesterday that managed to take down almost every virtual machine in Microsoft's Azure cloud:
Cascading technology errors stranded airline passengers around the world, halted hospital surgeries and crippled office workers’ computers on Friday in one of the most disruptive computer outages in years, highlighting how much of the world relies on potentially error-prone software from a handful of companies.
Technology experts said the meltdowns appeared to stem mostly from an error in a software update from CrowdStrike, whose technology is commonly used by businesses to defend against cyberattacks.
That defect affected computers that use Microsoft’s Windows, which powers hundreds of millions of personal computers and many back-end systems for airlines, digital payment, emergency services call centers and much more.
[B]ecause CrowdStrike’s digital protections are considered essential, its technology is given priority access on many computer systems. If something goes wrong with CrowdStrike software, that privileged access can grind computers to a halt.
CrowdStrike admitted that their software caused the problem:
- Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
- Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
- Windows hosts which are brought online after 0527 UTC will also not be impacted
- Hosts running Windows 7/2008 R2 are not impacted
- This issue is not impacting Mac- or Linux-based hosts
- Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
- Channel file "C-00000291*.sys" with timestamp of 0409 UTC is the problematic version.
Don't worry, you probably don't have CrowdStrike software on your PC at home; but you probably do log into your Windows PC through Microsoft Active Directory, which runs on virtual machines in the Azure cloud that depend on Falcon Sensor.
This time, the random person in Nebraska turned out to be a multimillion-dollar corporation in Austin, Texas. Though, I suspect, several random people in Texas are now looking for new jobs.