The CrowdStrike outage was not an attack. Plan for it anyway.

About this series

The day the world stopped

On July 19, 2024, a Friday morning for many, millions of Windows computers displayed a blue screen and stopped working. This was not the work of a sophisticated hacker group or a foreign government. It was caused by a faulty content update for the CrowdStrike Falcon sensor, a security tool used by businesses to watch for threats (show definition).

Microsoft estimated that the update affected roughly 8.5 million Windows devices globally. While that number is less than one percent of all Windows machines, the affected systems were concentrated in the most critical parts of the economy. Airlines grounded flights, hospitals postponed surgeries, and broadcasters went off the air. Even small businesses that didn't use the software directly felt the impact as their vendors and partners went offline.

Why a single file caused such damage

The crash happened because of a logic error in a file named Channel File 291. This file controls how the Falcon sensor evaluates certain system activities. Because Endpoint detection and response (EDR) (show definition) software needs deep access to the operating system to stop attacks, a bug in that software can cause the entire system to crash. In this case, the Windows kernel—the core of the operating system—could not handle the error, leading to the Blue Screen of Death (BSOD).

The recovery was slow because the machines were stuck in a boot loop. They would crash before they could connect to the internet to download a fix. This meant IT teams had to manually intervene, often machine by machine. They had to boot into Safe Mode or the Windows Recovery Environment and delete the specific faulty file. For companies with thousands of remote laptops or servers in locked data centers, this was a massive undertaking that took days to resolve.

Availability is part of security

In the security world, we often talk about the CIA triad: Confidentiality, Integrity, and Availability. The CrowdStrike incident was a failure of availability. It proved that a system you cannot use is just as useless as a system that has been breached. If your business depends on a single piece of software to function, you have a single point of failure.

Many businesses assume that their security tools are infallible. But every tool is a piece of software, and software can have bugs. Relying on one vendor for your entire security stack—or your entire productivity stack—creates concentration risk. If that vendor has a bad day, you have a bad day too.

Planning for the next failure

You cannot prevent every vendor failure, but you can plan for how you will react. A Incident response plan (show definition) should not just cover what to do after a hack. It needs to cover what to do when your critical tools go dark. This is the difference between simple backups and true business continuity.

• Identify your most critical systems. Which ones would stop your business in an hour? Which ones could wait a day?
• Use update rings. Do not let every machine update at the exact same time. Patch a small group of test machines first, then a larger group, and then the rest of the company.
• Keep a hard copy of your recovery plan. If your computers are down, you cannot read a PDF stored on your server.
• Diversify where it makes sense. If you use one cloud for storage, maybe use another for your essential communications.

Steps to build resilience

• Create a list of all critical third-party software and their support contacts
• Document the manual recovery steps for your most important servers
• Configure software updates to roll out in stages rather than all at once
• Verify that IT staff have the local Administrator access (show definition) and keys needed to fix machines manually
• Schedule a 'fire drill' to see how long it takes to restore one critical system without using your main management tools

Beyond the recovery

The lessons of July 2024 are still relevant two years later. The world is more connected now, and we rely even more on automated systems. Automation saves time, but it can also spread a mistake at lightning speed. Real resilience comes from having the people and the plans to step in when the automation fails.