Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

The CrowdStrike outage explained: Channel File 291

The CrowdStrike outage of July 19, 2024 crashed 8.5 million Windows PCs. What Channel File 291 did, why it got past testing, and what staged rollouts change.

The CrowdStrike outage explained: Channel File 291

The CrowdStrike outage happened on July 19, 2024, when a content update for the Falcon sensor on Windows made every machine that received it crash with a blue screen (BSOD). Microsoft estimated that 8.5 million Windows devices were affected. The bad file was live for 78 minutes, but because the crash happened in a kernel driver at boot, many machines needed someone to fix them by hand.

It was not a cyberattack, and the update was not new code. It was a configuration file, pushed to the whole fleet at once. That last detail is why it became the reference incident for staged rollouts.

What happened on July 19, 2024?

CrowdStrike’s technical details post gives the timeline. At 04:09 UTC it released a sensor configuration update to Windows systems. The update was remediated at 05:27 UTC. Windows hosts running Falcon sensor 7.11 or later that were online and downloaded the update in that window were at risk. Linux and macOS didn’t use the file and weren’t affected.

The update was a new version of Channel File 291, the files named C-00000291*.sys in C:\Windows\System32\drivers\CrowdStrike\. Despite the .sys extension, CrowdStrike says channel files are not kernel drivers; they are data the driver reads. Channel File 291 controls how Falcon evaluates named pipe execution on Windows, and this update targeted malicious named pipes used by common command-and-control frameworks.

The next day, Microsoft’s David Weston wrote that “CrowdStrike’s update affected 8.5 million Windows devices, or less than one percent of all Windows machines.” Less than one percent sounds small; it was concentrated in organizations that run enterprise endpoint security, which is why so many businesses stopped at once.

Why did a config file crash Windows?

The External Technical Root Cause Analysis (August 6, 2024) explains the mechanism. Some vocabulary first, in CrowdStrike’s terms:

  • Template Type: a detection capability compiled into the sensor, with a fixed set of input fields.
  • Template Instance: matching criteria (regexes) that detection engineers write against a Template Type.
  • Rapid Response Content: a bundle of Template Instances, shipped as a channel file, so the sensor can detect new behavior without a sensor code change.
  • Content Interpreter: the C++ code in the sensor that tests inputs against those regexes.
  • Content Validator: the check that runs on channel files before they ship.

Sensor 7.11, released in February 2024, added a new IPC Template Type for named pipes and other Windows interprocess communication. Its definition declared 21 input fields. The sensor code that called the Content Interpreter for it supplied only 20.

Nothing failed for months, because every Template Instance used a wildcard for the 21st field, so the interpreter never had to read it. The preliminary review lists the history: the IPC Template Type passed a stress test and the first instance went to production on March 5, 2024, and three more instances shipped between April 8 and April 24 and “performed as expected in production.”

On July 19, two more instances shipped, and one of them used a non-wildcard criterion on the 21st field. The Content Validator passed it, because it checked against the definition, which said 21 inputs. On the next IPC notification from the OS, the interpreter tried to read the 21st value from a 20-element array, an out-of-bounds read in a kernel-mode driver, and Windows crashed.

The RCA’s summary is that it took a “confluence” of three issues: the 21-versus-20 mismatch, a latent out-of-bounds read in the interpreter, and no test with a non-wildcard value in the 21st field. Each was survivable alone.

Why was recovery so slow?

The RCA says CrowdStrike’s driver loads “from an early phase of system boot”, so a machine still holding the bad file could crash again on restart before it was online long enough to fetch the fixed one. CrowdStrike’s remediation hub told admins to try rebooting on a wired network so the host could download the reverted file in time. For cloud and virtual machines, the fix was to detach the OS disk, locate the files matching C-00000291*.sys, delete them and reattach. The same page warns that “Bitlocker-encrypted hosts may require a recovery key”, which turns a quick fix into a hunt for keys per machine.

Rolling forward took 78 minutes. Rolling back the damage took days, because the failure removed the remote management path that would normally deliver the fix.

What would a staged rollout have changed?

The sixth finding in the RCA is one sentence: “Each Template Instance should be deployed in a staged rollout.” The mitigation describes the standard pattern: content that passes canary testing is “successively promoted to wider deployment rings or rolled back if problems are detected”, with bake-in time at each ring while telemetry comes in. CrowdStrike also gave customers control over where and when Rapid Response Content is deployed.

Here is a simulation of what that buys. A fleet of 100,000 hosts gets a bad update. Each host either triggers the bug or doesn’t. In the staged version the update goes to rings covering 1%, 10%, 50% and then 100% of the fleet, and after each ring a health gate halts the rollout if more than 0.1% of updated hosts crash. The fleet, rings and threshold are made up for illustration.

# rollout.py: blast radius of a bad update, all at once vs. staged rings with a health gate.
# Python 3 stdlib only. A host "crashes" if it gets the update and is one of the hosts the bug hits.
import random

FLEET = 100_000
RINGS = [("canary", 0.01), ("early", 0.10), ("broad", 0.50), ("all", 1.00)]  # cumulative share
GATE = 0.001  # halt if more than 0.1% of the hosts updated so far crash during bake time

def rollout(bug_rate, rings, seed=7):
    rng = random.Random(seed)
    hosts = list(range(FLEET)); rng.shuffle(hosts)
    hit = {h for h in hosts if rng.random() < bug_rate}  # hosts whose config triggers the bug
    done = 0
    for name, share in rings:
        upto = int(FLEET * share)
        done = upto
        crashed = sum(1 for h in hosts[:upto] if h in hit)
        if crashed / upto > GATE:
            return f"halted after {name:<6} updated {upto:>6}  crashed {crashed:>6}"
    return f"completed          updated {done:>6}  crashed {crashed:>6}"

for bug_rate, label in [(1.0, "bug hits every host"), (0.02, "bug hits 2% of hosts"),
                        (0.0005, "bug hits 0.05% of hosts")]:
    print(label)
    print("  all at once:", rollout(bug_rate, [("fleet", 1.0)]))
    print("  staged     :", rollout(bug_rate, RINGS))

assert "canary" in rollout(1.0, RINGS) and "completed" in rollout(0.0, RINGS)

Output (Python 3.13.11):

bug hits every host
  all at once: halted after fleet  updated 100000  crashed 100000
  staged     : halted after canary updated   1000  crashed   1000
bug hits 2% of hosts
  all at once: halted after fleet  updated 100000  crashed   2013
  staged     : halted after canary updated   1000  crashed     21
bug hits 0.05% of hosts
  all at once: completed          updated 100000  crashed     46
  staged     : completed          updated 100000  crashed     46

The first case is July 19: a bug that every Windows host running the affected sensor would hit. All at once, the whole fleet goes down before anyone can react. Staged, the canary ring takes the hit and the gate stops it at 1% of the fleet. A canary doesn’t need to be clever to catch a bug that crashes everything; it only needs to exist and to be waited on.

The second case shows the gate working on a partial bug: about 21 crashes instead of about 2,013. The third case is the honest limit. A bug that hits 0.05% of hosts is under the 0.1% threshold, so the gate passes it and staging changes nothing. Rare failures need either a tighter gate, a longer bake, or a canary ring chosen to cover unusual configurations, which costs time on every release.

Why wasn’t content staged already?

Because speed is the product. The RCA describes Rapid Response Content as the way the sensor responds to new threats “at operational speed” without a sensor release. Each sensor release goes through Microsoft’s WHQL certification; Rapid Response Content is, in the RCA’s words, “configuration data; it is not code or a kernel driver”, and ships between releases. A security vendor that waits a day to push a detection for a new attack is a day behind the attacker.

That trade-off is real, and it’s not unique to security. Feature flags, routing rules, rate-limit configs and model routing tables are all “just config”, and all can take down a fleet. The RCA’s mitigations amount to treating content like code: validate field counts at compile time, add a runtime bounds check, test every field with non-wildcard values, test every instance in the interpreter, and stage the deploy. The bounds check alone (added July 25, 2024, per the RCA) would have turned a kernel crash into a failed match.

What I would do

My lens is the data platform side, where the dangerous deploys were schema changes and pipeline configs, not binaries. The lessons transfer directly.

  1. Rings for everything that reaches production, including config. If a file can change runtime behavior, it rolls out in stages with a bake time and an automatic halt. “It’s only data” is the sentence that precedes most of these incidents.
  2. Make the gate measure the failure you fear. A crash, a spike in error rate, or hosts going silent all count. Silence is a signal: a crashed host stops sending telemetry. Wire that into OpenTelemetry metrics the gate reads, not a dashboard a person has to watch.
  3. Validate against the consumer, not the spec. The Content Validator checked the file against the definition. It never ran it through the code that would consume it. Contract tests should exercise the real reader.
  4. Fail closed in the reader. A parser in a privileged process that sees an out-of-range field should reject the record, not dereference it.
  5. Keep a path to recover that doesn’t depend on the thing that broke. If a bad update can stop the machine from booting far enough to receive the fix, plan the out-of-band recovery before you need it.

A platform team is the right owner for ring definitions and gates, so every service gets them by default. The same discipline applies to what you ship from upstream: an SBOM tells you what changed in a release, and the xz-utils backdoor is a reminder that trusted updates are still updates.

FAQ

What caused the CrowdStrike outage?

A Rapid Response Content update in Channel File 291 asked the Falcon sensor’s Content Interpreter to read a 21st input field when the sensor supplied only 20. The out-of-bounds read in the kernel driver crashed Windows. The update was released at 04:09 UTC on July 19, 2024 and remediated at 05:27 UTC.

How many computers did the CrowdStrike outage affect?

Microsoft estimated 8.5 million Windows devices, less than one percent of all Windows machines.

Was the CrowdStrike outage a cyberattack?

No. CrowdStrike’s CEO George Kurtz stated on July 19, 2024: “This was not a cyberattack.”

Why were Mac and Linux machines not affected?

Linux and macOS sensors don’t use Channel File 291, according to CrowdStrike’s technical details post.

What is a staged rollout?

Deploying a change to a small group first (the canary), waiting while health signals come in, then promoting it to progressively larger rings, and halting or rolling back automatically if the signals go bad. CrowdStrike added this for Template Instances after the incident.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.