Chaos Engineering

What it is

The discipline of running controlled experiments on a system by deliberately injecting failures (killing instances, adding network latency, severing dependencies, exhausting resources) to verify that the system tolerates them as designed. Popularized by Netflix's Chaos Monkey, it treats resilience as a hypothesis to be tested, not assumed.

The problem it solves

Resilience features (failover, retries, circuit breakers, redundancy) are usually built and then never exercised until a real incident, when they often turn out to be broken. Untested failure paths are a false sense of safety. Chaos engineering surfaces these weaknesses on your schedule, under observation, instead of during a 3am outage when the blast radius and stress are maximal.

How it works

Form a hypothesis about steady-state behavior ('p99 latency stays under 200ms and error rate under 0.1%'). Define a small blast radius. Inject a real-world fault: terminate an instance, add latency to a dependency, drop a network link, fill a disk. Observe whether steady state holds. If it breaks, you've found a weakness to fix; if it holds, confidence increases. Mature programs run continuously and automatically (Chaos Monkey randomly kills production instances) with automated guardrails to halt the experiment if it goes too far.

Why use it

• Finds resilience gaps on your schedule instead of during a real outage
• Continuously validates that failover, retries, and redundancy actually work as designed
• Builds organizational confidence and forces good observability and automated recovery

What it costs you

• Running it in production is genuinely risky: a poorly-scoped experiment can cause the outage it was meant to prevent
• Requires solid observability and automated rollback first, because chaos without monitoring is just sabotage
• Cultural and organizational buy-in is hard; deliberately breaking production is a tough sell without trust

Where it shows up in our architectures

• Netflix →

  The origin of Chaos Monkey, which randomly terminates production instances so engineers are forced to build services that survive node loss

• Uber →

  Injecting latency and killing service instances validates that dispatch degrades gracefully and circuit breakers trip during a regional surge

• Payment Gateway →

  Simulating Stripe/card-network slowness verifies the circuit breakers and idempotent retries hold without double-charging

Gotchas

• Don't run chaos in production until you have the observability and automated rollback to detect and stop a runaway experiment; otherwise you're just causing outages.
• Start with the smallest blast radius (one instance, one dependency, off-peak) and expand as confidence grows. Game days in staging first, then carefully into prod.
• Always have an abort button and automated guardrails that halt the experiment if steady-state SLOs are violated. A chaos experiment that you can't stop is an incident.
• Chaos engineering tests resilience you've already built; it's not a substitute for designing for failure (redundancy, failover, idempotency). It validates, it doesn't create.