Fault protection for AI agents: lessons from machines you cannot reboot

When Richard Feynman investigated the Challenger disaster, he ended his appendix to the Rogers Commission report with a sentence that belongs on every engineer's wall: "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled."

Feynman was a Caltech physicist, and Caltech manages the Jet Propulsion Laboratory — the place that lands machines on Mars that no engineer will ever touch again. When you cannot send a technician, cannot reboot by hand, and cannot even talk to the machine for twenty minutes each way, you stop designing for success and start designing for the run that goes wrong. Production AI needs exactly that inversion.

Success-oriented design is how pilots die

Most AI systems are built success-first: assume the model returns something sensible, the connector responds, the data is clean, and the happy path holds. Demos reward this, because demos are the happy path by definition.

JPL cannot afford the assumption. A spacecraft's software is built around a blunt premise: components will fail, inputs will be corrupt, and the unexpected will happen — so what does the system do then? The discipline of answering that question systematically is called fault protection, and it is the difference between a probe that survives an anomaly and one that tumbles silently into the dark.

Your agent, chasing invoices or drafting quotes at 3 a.m. with no human watching, is in the same situation more often than you think.

First principle: assume the run will fail, then bound the damage

Fault protection does not try to prevent every fault — that is impossible. It tries to detect faults quickly and contain their consequences. Reframed for AI, the design questions become concrete:

• What happens when the model hallucinates a value that passes basic format checks?
• What happens when a connector returns stale or partial data?
• What happens when an eval times out, or a downstream system is unreachable mid-workflow?

If your only answer is "it probably won't," you have a demo, not a system.

Safe mode: the most important state your agent has

When a spacecraft detects something it cannot handle, it enters safe mode: it stops non-essential activity, puts itself in a known stable configuration, points its antenna at Earth, and waits for instructions. It does the minimum, preserves itself, and phones home.

Every serious AI workflow needs a safe mode, and it is the same three moves: halt the risky action, preserve full context (inputs, intermediate state, the exact model output), and escalate to a human with everything needed to decide. An agent that is unsure should never improvise an action against the world. It should degrade to safe mode — which, not coincidentally, is what a well-designed approval gate delivers automatically.

Fault containment regions

JPL partitions spacecraft into fault containment regions so that a failure in one subsystem cannot cascade into others. The operating-layer equivalent is isolation between workflows: a runaway retry loop, a poisoned input, or a misbehaving connector in one workflow must not be able to exhaust shared resources or corrupt the state of unrelated ones. Bulkheads, per-workflow budgets, and hard timeouts are your containment regions.

Feynman's estimate: know your real failure rate

Feynman's most damning finding was cultural, not technical. Management claimed the Shuttle's failure probability was roughly 1 in 100,000. The working engineers estimated closer to 1 in 100. The gap was not physics; it was optimism reported upward as fact.

The AI equivalent is everywhere. Teams cite the polished demo as the system's reliability. But nature cannot be fooled: what is your measured hallucination rate on real inputs? Your eval failure rate at production volume? Your connector error rate under load? If you cannot answer with numbers from real runs, you are managing to the 1-in-100,000 story while operating a 1-in-100 system.

A footnote from information theory

Claude Shannon proved that you can achieve reliable communication over an unreliable channel by adding redundancy — encoding information so errors can be detected and corrected. The same logic applies to unreliable model outputs: cross-check a critical value against a second source, validate against known constraints, or require agreement between independent checks before acting. Reliability is bought with redundancy; there is no free version.

What this means for the operating layer

At PromptRails the design starts from the assumption that runs fail. Guardrails catch bad outputs before they act. Workflows have a safe mode — halt, preserve, escalate — instead of a blind happy path. Runs are isolated so one failure cannot cascade. Full tracing means that when something goes wrong we can see exactly what happened, which is JPL's other rule: an anomaly you cannot reconstruct is one you cannot fix.

You will not send your agents to Mars. But like a spacecraft, they act while no one is watching, on systems that matter, where a silent failure is the expensive kind. Design for the failure you cannot prevent. Nature cannot be fooled.

References

1. Richard P. Feynman, Personal Observations on the Reliability of the Shuttle, Appendix F, Report of the Presidential Commission on the Space Shuttle Challenger Accident, 1986.
2. Robert D. Rasmussen, Goal-Based Fault Tolerance for Space Systems Using the Mission Data System, Proceedings of the IEEE Aerospace Conference, 2001.
3. Claude E. Shannon, A Mathematical Theory of Communication, Bell System Technical Journal, 27, 1948.
4. Nancy G. Leveson, Engineering a Safer World: Systems Thinking Applied to Safety, MIT Press, 2011.

---

Want to pressure-test how your workflow behaves when a run goes wrong? Book a 30-minute call.