There is a distinction underneath all of reliability engineering, and it is worth being pedantic about because everything else follows from it.
The idea
A fault is one component deviating from its spec. A failure is the system as a whole stopping doing what the user needed. Faults are inevitable. Failures are optional.
A disk that returns corrupt bytes is a fault. A user who cannot load their invoice is a failure. The entire discipline consists of building things where the first does not turn into the second.
This reframes the goal in a way that is much more useful than “make it reliable.” You are not trying to prevent faults — you cannot; the universe is not cooperating. You are trying to break the causal chain from fault to failure.
Three kinds of fault, and why they behave differently
Hardware faults
Disks die. Memory flips a bit. A backhoe finds a fibre line. For a long time this was the whole story, and the answer was redundancy: RAID arrays, dual power supplies, hot-swappable everything.
The useful property of hardware faults is that they are mostly independent. One machine’s disk dying tells you almost nothing about whether another machine’s disk is about to die. That independence is what makes redundancy work — two disks with a 1-in-1000 annual failure rate fail together far less often than one does alone.
Software faults
A software fault is a bug that lies in wait for a specific condition: the leap second, the 2GB file, the input containing an emoji, the cache that fills at exactly the wrong rate.
These are far nastier than hardware faults for one reason:
The dangerous property
Software faults are systematic and correlated. The same bug is on every one of your nodes, waiting for the same trigger. Redundancy does not help — you have made three identical copies of the same mistake.
This is why adding a second server does nothing for a null-pointer exception in your request handler, and why the fixes for software faults look completely different: careful thinking about assumptions, tests that cover the boundaries, process isolation, letting processes crash and restart, and measuring things in production so a violated assumption is visible before it is fatal.
Human faults
Configuration changes made by operators are, in most surveyed systems, the single largest cause of outages — well ahead of hardware.
The unproductive response is to demand that humans be more careful. Humans are not more careful; they are tired, and it is 4pm on a Friday. The productive responses design around it:
- Make the dangerous thing hard and the safe thing easy. A staging environment that is genuinely like production. A migration tool that refuses to run without a dry run first.
- Decouple where mistakes happen from where they hurt. A sandbox with real data and no ability to affect real users.
- Make recovery fast. Fast rollback beats careful deployment, because you will need rollback either way and the careful deployment did not prevent it.
- Measure. If the error rate is on a screen someone looks at, a bad deploy is a five-minute incident instead of a five-hour one.
Causing faults on purpose
If failures come from untested paths, then the only way to know your fault tolerance works is to use it. Continuously.
That is what chaos engineering actually is, stripped of the branding: killing a random production node during working hours, on purpose, so that the code path which handles a dead node is executed every day rather than for the first time during a real incident at 3am.
The idea
A fault-tolerance mechanism that is never exercised does not exist. It is untested code that you believe works, which is the most expensive kind.
Note the “during working hours” part. The point is not to prove the system survives; it is to be watching when it doesn’t.
When reliability is not worth it
Reliability costs real money and real engineering time, and there are honest cases for buying less of it: a prototype for ten internal users, a free tier where the deal is explicitly best-effort, a scraper that can simply run again tomorrow.
The trap is that this trade-off is usually made implicitly, by not thinking about it, and then discovered later at the worst moment. Making it explicitly — we accept up to four hours of downtime a month on this service, and here is what we would have spent to avoid it — is a completely different thing from having drifted into it.
Check yourself
What to take away
Faults are the input; failures are the output; your design is the function in between. Hardware faults are independent and yield to redundancy. Software faults are correlated and do not. Human faults are the most common of all, and yield only to systems that assume humans will make them.