Skip to content
Software Architecture Guide

Fault Tolerance

Explanation

Section titled “Explanation”

ISO/IEC 25010 defines fault tolerance as the degree to which a system keeps operating as intended despite hardware or software faults (see ISO/IEC 25010 Product Quality Model). The aim is that one broken part does not break the whole service.

Two neighboring qualities are separate in the same model. Availability is how far the system is operational and accessible when someone needs it. Recoverability is the ability to recover the data directly affected after an interruption or failure. Fault tolerance is about continuing to work while the fault is present.

Teams combine several tactics. A tactic is a proven design move (see Architectural Tactic).

Detecting faults

  • Heartbeat: nodes exchange periodic messages, so a missing message signals trouble.
  • Ping/echo: one node sends an asynchronous request and expects a reply from another node.
  • Exception detection: the system notices a condition that alters normal execution. The exception must then be handled.

Redundancy and recovery

  • Active redundancy (hot spare): two or more elements process the same inputs in parallel.
  • Passive (warm) spare: a standby that receives periodic state updates from the active nodes.
  • Cold spare: a standby that takes over without the failed node’s state.
  • Rollback: return the system to a previous known good state.
  • Retry: repeat an action after a suitable delay. This suits transient faults, which usually correct themselves. It is only safe for idempotent operations. Otherwise a repeat may run the operation twice, with unintended side effects.

Containing faults

  • Circuit breaker: wraps a protected call. After a failure threshold, it rejects further calls without making them. This prevents cascading failure. Retry expects eventual success, while the breaker stops operations likely to fail.
  • Bulkhead: isolates parts of an application into pools, so one failing part leaves the others working.

Graceful degradation means a failed component leads to reduced-mode operation, not complete failure. A circuit breaker can trigger it.

To test fault tolerance, run chaos experiments. Each one targets specific faults and has a clear hypothesis, derived from failure mode analysis. Also test that the workload degrades gracefully.

Example

Section titled “Example”

An online shop calls a separate recommendation service. That service starts failing. The shop retries a read-only request once after a short delay. After repeated failures, a circuit breaker opens and the shop stops calling the service. Product pages still load, without recommendations. Checkout runs in its own pool, so it is not slowed down. The shop works in a reduced mode instead of failing completely.

Trade-offs

Section titled “Trade-offs”

The facts here show some limits. Retry is unsafe for operations that are not idempotent. Retry and circuit breaker pull in different directions: one expects success, the other stops calls. Choose each tactic for the fault type you expect. Spares also differ in how much state they keep. A cold spare takes over without the failed node’s state, while a passive spare receives periodic updates. Compare such options against other qualities (see Quality Attribute Trade-off).

A circuit breaker stops calls to a failing service and returns a reduced-mode response.

Diagram source 
flowchart LR
  C[Caller] --> B{Circuit breaker}
  B -->|closed| S[Remote service]
  B -->|open| F[Reduced-mode response]
  S -->|repeated failures| B

Checks

Section titled “Checks”
  • The document states fault tolerance as its own quality requirement, separate from availability and recoverability.
  • Retries are limited to idempotent operations.
  • Calls to unreliable dependencies are protected by a circuit breaker or bulkhead.
  • The document says how the system degrades gracefully when a component fails.
  • Chaos experiments name a specific fault and a clear hypothesis.
Section titled “Related”

Sources

Section titled “Sources”