Skip to content
Software Architecture Guide

Recoverability

Explanation

Section titled “Explanation”

Recoverability is a part of reliability in the ISO/IEC 25010 quality model (see the article ISO/IEC 25010 Product Quality Model). The standard describes it as the degree to which a product or system can recover the data directly affected when an interruption or failure happens.

Measuring it: MTTR

The usual measure is MTTR. The abbreviation is ambiguous. It can stand for three things:

  • Mean Time to Respond: the time from detecting a problem to the start of active response.
  • Mean Time to Repair: the time needed to fix the affected component.
  • Mean Time to Recovery: the time needed to restore the service to normal operation.

Repair and recovery differ. Say which meaning you use. Azure defines MTTR as the time to restore a component after a failure is detected. It groups MTTR with RTO, RPO and MTBF as recovery targets.

Related targets

RTO is the maximum downtime allowed for a single incident. MTTR is an average over several incidents. RPO is the maximum acceptable time since the last data recovery point, so it limits data loss. Together, RTO and RPO bound both downtime and lost data.

Agreements

An SLA is a formal contract. It has financial and legal consequences if the organization fails to deliver. An SLO is an internal target and is a different thing.

Ways to recover faster

  • Failover: switch to a redundant copy of a component. With several deployment units (“stamps”) in multiple regions, a failure in one can be recovered by failing over to another. This raises the workload’s overall SLO.
  • Automatic triggers: set recovery thresholds that allow for short-lived (transient) faults. For example, if a web node is unavailable for more than five minutes, add a new node to the pool automatically.
  • Recovery-oriented design: recovery-oriented computing improves availability by shortening recovery time, not by trying to prevent every failure.

Example

Section titled “Example”

An online shop runs in two regions. The database in region A fails. Monitoring detects it, and traffic fails over to region B while engineers repair region A. The team had set a maximum downtime for one incident (RTO) and a maximum age of the last data copy (RPO). It then checks the incident against both targets and adds the duration to its average recovery time.

Trade-offs

Section titled “Trade-offs”

Recovery targets must match what the business needs and what an SLA promises. Tighter targets need more redundancy and automation, which the facts above do not quantify. Thresholds that react too fast may trigger recovery for faults that would have cleared by themselves, so they should allow for transient faults. See the article Quantifying Recovery and Availability Trade-offs.

Phases between a failure and restored service; failover can shorten the path.

Diagram source 
flowchart LR
  A[Failure occurs] --> B[Detect]
  B --> C[Respond]
  C --> D[Repair]
  D --> E[Recover: normal operation]
  B -.->|failover to redundant copy| E

Checks

Section titled “Checks”
  • The document states which meaning of MTTR (respond, repair or recovery) it uses.
  • Recovery targets are defined per incident (RTO) and for acceptable data loss (RPO).
  • Contractual commitments (SLAs) are distinguished from internal targets (SLOs).
  • Failover to redundant components is described for the failures the system must recover from.
  • Recovery thresholds allow for transient faults and name the automatic action they trigger.
Section titled “Related”

Sources

Section titled “Sources”