Step 11: Evaluate the architecture

What you get from this step

You get a written check of your architecture against your most important quality scenarios. The check lists four kinds of findings:

Risks: decisions that may lead to bad outcomes for a scenario.
Non-risks: decisions the team examined and found sound.
Sensitivity points: decisions that one quality depends on strongly.
Trade-off points: decisions that help one quality and hurt another.

The team also groups the risks into themes. Each theme names the business goal it threatens. You record the results in arc42 section 11. See ATAM Results and Placing ATAM Evaluation in arc42.

The method behind this step is the Architecture Tradeoff Analysis Method (ATAM). It shows how quality goals pull against each other. It also exposes risks that could block business goals. Read Software Architecture Evaluation for the general idea.

What you need first

Quality scenarios. Write them as specific, measurable scenarios, for example in arc42 section 10. Vague goals give vague findings. See Insufficient ATAM Application.
Priorities for the scenarios. The stakeholder group brainstorms scenarios and votes on them. See Scenario Weight Assignment and Prioritization.
Recorded key decisions. Each one names the chosen option, the reasons and all consequences, including negative ones. See Architectural Decision Records as Evaluation Prerequisite.
Business drivers. Document them next to the architecture description. The risk themes need them. See ATAM Results.
A time to evaluate. ATAM works best early, before major spending on implementation. See ATAM Application Phases.

If something is missing

No measurable scenarios. Write them first, then vote on them. Do not evaluate against loose wishes.
No decision records. Write down the important, expensive, large-scale or risky decisions before the review. The reviewers need something concrete to check.
Existing system with doubtful documentation. Recover the real architecture first. Do not simply trust the old documents. See ATAM Application Phases.
Little time or few people. A full evaluation takes days and many people. Choose a lighter method and say so in the report:
- Lightweight ATAM delivers results in about a day but finds fewer risks.
- Lightweight Approach for Software Reviews (LASR) is a short workshop with first results within hours.
- Risk Storming is a quick, collaborative way to find and rate risks.
No external reviewers. A full ATAM uses an external team. A lightweight review may use the project team, outsiders or both. Record who ran it.

See Advantages and Drawbacks of ATAM for the effort trade-off.

How to do it

Choose the variant and estimate the effort. A full ATAM takes about three to four days. A lightweight one takes about a day. Write down the duration and the participants before you start. See ATAM Phase 1 and 2 Activities and Lightweight ATAM.
Form the team and pick the stakeholders. A full ATAM uses an external team of three to five people. Assign the roles: evaluation leader, scribes, timekeeper and questioners. Agree on the stakeholder list with the key decision makers. See ATAM Evaluation Team and Stakeholder Selection and ATAM Roles.
Brief everyone. The evaluation leader explains the method to all participants. Then present the business drivers and the architecture. See ATAM Phase 1: Architecture-centric Analysis.
Settle the scenarios. Refine the quality goals into scenarios with a stimulus, a response and a priority. Let the whole stakeholder group add scenarios and vote. See Scenario Weight Assignment and Prioritization.
List the architectural approaches. The architect names the approaches. Only list them here. Do not attach findings yet. See Architectural Approach Identification.
Analyze the approaches against the top scenarios. Take the highest-priority scenarios first. For each one, ask which decisions make it work or fail. Record each finding as you go:
- A risk names a decision and the scenario it could make fail.
- A non-risk names a scenario, a decision and the reason it is sound.
- A sensitivity point names a decision and the quality that reacts strongly to it.
- A trade-off point names the quality it helps and the one it harms.
Use concrete scenarios as prompts. See Scenario-based Questions from Tactics.
Group the risks into themes. Each theme names the business driver it threatens. See ATAM Results.
Rank the risks and plan mitigations. Rate each risk by probability and impact. Order the list by priority. Give each high-priority risk a mitigation with an owner and a date. See Risk Storming.
Write the results down. Deliver an out-brief, a written report or both. List non-risks next to the risks. In section 11, separate risks, sensitivity points and trade-off points. Name the stakeholders who helped find them. Record accepted trade-offs in section 9 with their reasons. Say where technical debt is recorded if section 11 no longer holds it. See Placing ATAM Evaluation in arc42, Trade-off Point and Technical Debt.

Example: ClinicBook

ClinicBook is an online booking system for 40 small medical practices. The team is small and the launch date is fixed. A full ATAM would cost too much. The team plans a lightweight ATAM of one day. It writes down that the review may miss some risks. Two outside reviewers and the ClinicBook team run it. Two practice staff members attend as stakeholders. The team records all of this in its report.

The business drivers are privacy of health data, availability in the morning peak, a working link to the practices’ own software and the launch date.

The group votes on its scenarios. Three get the top priority:

S1 (availability): On a weekday at 8:00, 2,000 patients book at once. Each booking is confirmed within 3 seconds.
S2 (integration): A receptionist books a slot in the practice’s own software. A patient tries to book the same slot a minute later. ClinicBook must not create a double booking.
S3 (privacy): An SMS reminder goes out. It must not reveal health information.

The team checks the decisions behind these scenarios. The decisions are recorded in the team’s decision records.

Type	Finding	Scenario
Risk	ClinicBook copies each practice calendar every 15 minutes. A slot changed in the practice software during that gap can be booked twice.	S2
Sensitivity point	The copy interval strongly affects how fresh the availability data is.	S2
Trade-off point	A shorter interval gives fresher data. It also loads the old practice systems more and may hurt their availability.	S2, S1
Trade-off point	A reminder with no medical details protects privacy. It also helps the patient less.	S3
Non-risk	If the SMS provider fails, reminders wait in a queue and go out by e-mail. The team walked through this decision and found it sound for the reminder scenario.	S3

The team groups the two integration findings into one theme: “Calendar data from legacy systems is not reliable enough.” The theme threatens two drivers: trusted bookings and the fixed launch date. It ranks the double-booking risk first. The mitigation is to check the slot again in the practice software at the moment of booking. The team assigns an owner and a date.

The accepted privacy trade-off goes into section 9 with its reasons. The findings go into section 11, grouped as risks, sensitivity points and trade-off points. Each finding names its scenario and the people who helped find it.

Done when

The report says whether you ran a full or a lightweight review, who took part, and that a lightweight review may miss risks. See Lightweight ATAM.
The evaluated scenarios are measurable and were prioritized by a vote of the stakeholders. See Scenario Weight Assignment and Prioritization.
Each risk names a decision and a scenario. Each non-risk names a scenario, a decision and a reason. See Insufficient ATAM Application.
Sensitivity points and trade-off points each name the qualities they affect. Trade-off points name at least two. See Trade-off Point and Sensitivity Point.
Risks are grouped into themes, and each theme names the business driver it threatens. See ATAM Results.
Risks are ordered by priority, and high-priority ones have a mitigation. See Risk Storming.
Section 11 holds the results, and it states where technical debt is recorded. See Placing ATAM Evaluation in arc42.