Skip to content
Software Architecture Guide

Pipes and Filters

Explanation

Section titled “Explanation”

Pipes and Filters breaks one large processing task into a sequence of smaller, independent steps. Each step is a filter. Filters are linked by pipes, which carry the output messages of one filter to the next filter.

Filters. In the basic form, a filter has one input and one output. It receives a message from the pipe before it, does its work, and sends a result into the pipe after it. A filter can transform a message. It can also test a message against criteria, which adds conditional logic. Filters are independent, self-contained, and typically stateless, meaning they keep no memory between messages.

Pipes. Pipes only connect. They contain no routing and no logic.

Free ordering. A filter knows only the format of its input and its output. So filters can be arranged in any order, as long as each filter’s output format matches the next filter’s input format. A standardized format makes reordering easier. Because all filters share the same external interface, you can add new filters, omit existing ones, or rearrange them without changing the filters themselves.

Implementation. Teams usually build this as one monolithic pipeline, meaning one deployed unit. A pipeline can also use message queues as pipes, with each filter listening on a queue. For example, Azure Functions with Queue Storage can form an image-processing pipeline. Each filter reads a message, does its work, and posts a new or transformed message to the next queue.

Error handling. Every filter must decide what to do on a breaking error. It either fails the pipeline or passes the exception on. Filters should also be idempotent, so repeating the same message is safe.

Example

Section titled “Example”

An order-intake pipeline has three filters. The first checks that an order has all required fields. The second converts the order into the internal format. The third adds a shipping label. Each filter reads from its input queue and writes to its output queue. If the team later needs a fraud check, they add a fourth filter between two existing ones. The other three stay unchanged, as long as the formats on both sides still match.

Trade-offs

Section titled “Trade-offs”

Benefits. Filters can be added, omitted, or rearranged without changing them. Small single-purpose steps are easy to swap and extend.

Costs.

  • The time to process one request depends on the slowest filter. That filter can become a bottleneck. Running several parallel instances of a slow filter improves throughput.
  • In the usual monolithic form, a failed filter or pipe likely fails the whole pipeline. Any change calls for retesting end to end.
  • Filters must agree on input and output formats. Each filter must also handle its own errors.

Messages flow from a source through filters, linked by pipes, to a sink.

Diagram source 
flowchart LR
  S[Source] -->|pipe| F1[Filter 1: validate]
  F1 -->|pipe| F2[Filter 2: convert]
  F2 -->|pipe| F3[Filter 3: enrich]
  F3 -->|pipe| K[Sink]

Checks

Section titled “Checks”
  • Each filter’s input and output formats are documented, and adjacent filters have matching formats.
  • Pipes contain no routing or business logic; all logic sits in filters.
  • The document states how each filter handles breaking errors and whether it is idempotent.
  • The slowest filter is identified, with a plan such as parallel instances to avoid a bottleneck.
  • The document states whether the pipeline is monolithic and what a failed filter means for the whole pipeline.
Section titled “Related”

Sources

Section titled “Sources”