Skip to content
Software Architecture Guide

Stream Connector

Explanation

Section titled “Explanation”

A stream connector moves data between parts of a system in a steady flow. Each part reads input from the previous one and writes output to the next one. Unix pipelines work this way: each process uses standard streams to read from its predecessor and write to its successor.

Pipes and filters. This style (see the article Pipes and Filters) splits work into small steps called filters. A pipe connects one filter to the next and carries the output messages of one filter to the input of the next. In the basic form, a filter has one input port and one output port.

Running at the same time. In a Unix pipeline, the processes run concurrently. The downstream process starts while the upstream one is still running. So work on the data begins before the sender has finished.

Buffering. A sender and a receiver may work at different speeds. A Unix pipe handles this with a buffer. It holds the sender’s output until the receiver reads it. On most Linux systems, this buffer is 64 KiB.

Back pressure. Some stream technologies let a slow receiver slow the sender down. Reactive Streams is an initiative that standardizes asynchronous stream processing with non-blocking back pressure.

Example

Section titled “Example”

A team builds a log-cleaning chain. A first step reads raw log lines. A second step removes noise. A third step counts errors. Pipes connect the steps. All three run at once, so the counter starts on the first lines while the reader is still reading later ones. If the counter is slower, a buffer holds the waiting lines.

Trade-offs

Section titled “Trade-offs”

A buffer has a fixed size, so it can only smooth out differences in speed. The verified facts here do not say what happens when a buffer is full, or what reliability guarantees a given stream connector offers. Check these points for each technology you choose.

A simple pipeline: data flows from a source through two filters to a sink.

Diagram source 
flowchart LR
  S[Source] -->|pipe| F1[Filter A]
  F1 -->|pipe, buffer| F2[Filter B]
  F2 -->|pipe| K[Sink]

Checks

Section titled “Checks”
  • The document states how each pipe connects filters and which output feeds which input.
  • The document says how differences in sender and receiver speed are handled, for example by a buffer.
  • Where back pressure is needed, the document names the mechanism or standard used.
Section titled “Related”

Sources

Section titled “Sources”