Skip to content
Software Architecture Guide

Event Sourcing with Distributed Event Log

Explanation

Section titled “Explanation”

The classic Shared Database style (see the article Shared Database) has applications store their data in one common database. A distributed event log is a special-purpose variant of that idea. Instead of tables, the shared store is a log of events.

Event sourcing in brief. Event sourcing stores every change to application state as a sequence of events. The log can rebuild the current state. Replaying it from a blank state up to a chosen time or event also reconstructs past states.

How the log works as an integration medium. The log is append-only. Writers add events and do not edit old ones. The code that produces events does not know which systems consume them.

Readers pull events. Nothing is pushed to them. In Azure Event Hubs, an event hub is an append-only log, equivalent to a Kafka topic. Each consumer group is an independent view of the stream and tracks its own position.

Replay. A consumer can start at a specific offset (a position in the log), at a timestamp, or at the beginning or end. If it saves its position (a checkpoint), it can resume after a failure. It can also replay historical events by choosing an earlier offset.

Retention. Retention is time-based and configurable up to a tier maximum. The platform then removes events automatically. Nobody can delete them explicitly.

Ordering. Order is guaranteed only within a partition, which is one ordered sequence of events inside the log. A partition key keeps related events together and in order. Demanding absolute order across all events limits how many partitions you can use.

Relation to other patterns. CQRS (see Command Query Responsibility Segregation) pairs naturally with event sourcing. If the write model emits events for every update, read models can be built from them. Fowler also describes event sourcing as the foundation for patterns such as Parallel Models and Retroactive Events.

Example

Section titled “Example”

An online shop writes “order placed”, “order paid” and “order shipped” events to one log, using the order number as partition key. A billing service and a reporting service each read the log with their own position. Reporting breaks for a day. After the fix, it resumes from its saved position and catches up, as long as those events are still within retention.

Trade-offs

Section titled “Trade-offs”
  • Retention limits replay. Events disappear after the retention period, and they cannot be deleted on demand.
  • Ordering is partial. Order holds only within a partition. Demanding global order limits how many partitions you can use.
  • A broker is not a full event store. A broker such as Kafka typically lacks per-entity stream queries and optimistic concurrency. It mainly distributes events.
  • Duplicates are possible. Delivery is typically at-least-once, so consumers must be idempotent (see Idempotent Receiver).
  • Decoupling. Producers do not depend on subscribers, which lets you add new readers without changing writers.

Writers append to one log; each consumer group pulls from its own position.

Diagram source 
flowchart LR
 W1[Writer A] -->|append| L[(Event log, ordered per partition)]
 W2[Writer B] -->|append| L
 L -->|pull from offset| C1[Consumer group 1]
 L -->|pull from offset| C2[Consumer group 2]
 L -.->|pruned after retention| X[Removed]

Checks

Section titled “Checks”
  • The document states the retention period of the event log and what that means for replay.
  • Consumers are described as idempotent, because delivery is typically at-least-once.
  • Events that must stay in order share a partition key, and the document states where order is not guaranteed.
  • How each consumer tracks its position (offset or checkpoint) is documented.
  • If a broker is the system of record, the document explains how it covers per-entity queries and optimistic concurrency.
Section titled “Related”

Sources

Section titled “Sources”