Log-Based Streaming Architecture
A distributed, append-only, partitioned commit log (Kafka being the archetype) decouples producers from consumers: producers append records, consumers read at their own pace by tracking an offset. The log — not a queue — is the central abstraction, and it turns out to be the unifying primitive for messaging, stream processing, and data integration.
Unlike a traditional message queue that deletes a message once delivered, a log retains records for a configured window and lets many independent consumers read the same stream. Each consumer tracks its own offset (position), so reads are replayable and consumers are decoupled in time. This is what makes the log a backbone for event-driven systems rather than point-to-point messaging. Source: compiled from Kreps, "The Log"
Structure
- Topic — a named stream of records.
- Partition — a topic is split into partitions, each an ordered, append-only sequence. Ordering is guaranteed within a partition, not across them. Partitioning by key (e.g. user id) keeps related records ordered and enables parallelism (see Sharding and Partitioning Architecture).
- Offset — a record's position in a partition; consumers commit offsets to mark progress.
- Consumer group — consumers in a group split the partitions among themselves for parallel consumption; partition count caps parallelism. Different groups each get the full stream.
- Replication — partitions are replicated across brokers with a leader per partition; this is the same quorum/leader idea as Consensus Protocols (Raft and Paxos).
Delivery semantics
- At-most-once — commit offset before processing; lose records on crash.
- At-least-once — process then commit; crash before commit ⇒ reprocessing (the common default).
- Exactly-once — at-least-once delivery + idempotent/transactional processing; see Idempotency and Exactly-Once Processing.
Most systems run at-least-once and make consumers idempotent rather than chase true exactly-once delivery.
Design implications
- Partition key = ordering + parallelism boundary. Choose it so related events stay ordered and load spreads evenly; a hot key serializes a partition.
- Retention is a design choice. Long retention turns the log into a replayable source of truth (Event Sourcing and CQRS); short retention treats it as transport.
- Consumer lag is the health metric. Watch offset lag; growing lag means consumers cannot keep up.
- Backpressure via pull. Consumers pull at their own rate, so a slow consumer falls behind instead of being overwhelmed.
This is the transport layer beneath event-driven sagas and the streaming substrate for projections in Event Sourcing and CQRS.
Architecture Position
| Axis | Value |
|---|---|
| Family | Distributed systems primitives |
| Boundary owned | Append-only partitioned commit log for producers, consumers, offsets, and replay. |
| Read with | Event Sourcing and CQRS, Consensus Protocols (Raft and Paxos), Idempotency and Exactly-Once Processing |
| Use this page when | decoupling producers/consumers or designing replayable streams |
Timeline
-
2026-07-01 | Architecture category refresh added this page to the Distributed systems primitives family, linked it to Architecture System Map, and kept it standalone because it owns this boundary: Append-only partitioned commit log for producers, consumers, offsets, and replay. Source: User request, 2026-07-01
-
2026-06-02 | Page created. Captured log-vs-queue retention/replay, topic/partition/offset/consumer-group structure, in-partition ordering, replication-as-consensus, the three delivery semantics, and partition-key + lag implications. Source: compiled from Kreps, "The Log"; Kafka design docs