Observability Architecture
Monitoring tells you that something is wrong (known failure modes, dashboards, alerts). Observability lets you ask why — to interrogate a system's behavior from its outputs without shipping new code for each new question. The substrate is three signal types — logs, metrics, traces — emitted by instrumented code and collected into queryable backends.
The distinction matters at scale: you cannot predict every failure mode of a distributed system, so you instrument richly enough to debug unknown-unknowns after the fact. The architecture is a pipeline: instrument → collect → store → query/alert, with OpenTelemetry as the vendor-neutral standard for the first two stages. Source: compiled from observability literature
The three pillars
| Signal | Answers | Shape | Cost driver |
|---|---|---|---|
| Logs | "What happened at this point?" | Timestamped events (ideally structured/JSON) | Volume |
| Metrics | "How much / how often / how fast?" | Numeric time series with labels | Cardinality (label combinations) |
| Traces | "What was the path of this request?" | Spans linked by trace id across services | Volume × sampling |
Together they triangulate: a metric/alert says latency spiked, a trace shows which hop is slow, logs on that span say why.
For agent systems, traces become Agent Trajectory Evaluation data: model calls, tool calls, handoffs, guardrails, approvals, artifacts, and recovery steps are the request path.
Golden signals
Google SRE's four golden signals are the canonical starting metrics for any service: latency, traffic, errors, saturation. Alert on these (especially their SLOs) before drowning in bespoke dashboards. Source: Google SRE
Distributed tracing
A trace follows one request across services via context propagation: an incoming trace id + span id is read from headers, a child span is created for local work, and the context is injected into outgoing calls. The mesh can emit spans automatically at each hop, but in-process spans still need app instrumentation.
Design implications
- Structure logs. Unstructured text is grep-only; structured logs are queryable and joinable to traces by id.
- Cardinality is the metrics budget. High-cardinality labels (user id, request id) explode time-series storage; keep them in traces/logs, not metric labels.
- Sample traces. 100% tracing is expensive; head/tail sampling keeps the interesting (slow/error) traces affordable.
- Instrument first, orchestrate second. You cannot operate (or debug agents) without telemetry — Kevin's Observation Precedes Orchestration and the Monitorability Tax make this a hard prerequisite, and tuning breakers/latency budgets depends on it.
Dedalus's API Backend dual-audience error design fits here: rich structured operator logs (full diagnostics) vs safe user-facing errors — the operator side is the observability surface.
Architecture Position
| Axis | Value |
|---|---|
| Family | Safety and observability |
| Boundary owned | Signals, traces, logs, metrics, and debugging architecture. |
| Read with | Agent Trajectory Evaluation, Resilience Patterns (Circuit Breaker, Bulkhead, Retry), Service Mesh and Service Discovery Architecture |
| Use this page when | designing systems that explain failures |
Timeline
-
2026-07-01 | Architecture category refresh added this page to the Safety and observability family, linked it to Architecture System Map, and kept it standalone because it owns this boundary: Signals, traces, logs, metrics, and debugging architecture. Source: User request, 2026-07-01
-
2026-06-18 | Linked agent traces to Agent Trajectory Evaluation and Agent Runtime Architecture. Source: whole-wiki graph audit, 2026-06-18
-
2026-06-02 | Page created. Captured monitoring-vs-observability, the logs/metrics/traces pillars with cost drivers, the four golden signals, context-propagation tracing, and implications (structured logs, cardinality budget, sampling, instrument-before-orchestrate). Source: compiled from observability literature; Google SRE