Agentic Quality System
An agentic quality system is the full measurement and verification loop around AI-assisted work: tests, evals, doctors, traces, human review, production monitors, and memory updates.
Why this needs its own page
The wiki has many quality leaves: The Eval Loop (Slop Is an Output Problem), Eval-Driven Development, Agent Trajectory Evaluation, Doctor Pattern, Self-Maintaining Systems, Security and Review Skills, Browser Testing Skills, Frontend and Design Skills, CI checks, browser tests, and wiki doctor. The missing concept was the system boundary: quality is not one check, it is a stack.
Quality stack
| Layer | Catches | Example |
|---|---|---|
| Static checks | syntax, types, lint, formatting, obvious policy violations | TypeScript, Ruff, Ultracite, style doctors |
| Unit/invariant tests | deterministic correctness | small tests around business invariants |
| Browser/runtime tests | user-visible behavior | Playwright, browser-harness, screenshots |
| LLM evals | stochastic output quality | Eval-Driven Development, LLM-as-judge, rubric scores |
| Trajectory evals | path quality and tool behavior | Agent Trajectory Evaluation, traces, tool-call scoring |
| Human review | taste, judgment, risk, final accountability | PR review, design review, approval gates |
| Production monitors | real-world drift, latency, incidents | Self-Maintaining Systems, SLO alerts |
| Memory update | future recurrence | wiki updates, skill changes, new doctor checks |
The loop
The quality system is only complete when failures become future gates. Otherwise the team is just manually inspecting the same class of error forever, which is extremely funny if your goal is suffering.
Agent-specific additions
Agents require extra quality surfaces because they are non-deterministic and tool-using:
- trace the trajectory, not just the final message;
- score tool choice and tool arguments;
- preserve artifacts and state writes;
- separate evals for output quality, safety, and cost;
- make approvals part of the run record;
- update memory only after evidence is reviewed;
- encode repeated failures into skills/doctors.
Codex as feature tester
@gdb's Codex testing bookmark is a useful product signal: coding agents are moving from "write this patch" toward "exercise every feature path in the app and produce evidence." Route that work through the quality stack, not through ad-hoc clicking: the agent can explore, use browser/runtime checks, capture screenshots and traces, then promote important paths into Playwright, invariants, or evals. Source: X/@gdb, 2026-06-21
Kevin rule
When a page says "verify," it should name the quality layer:
- deterministic code: tests/types/lint,
- UI: browser screenshot and interaction checks,
- agent behavior: trajectory/eval suite,
- content/taste: rubric + humanizer/voice skill,
- wiki: build-index, doctor, qmd, log,
- production: monitors and rollback path.
Vague verification is just vibes with a clipboard.
Timeline
- 2026-07-04 | Added Codex-as-feature-tester as a quality-system signal: agents should explore app behavior and produce evidence, but recurring paths must be promoted into browser tests, invariants, or evals. Source: X/@gdb, 2026-06-21
- 2026-06-19 | Created to unify evals, doctors, traces, CI, reviews, production monitors, and memory updates into one cross-section quality concept. Source: whole-wiki concept review, 2026-06-19