Harness Engineering
The discipline of designing everything outside the model weights that makes an agent reliable. The shape is a while loop. The engineering is everything else.
Core Mental Model
An agent is a while loop. A subagent is a nested while loop. The agent harness is the rest of the code. A cloud agent is all of the above, running on EC2. Source: X/@BenjDicken, 2026-05-15
But the loop alone produces runaway tokens. What separates working agents from expensive noise is the loop invariants: stop conditions, evaluators, context window discipline. The while loop is the shape. The invariants are the engineering. Source: User, 2026-05-19
Relationship to Agent Harness
Agent Harness is now the canonical concept page for the product boundary: context, tools, workspace, permissions, memory, verification, observability, and human interface around the loop. This page owns the engineering discipline and failure-analysis method. Concrete pages such as Cursor Harness, Claude Code Harness, Codex Harness (OpenAI), Hermes Harness (Nous Research), and OpenClaw Harness should link back to Agent Harness when describing what layer they occupy.
Five Defense Layers
Every harness failure maps to one of five layers. Diagnose systematically. Source: Walking Labs, Lecture 01, 2026-05-19
| Layer | What it controls | Failure mode |
|---|---|---|
| Task specification | What the agent should do | Vague intent, ambiguous scope, no acceptance criteria |
| Context provision | What the agent knows | Missing architecture docs, implicit conventions, no AGENTS.md |
| Execution environment | Where the agent works | Missing deps, wrong versions, incomplete toolchain |
| Verification feedback | How the agent knows it succeeded | No tests, no lint, no type checks, premature "done" |
| State management | What persists across sessions | Lost discoveries, repeated exploration, no checkpoint |
Repo readiness for coding agents
Ramp's public Inspect anecdote is useful because it names the precondition behind high internal agent adoption: the agent did not become a coworker from model choice alone. The investment was repo readiness across main repos: dependencies installed, commands runnable and performant, skills installed and token-efficient, and tools available. Source: X/@rahulgs, 2026-06-19
That belongs in the execution-environment layer, not a new "Inspect" tool page. For Kevin's repos, this means agent readiness is a repo contract: setup scripts, local docs, skill routing, fast checks, and tool availability are part of the harness. A coding agent that cannot install, run, test, and inspect the project quickly is not autonomous; it is waiting for the operator to be the harness.
2026 AI Engineer Stack
The 2026 AI-engineering skill stack is now closer to distributed-systems engineering than prompt copywriting. A June 2026 bookmark checklist reinforces this page's core claim: learn harness engineering over prompt engineering, context engineering over long prompts, prompt-caching versus semantic-caching tradeoffs, KV-cache management, prefill versus decode latency, tool-protocol overhead, evals, and observability. Source: X/@divaagurlxw, 2026-06-04; Source: X/@divaagurlxw, 2026-06-24
That list belongs here because each topic is outside the model weights but inside the system that determines whether an agent run is reliable and economical:
| Competency | Harness question |
|---|---|
| Context engineering | What does the agent see, in what order, with what trust labels? |
| Prompt caching | Which stable context should be cheap and reusable? |
| Semantic caching | Which answers or tool results can be reused without lying? |
| KV-cache behavior | What does long context cost at prefill/decode time? |
| Tool protocol design | How much latency, auth, and error surface does each tool add? |
| Evals and observability | How does the system detect regression before a human notices? |
The linked distributed-systems thread now lives at Agent Resilience Patterns: circuit breakers become agent kill switches, bulkheads become per-tool blast-radius limits, saga patterns become compensation chains, service discovery becomes an MCP/tool registry, idempotency keys protect side-effecting retries, and distributed traces become model/tool/workspace action traces. This matters because the vocabulary is more operational than "make the agent reliable": it names which harness invariant is missing.
The Verification Gap
Agents declare victory before the work is done. Anthropic observed "context anxiety": when context windows fill, agents rush to finish, skip verification, and choose simple solutions over correct ones. The fix is an explicit Definition of Done with machine-verifiable conditions. Source: Walking Labs / Anthropic, 2026-05-19
Harness Patterns
Planner / Builder / Evaluator Loop
Anthropic's "Harness design for long-running application development" post is the current source-backed form of the AI Engineer Europe clip. The loop is three agents: a planner expands a short prompt into a product spec, a generator builds one chunk at a time, and an evaluator uses Playwright and explicit criteria to test the result before sending feedback back into the generator. The local video review shows the talk title "How to Build Agents That Run for Hours (Without Losing the Plot)" and slide material about keeping Claude on track and harness design for long-running agents. Source: Anthropic, 2026-03-24; Source: X/@AnatoliKopadze and local video contact sheet, 2026-07-04
The reusable rule is that "best loop wins" means separate roles and negotiated proof, not simply more autonomous runtime. The evaluator is load-bearing only when the task sits beyond what the current model does reliably by itself; otherwise it becomes overhead. Source: Anthropic, 2026-03-24
ARC AGI Harness Lessons
The ARC AGI rows are useful because they isolate harness design under an adversarial benchmark rather than under product-demo pressure. The durable lesson is not that Kevin should optimize for ARC; it is that winning harnesses expose what the loop is actually doing: search strategy, representation choice, candidate generation, verification, and when to spend another attempt. Source: X bookmark artifact audit, 2026-07-06
Keep the routing discipline strict:
- Benchmark harnesses are evidence for The Eval Loop (Slop Is an Output Problem) and Browser Testing Skills, not a general permission slip for agent autonomy.
- A harness should make the representation and verifier explicit enough that a human can see whether it is solving the task or overfitting the contest.
- Product harnesses need the same shape plus real-world state: auth, data, UI, deployment, cost, and rollback.
- Evaluation writing is itself product work; the "it's hard to eval" row belongs here because weak evals create false confidence even when the agent loop looks sophisticated. Source: X bookmark
2072390959774810145, 2026-07-06
Implementation Notes Pattern
When working from a spec, the agent maintains a running implementation-notes.html capturing divergences:
- Design decisions: choices where the spec was ambiguous
- Deviations: intentional departures with rationale
- Tradeoffs: alternatives considered, reasons for selection
- Open questions: items needing human confirmation
- Timestamps: along with worktree changes, branches cut, git decisions, versions
This creates an audit trail that maps agent decisions to human-reviewable context. Source: User, 2026-05-19
The source thread's practical value is the permission boundary: the human specifies the outcome, while the agent records the decisions that were not fully specified. That keeps ambiguity from either blocking progress or disappearing into an unreviewable final diff. Source: X/@trq212 self-thread, 2026-05-18
Cloudflare Vulnerability Discovery Harness
Cloudflare's Project Glasswing (testing Anthropic's Mythos Preview) demonstrates harness engineering at scale across 50+ repositories. Their 8-stage pipeline: Source: Cloudflare Blog, 2026-05-18
| Stage | Purpose |
|---|---|
| Recon | Read codebase, produce architecture doc with trust boundaries and entry points |
| Hunt | ~50 agents run concurrently, each scoped to one attack class + one component |
| Validate | Independent agent tries to disprove each finding (different prompt, different model) |
| Gapfill | Areas touched but not covered thoroughly get re-queued |
| Dedupe | Findings sharing the same root cause collapse into one record |
| Trace | Confirm whether attacker input actually reaches the bug from outside the system |
| Feedback | Reachable traces become new hunt tasks (closed-loop improvement) |
| Report | Structured output against a predefined schema |
Key insight: pointing a generic coding agent at a repo does not work. Narrow scope, adversarial review, split reasoning chains, and parallel narrow tasks all outperform one exhaustive agent. Source: Cloudflare Blog, 2026-05-18
The reviewed Eugene Yan artifact preserves the harness as a compact operator diagram: Recon -> Hunt -> Validate -> Gapfill -> Dedupe -> Trace -> Feedback -> Report. Treat it as the operator-level shape of the Cloudflare blog post, not a separate framework. Source: X/@eugeneyan and local image review, 2026-07-04
Kevin's Agent Operating System
Kevin's personal harness at Dedalus and in the personal wiki follows the same principles:
- Rules as harness:
.mdcfiles that scope agent behavior per file type, always-on conventions - Verification hooks: pre/post-edit formatting, lint, type checks
- Skills as invariants: reusable playbooks that constrain agent behavior to proven patterns
- Doctors as evaluators: subsystem-specific health checks that run after edit sessions
- Wiki as persistent state: compiled knowledge that survives sessions and agents
See Agent Operating System for the full architecture. Source: compiled from wiki, 2026-05-19
Fan-Out Needs Verifiers
Cursor's /orchestrate launch is a clean product example of the verifier layer: planners spawn workers that write code and verifiers that run it; a failed verifier result becomes the next worker's input. The key harness idea is not "more agents." It is separate execution from evaluation so the system has a reason to retry instead of a worker marking itself done. Source: X/@cursor_ai self-thread, 2026-05-07
Peter Steinberger's "Just Talk To It" article adds the human-control side: watch blast radius, interrupt when the actual scope diverges from expected scope, ask for status, then continue, abort, or split. For Kevin, this complements worktree-first safety: the operator loop can be fast, but parallel agents still need owned commits, precise staging, or isolated worktrees. Source: Peter Steinberger, read 2026-06-30
Key References
- Learn Harness Engineering (Walking Labs): best structured course on the topic
- OpenAI: Harness Engineering: 1M-line Codex experiment
- Anthropic: Effective Harnesses: long-running agent patterns
- Cloudflare: Project Glasswing: vulnerability harness at scale
The Rule
When an agent fails, check the harness before the model. One AGENTS.md file can outperform upgrading to a more expensive model. Source: Walking Labs, Lecture 01, 2026-05-19
Product time is still outside the harness
Harnesses expand coverage, make agent work legible, and reduce false confidence. They do not make product time disappear. Long Live Hard SaaS generalizes the Cloudflare Glasswing lesson from security to SaaS: many bugs only appear when a maintained product is used across real flows, state, auth, billing, storage, and deployment boundaries. The harness helps find and prove more issues; the SaaS company still owns the long maintenance loop. Source: raw/notes/saas-bug-moat-slack-thread-2026-06-22.md; Cloudflare Blog, 2026-05-18
Harness portability (app embedding)
The AI SDK already proved model portability — swap @ai-sdk/openai for @ai-sdk/anthropic without rewriting the app. AI SDK 7 extends the same idea to harness portability: HarnessAgent normalizes Claude Code, Codex, Pi (and future adapters) behind one session + sandbox + stream API, so product code does not re-implement each harness's skills, permissions, and compaction. Write the agent integration once; swap the harness as the ecosystem moves. This is the application-layer complement to Kevin's harness architecture pages (Claude Code Harness, Codex Harness (OpenAI), Cursor Harness) — those document what each runtime is; AI SDK HarnessAgent documents how to embed them. Source: Vercel changelog + AI SDK harness docs, 2026-06-12
Timeline
- 2026-07-06 | Added ARC AGI harness rows as benchmark-level evidence for explicit representation/search/verifier design, plus the eval-writing caveat that weak product evals create false confidence. Routed the TRINITY coordinator row to LLM Council instead of treating it as generic harness advice. Source: X bookmark artifact audit, 2026-07-06
- 2026-07-04 | Source-reviewed the earlier @divaagurlxw AI-engineer checklist row. It adds no durable artifact beyond the root checklist, but it is high-bookmark evidence that caching, KV-cache behavior, latency, tool overhead, evals, and observability now belong on the AI-engineering skill map. Source: X/@divaagurlxw, 2026-06-04
- 2026-07-04 | Source-reviewed @rahulgs's Ramp Inspect anecdote: "75%+ code at Ramp now comes from Inspect" is useful mainly as evidence for repo readiness as harness work. The durable lesson is dependency setup, runnable/performance-checked commands, token-efficient skills, and configured tools across main repos, not a public tool adoption recommendation. Source: X/@rahulgs, 2026-06-19
- 2026-07-04 | Reviewed @BenjDicken's artifact reply image for the "agent = while loop" bookmark. The image is a general marketing meme, so the durable harness claim remains the tweet's taxonomy rather than the reply image. Source: X/@BenjDicken reply and local image review, 2026-07-04
- 2026-07-04 | Added Eugene Yan's Cloudflare vulnerability-discovery diagram as artifact-level evidence for the eight-stage Project Glasswing harness. Source: X/@eugeneyan, 2026-05-18; Source: Cloudflare Blog, 2026-05-18
- 2026-07-04 | Added Anthropic's planner/builder/evaluator loop from the AI Engineer Europe bookmark and the official harness-design post. The durable lesson is role separation plus evaluator-negotiated proof for long-running app builds, not simply "more agents." Source: X/@AnatoliKopadze, 2026-06-20; Source: Anthropic, 2026-03-24
- 2026-07-03 | Source-reviewed the @trq212 implementation-notes self-thread and tightened the pattern around ambiguity: agents may decide locally, but must keep a running reviewable record of decisions, deviations, and tradeoffs. Source: X/@trq212, 2026-05-18
- 2026-06-30 | Added verifier fan-out and blast-radius operator control from the
/orchestrateand Peter Steinberger bookmark review: planners/workers need independent verifiers, and long-running agent work should be interrupted when observed blast radius diverges from expected blast radius. Source: X/@cursor_ai, 2026-05-07; Source: Peter Steinberger, read 2026-06-30 - 2026-06-30 | Promoted the @divaagurlxw distributed-systems mapping into Agent Resilience Patterns so circuit breakers, bulkheads, sagas, idempotency keys, health checks, backpressure, canaries, and traces have explicit agent-harness equivalents. Source: X/@divaagurlxw, 2026-06-24
- 2026-06-18 | Split the generic "agent harness" alias into canonical Agent Harness and wired this page as the engineering discipline behind that concept. Source: whole-wiki graph audit, 2026-06-18
- 2026-06-25 | Added the AI-engineer stack checklist: harness engineering, context engineering, caching/KV-cache tradeoffs, latency, tool-protocol costs, evals, and observability. Source: X/@divaagurlxw, 2026-06-24
- 2026-06-22 | Added the product-time boundary: harnesses improve coverage and proof, but hard SaaS still compounds through real usage, regression, and maintenance loops. Source: raw/notes/saas-bug-moat-slack-thread-2026-06-22.md
- 2026-06-15 | 7-phase pipeline + Matt Pocock's skills catalogued as a concrete harness instance: small composable skills as loop invariants (grill→prototype→PRD→issues→implement→review), with
tdd/diagnoseas feedback loops andreviewas the verification gate. Source: github.com/mattpocock/skills, 2026-06-15 - 2026-06-08 | @divaagurlxw checklist: harness engineering > prompt engineering; context engineering; prompt vs semantic caching; KV cache. 7,184 bookmarks. Source: X/@divaagurlxw, 2026-06-08
- 2026-06-12 | Added harness portability section — Vercel AI SDK 7
HarnessAgentas unified embed API for Claude Code / Codex / Pi. Source: Vercel changelog, 2026-06-12 - 2026-06-06 | @eng_khairallah1: "You're not supposed to prompt Claude. You're supposed to build a system that prompts itself." 20,287 bookmarks — viral restatement of harness-over-prompting. Source: X/@eng_khairallah1, 2026-06-06
- 2026-05-19 | Page created. Compiled from Walking Labs course, OpenAI/Anthropic harness engineering articles, Cloudflare Project Glasswing blog post, and Kevin's mental model (agent = while loop, invariants = the engineering). Source: User + Walking Labs + Cloudflare, 2026-05-19
- 2026-05-18 | @trq212's "maintain a running implementation-notes.html" spec technique traced as the viral source of the implementation-notes pattern. 1,342 likes, 2,249 bookmarks. Source: X/@trq212, 2026-05-18
- 2026-05-18 | Cloudflare tested Anthropic Mythos against 50 of its own repos (3,964 likes, 4,805 bookmarks); Eugene Yan distilled the Recon/Hunt/Validate/Gapfill/Dedup pipeline. Source: X/@Cloudflare, 2026-05-18
- 2026-05-15 | The "agent = while loop" mental model originates from @BenjDicken's "engineering cheatsheet 2026" tweet (983 likes), predating the prior in-page User citation. Source: X/@BenjDicken, 2026-05-15