Agent Harness, Runtime, Memory, and Evals
Agent systems become legible when harness, runtime, memory, tools, skills, and evals are separate layers with explicit proof instead of one chat loop. Source: User request, 2026-06-27; Agent Harness
When To Use This
Use this page when designing or comparing agent products, coding harnesses, long-running runtimes, memory systems, eval loops, or portable agent docs. Source: Agent Runtime Architecture, 2026-06-27
Decision Matrix
| Need | Route | Verification |
|---|---|---|
| Compare agent products | Agent Harness and Cross-Agent Harness Portability | Capability/layer table |
| Compare local vs hosted coding agents | Local Coding Agents plus concrete harness pages | Same receipt fixture across model/runtime/harness |
| Runtime/sandbox design | Agent Runtime Architecture | State and isolation model |
| Memory | Agent Memory System Architecture | Read/write/forget policy |
| Quality gate | The Eval Loop (Slop Is an Output Problem) and BINEVAL | Eval input, atomic criteria, expected output, and result |
| Portable context | Agent-Docs Mesh and generated packs | Pack/check output |
Core Model
Harness is the interface that lets an agent operate. Runtime is where it executes. Memory is what persists. Skills and tools are capabilities. Evals decide whether the system is getting better. Collapse them and debugging becomes vibes. Source: Agent Harness; The Eval Loop (Slop Is an Output Problem)
The emerging durable shape is file-native: an agent is a directory that carries its own identity, tools, skills, subagents, connections, schedules, evals, package metadata, and traces. Vercel's Vercel Eve made this explicit, and Omar Sar's 2026-06-27 bookmark restated the same pattern as a generic agent directory: tools say what it can do, skills say when to do it, subagents say who it delegates to, connections say how it reaches the outside world, schedules say when it runs, and evals prove it still works. Source: X/@omarsar0, 2026-06-27; Source: Vercel Eve
The self-improving version adds two extra files to the mental model: a receipt fixture and an evaluator-evolution rule. Agent Self-Improvement Eval Library supplies the local receipt suite, while Red Queen Gödel Machine supplies the principle that the evaluator must harden when the agent learns to pass the current bar. Loopy then turns the improvement behavior into a bounded loop instead of an open-ended "try again."
For Local Coding Agents, keep the same receipt standard. A local stack can win on privacy, cost, modifiability, and learning, but it only earns a production route when it can pass comparable task receipts: prompt, workspace, diff, tests, browser proof if applicable, and failure taxonomy. Source: Sebastian Raschka, 2026-06-27
Tool And Skill Routing
Use agent harness pages for product comparison, architecture pages for runtime and memory design, skills for executable workflows, and eval pages for quality gates. Generated packs should include this cluster before listing every harness/tool mirror. Source: Wiki Capsules And Agent Doc Packs, 2026-06-27
Failure Modes
- Treating model choice as the whole agent system.
- Mixing memory facts with temporary session context.
- Adding tools without evals or observability.
- Comparing harnesses by feature lists without runtime and permission boundaries.
Evidence And Timeline
Proof should name the harness, runtime, memory policy, capability registry, eval criteria, and operational checks. Source: Agent Control Plane Architecture; The Eval Loop (Slop Is an Output Problem)
Concept Position
| Field | Value |
|---|---|
| Concept family | Brain, memory, and retrieval |
| Concept owned | Agent systems become legible when harness, runtime, memory, tools, skills, and evals are separate layers with explicit proof instead of one... |
| Category map | Concept System Map |
Timeline
- 2026-07-01 | Concepts category refresh added this page to the Brain, memory, and retrieval family, linked it to Concept System Map, and kept it standalone because it owns this reusable mental model: Agent systems become legible when harness, runtime, memory, tools, skills, and evals are separate layers with explicit proof instead of one... Source: User request, 2026-07-01
- 2026-06-27 | Canonical cluster page created to make agent harness/runtime/memory/evals a single readable synthesis above the individual architecture and tool pages. Source: User request, 2026-06-27
- 2026-06-28 | Added the file-native agent directory model and linked BINEVAL as the current binary-question eval method for making agent quality gates inspectable. Source: X/@omarsar0, 2026-06-27; Source: arXiv:2606.27226
- 2026-06-29 | Added self-improvement receipt evals, Loopy loop-crafting, and Red Queen evaluator hardening as the file-native extension. Source: User request, 2026-06-29
- 2026-06-30 | Added local-vs-hosted coding agent comparison as an eval route: compare the same receipt fixture across the model, runtime, and harness rather than arguing from provider branding. Source: X/@rasbt, 2026-06-27