Agent Self-Improvement Eval Library

Kevin's agent eval library turns a run into a receipt, scores the receipt against durable expectations, and evolves the evaluator when the agent either fails or saturates the current suite.

File-Native Structure

The local implementation follows the Vercel Eve lesson that agents should be visible as files:

evals/agent-self-improvement/
  suite.json                       # benchmark cases + evaluator-evolution rules
  examples/passing-receipt.json    # fixture receipt
scripts/run-agent-self-evals.ts    # scorer
skills/productivity/agent-eval-library/SKILL.md
skills/productivity/loopy/SKILL.md

The runner is intentionally deterministic. It does not ask another model to decide whether the agent was good; it checks whether the receipt contains required evidence such as qmd search, skill reads, source verification, image inspection, routed page promotion, index rebuild, qmd embedding, routing doctor, and log writeback.

Why Receipts

An agent trajectory is too large to grade by memory. A receipt compresses the important path facts into a reviewable object:

• events,
• commands and exit codes,
• files touched,
• promoted pages,
• notes about evaluator hardening.

That gives the evaluator a stable input. It also makes missing proof visible: if the agent claims it searched the wiki or inspected an image, the receipt must say so.

How To Run

npm run agent-self-eval
npm run agent-self-eval -- --receipt evals/agent-self-improvement/examples/passing-receipt.json

The suite currently checks five behaviors:

1. brain-first search before external claims,
2. skill-first routing before repeated-work codification,
3. bookmark artifact promotion into durable pages,
4. runnable eval proof,
5. graph closeout with index/qmd/routing/log evidence.

Evaluator Evolution

The suite includes Red Queen rules from Red Queen Gödel Machine:

• failure creates a new deterministic check or tighter receipt requirement,
• receipt loopholes become negative fixtures,
• saturated scores add harder held-out cases,
• broad always-passing checks split into atomic BINEVAL questions,
• cost/minimality assertions prevent "win by doing everything."

This is the local answer to reward hacking: the evaluator is not frozen.

Eval Source Map

The Awesome Evals thread and repo sharpen the local suite's design rule: every check must name its evaluated object before choosing a scorer. Final text can use binary rubric questions, structured output can use deterministic assertions, agent work should prefer trajectory receipts and state diffs, and self-improvement needs evaluator-evolution cases that the current judge would miss. Source: X/@xdotli, 2026-06-24; Source: GitHub benchflow-ai/awesome-evals, accessed 2026-06-30

For Kevin's agent receipts, this means the suite should keep adding narrow checks for actual misses: missing wiki search, unread skill, skipped visual artifact, unstated version snapshot, weak source verification, missing log/index/qmd closeout, or claims that a broad doctor proves a narrower requirement it does not cover. Those are trajectory failures, not prose failures, so they belong in deterministic receipt checks before any LLM judge.

Concept Position

---

Timeline

• 2026-07-01 | Concepts category refresh added this page to the Brain, memory, and retrieval family, linked it to Concept System Map, and kept it standalone because it owns this reusable mental model: Kevin's agent eval library turns a run into a receipt, scores the receipt against durable expectations, and evolves the evaluator when the... Source: User request, 2026-07-01
• 2026-06-30 | Folded the Xiangyi Li / BenchFlow eval source map into the local suite model: choose scorers by evaluated object, prefer deterministic/state/trajectory checks before LLM judges, and reserve Red Queen evaluator evolution for self-improving agents. Source: X/@xdotli, 2026-06-24; GitHub benchflow-ai/awesome-evals, 2026-06-30
• 2026-06-29 | Created the first local self-improvement suite and runner after Kevin asked Codex to build an eval library to benchmark itself and self-improve. Source: User request, 2026-06-29