agent-eval-library

Generated source view for the actual executable productivity/agent-eval-library skill. The durable routing article is Agent Operations Skills. Source: skills/productivity/agent-eval-library/SKILL.md

Runtime Source

Bundled Resources

No bundled resource files.

Description

Build, run, and evolve Kevin's agent self-evaluation library. Use when the user says "benchmark yourself", "self improve", "agent eval library", "evaluate the agent", "make the judge harder", "Red Queen", "Gödel Machine", "eval the loop", or asks to turn an agent failure into a benchmark, receipt, rubric, or regression case.

Skill Source

---
name: agent-eval-library
description: Build, run, and evolve Kevin's agent self-evaluation library. Use when the user says "benchmark yourself", "self improve", "agent eval library", "evaluate the agent", "make the judge harder", "Red Queen", "Gödel Machine", "eval the loop", or asks to turn an agent failure into a benchmark, receipt, rubric, or regression case.
origin: personal
source_slug: agent-eval-library
---

# Agent Eval Library

Use this skill when an agent needs to benchmark its own behavior, not merely finish the task.

The local eval library lives at:

- `evals/agent-self-improvement/suite.json` — benchmark cases and evaluator-evolution rules
- `scripts/run-agent-self-evals.ts` — deterministic receipt scorer
- `evals/agent-self-improvement/examples/` — example receipts and fixtures
- `wiki/concepts/agent-self-improvement-eval-library.md` — architecture and operating model

## Core Model

An agent run should leave a small JSON receipt of what mattered:

```json
{
  "runId": "short-run-id",
  "events": [{ "type": "qmd-search", "detail": "searched wiki first" }],
  "commands": [{ "cmd": "npm run build-index", "exitCode": 0 }],
  "filesTouched": ["wiki/tools/loopy.md"],
  "promotedPages": ["wiki/tools/loopy.md"],
  "notes": ["why this run changed the evaluator"]
}
```

Then run:

```bash
npm run agent-self-eval
npm run agent-self-eval -- --receipt evals/agent-self-improvement/examples/passing-receipt.json
```

The first command validates the suite. The second scores a run receipt.

## Evaluation Loop

1. **Capture the receipt.** Record brain-first search, skill reads, source verification, edits, checks, and graph writeback.
2. **Run the suite.** Use `npm run agent-self-eval -- --receipt <path>`.
3. **Fix the agent or the work.** A failed assertion means the behavior was missing or the receipt hid the evidence.
4. **Evolve the evaluator.** If the agent passes too easily, add a harder held-out case from a real failure.
5. **Commit the test before the behavioral fix** when the failure is recurring.

## Choose The Grader By The Object

Before adding a case, decide what is actually being graded. Use
`wiki/tools/awesome-evals.md` as the source map and prefer the narrowest grader
that proves the behavior.

| Object being graded | Default grader | Use when |
| --- | --- | --- |
| Structured final output | deterministic assertions | JSON shape, citations, links, exact labels, policy fields, build artifacts |
| Subjective final output | binary LLM judge after error analysis | taste, usefulness, tone, critique quality, when deterministic checks are insufficient |
| Tool trajectory | trajectory match plus trace review | known workflows where the expected tool sequence matters |
| Agent outcome | environment or state diff | DB/files/API state changed; the transcript can lie |
| Harness quality | run receipt plus trace spans | tool choice, context loading, approvals, costs, closeout proof |
| Benchmark integrity | contamination, label-error, and saturation checks | comparing models or publishing capability claims |
| Self-improving agent | evolving evaluator with held-out negatives | the agent can learn to satisfy a fixed judge |

Concrete rules from the current eval source map:

- Build judges from real failure analysis, not from imagined rubrics.
- Keep pass/fail criteria binary; split vague standards into BINEVAL-style yes/no questions.
- Track true-positive and true-negative rates separately when validating a judge.
- Prefer state checks, unit tests, schema checks, and verifiable rewards before LLM judges.
- Distinguish `pass@k` capability from `pass^k` reliability before reporting numbers.
- Keep offline regression suites and online production monitoring separate.
- Reuse an eval as an RL reward only after hardening loopholes; training pressure exploits weak verifiers.

## Red Queen Rule

A stable judge becomes a target. When the suite saturates, make the evaluator harder before celebrating:

- split broad checks into BINEVAL-style yes/no assertions
- add a negative fixture that would have fooled the old judge
- add cost, minimality, or tool-ordering checks
- add a production or bookmark-artifact failure as a held-out case

The judge must improve alongside the agent.

## Related Skills

- `loopy` — craft bounded agent loops and loop skills.
- `x-bookmark-absorb` — promote bookmark evidence into pages, skills, tools, playbooks, diagrams, and capsules.
- `skill-creator` — create or prune executable skills after an eval exposes a repeated procedure.
- `agent-iteration-loop` — implementation loop that should emit receipts for substantial work.

---

Timeline

• 2026-07-15 | Generated a browseable source page from the actual executable skill file. Source: skills/productivity/agent-eval-library/SKILL.md