The Eval Loop (Slop Is an Output Problem)

Slop is an output problem, not a prompt problem. Better prompts, bigger models, memory, and context files are all input-side fixes for a thing that was never broken. The missing layer is measurement: an eval loop that scores every output against a standard before (and after) it ships. Almost nobody runs this loop — that's the slop.

The reviewed Hermes-operator artifact independently compresses this page's thesis: "slop is an output problem, not a prompt problem." Its diagram contrasts input-side tweaks (better prompts, bigger model, memory, context files) with the missing output gate: score every run 0-1 against a benchmark, ship only above threshold, and feed failures/thumbs-down back into the case set. Treat it as corroborating operator evidence for the existing Hermes Agent (Nous Research) eval-loop route, not a separate new framework. Source: X/@EXM7777 and local artifact review, 2026-07-03

The core argument

A prompt is a hypothesis, the output is the result, and an eval is the only thing that closes the loop between them. Because the model is non-deterministic, the same prompt run twice gives two answers — so even a perfect prompt produces slop on some fraction of runs, and you have no idea which runs until a user is staring at one. A perfect prompt isn't a quality guarantee; it's "a slightly better coin flip, and you're shipping every flip." Source: User essay, 2026-06-01

People keep reaching for prompts because the prompt is the only lever they can see and editing it feels like control. Measurement is invisible and unsexy, so the whole conversation stays stuck on the one lever that can't solve the problem alone. The people whose AI output is consistently clean aren't better at prompting — they have a second lever: they measure every output against a standard before it ships. Source: User essay, 2026-06-01

The analogy: a factory shipping defects doesn't have a worker problem, it has a quality-control problem — nobody checks the output before it leaves the building. You built half a system (the generate half) and blamed the half that works.

Two places slop lives — same disease, same cure

• Content output — tweets, articles, emails, posts. Slop looks technically fine and completely hollow; it dies loudly in public.
• Product output — the AI feature, agent, chatbot, extraction pipeline. Slop is a confident wrong answer, a hallucinated number, a broken JSON payload, a tone drift three deploys later; it scales in silence as users quietly leave.

Both are un-measured AI output going straight to an audience with no gate in between. The only difference is stakes and visibility. One quality system grades both.

The eval loop = unit tests for the non-deterministic

You are not testing whether the code runs; you are testing whether the output is good, across enough cases that one bad run can't hide. It runs in three places:

1. Before you ship — run the new prompt/model against a saved case set and confirm it didn't get worse (regression testing; stops a change that fixes one thing and silently breaks three).
2. At runtime — score output as it's generated and let conditional logic catch failures before the user sees them (guardrail).
3. In production — score a sample of real executions continuously, so you see quality degrading the day it starts, not the week a client complains.

The moment quality becomes a number, slop stops being a feeling and becomes a bug. "You can't debug a vibe; you can debug a score that dropped from 0.82 to 0.61."

The benchmark: three parts (content and product)

Skip any one part and you have a wish, not a gate. A vague rubric ("is this good and engaging") produces a vague score; a specific rubric produces a score you can trust — "the judge inherits your taste only if you actually write your taste down." The threshold only works if you never let a 0.6 through because you liked it (takes the late-night ego out of the decision). Source: User essay, 2026-06-01

BINEVAL is the current concrete method for making LLM-as-judge rubrics inspectable: break a criterion into atomic yes/no questions, answer them independently, and aggregate the verdicts. Use it when a holistic judge score hides why an output failed; keep deterministic validators first where the task is symbolic. Source: arXiv:2606.27226, 2026-06-25

Diana Hu (YC) compresses the same point to a slogan: "taste is just an eval you haven't written down yet." You make taste legible by reading 1,000+ real failure traces, grouping them into a failure taxonomy (what researchers call open / axial coding), and treating that taxonomy as the eval suite. Generic benchmarks (MMLU, HLE) measure "is it smart"; you need to measure "did it do the job" — different questions. Because failure modes are domain-specific, "evals are the moat now, not code" (same models, leaked prompts, scrapeable data), and that is precisely why vertical AI companies can beat general models. A benchmark you can saturate but can't tie to customer value is a vanity metric. Source: X/@sdianahu, 2026-06-04

The compounding arrow (the whole game)

Failures + thumbs-down on production runs become new test cases written back into the suite. That failed run becomes a permanent check, so the quality floor rises on its own every week. In the diagram this is the dashed arrow from "failures + thumbs-down" back to "benchmark" — "this arrow is the whole game," and almost nobody runs it.

Red Queen Gödel Machine adds the next constraint for agent self-improvement: the evaluator itself cannot stay frozen. A fixed judge invites reward hacking and saturation. The local rule is: when an agent fails, add a more precise deterministic check; when it saturates the suite, add harder held-out cases or split broad criteria into BINEVAL-style atomic questions. The implementation route is Agent Self-Improvement Eval Library, and Loopy owns the bounded loop around the evaluator.

Why it matters here

This is the same gate-shape as teacher-mode (verify against a standard, fail → re-teach, completion gate) and the verification-before-completion ethos — evidence before assertions, applied to output quality. It is the cure for the vibe-driven failure mode, and it is what Hermes is built to run as a standing loop (skills as the judge, memory as ground truth, cron as the monitor, approval buttons as the gate). For Kevin's own content, the Security and Review Skills + kevin-voice skills are the content rubric/judge — the eval loop applied to his writing. For user-facing apps that must block harmful text/images, OpenAI Moderation API is a complementary runtime harm classifier (policy signals); it does not replace custom rubrics for product quality. Source: compiled, 2026-06-01; updated 2026-06-12

---

Timeline

• 2026-07-03 | Deep-reviewed the Hermes anti-slop operator diagram and attached it as artifact evidence for the same eval-loop thesis: input-side prompt/model/context changes are half a system without an output score gate and failure-feedback loop. Source: raw/x-bookmarks/enriched/2061086049326256356.json
• 2026-06-29 | Added the Red Queen evaluator-evolution rule and linked the local self-improvement eval runner as the implementation route. Source: arXiv:2606.26294; User request, 2026-06-29
• 2026-06-26 | Linked Awesome Evals as the source map for agent-eval papers, talks, tools, benchmarks, LLM-as-judge patterns, trajectory grading, and RL-environment thinking. Source: GitHub benchflow-ai/awesome-evals, 2026-06-26
• 2026-06-28 | Added BINEVAL as the binary-question method for turning vague LLM-as-judge rubrics into inspectable criterion-level verdicts. Source: arXiv:2606.27226
• 2026-06-18 | Added Agent Trajectory Evaluation as the agent-run extension of this output-quality loop. Source: whole-wiki graph audit, 2026-06-18
• 2026-06-15 | Phase 7 (Review) mapped to the eval loop: Matt Pocock's review skill checks Standards (rubric) + Spec (matches the PRD) in parallel sub-agents, and tdd's red-green-refactor is the same gate at the unit level. Source: github.com/mattpocock/skills, 2026-06-15
• 2026-06-15 | Satya Nadella's essay "A frontier without an ecosystem is not stable" corroborates the thesis from firm economics: "private evals should capture whether a model is actually improving against outcomes that matter to the business (not just external benchmarks)," plus private RL on real internal traces — i.e., the eval loop + compounding arrow as the firm's IP. See Own the Learning Loop. Source: X/@satyanadella, https://x.com/satyanadella/status/2066182223213293753; surfaced by User, 2026-06-15
• 2026-06-12 | Cross-linked OpenAI Moderation API as complementary harm-classifier for user-facing runtime guardrails (distinct from custom eval rubrics). Source: User capture, 2026-06-12
• 2026-06-04 | Diana Hu (YC) thread corroborates the thesis: "taste is just an eval you haven't written down yet"; evals come from error analysis (1,000+ traces → failure taxonomy via open/axial coding), are domain-specific, and are "the moat now, not code." 624 likes, 566 bookmarks. Source: X/@sdianahu, 2026-06-04
• 2026-06-01 | Page created from Kevin's essay "How To Fix AI Slop (Using Hermes)." Compiled the output-vs-input thesis, the non-determinism argument ("a perfect prompt is a slightly better coin flip"), content vs product slop as one disease, the three run-sites (pre-ship regression / runtime guardrail / production sampling), the three-part benchmark, and the failures→new-test-cases compounding arrow. Diagram saved to wiki/assets/eval-loop-diagram.png. Source: User, 2026-06-01