Retrieval Quality

Retrieval quality is the degree to which a search or RAG system returns the right context, at the right granularity, with enough provenance for the downstream answer or agent action to be correct.

Core idea

Retrieval is not "did we get semantically similar chunks?" It is "did the agent receive the context needed to answer or act correctly?"

Good retrieval optimizes five properties:

1. Recall: the needed source appears in the candidate set.
2. Precision: irrelevant context does not crowd out useful context.
3. Granularity: the chunk is large enough to understand and small enough to rank.
4. Freshness: current pages/sources outrank stale ones when facts change.
5. Provenance: the answer can cite where context came from.

RAGAS and ARES both treat RAG evaluation as multi-dimensional rather than a single answer score. RAGAS separates retrieval context, faithful use of context, and final generation quality; ARES evaluates context relevance, answer faithfulness, and answer relevance using synthetic data plus a smaller human-annotated set. Source: RAGAS paper, https://arxiv.org/abs/2309.15217, 2026-06-19; ARES paper, https://arxiv.org/abs/2311.09476, 2026-06-19

Failure mode

Bad retrieval creates confident wrongness with citations nearby. The agent finds pages that sound related, quotes a plausible line, and misses the one source that actually controls the decision.

Common causes:

• embedding-only search misses exact identifiers;
• exact search misses paraphrases;
• stale chunks remain after docs change;
• chunks split away the heading, date, or source;
• reranker optimizes topicality, not answerability;
• retrieval returns many snippets but no surrounding page context;
• final answer is evaluated without evaluating retrieval separately.

Retrieval pipeline

Each stage has a distinct failure surface. If the query rewrite is bad, the retriever never sees the right intent. If candidates are weak, the reranker cannot recover. If assembly strips provenance, the answer cannot be trusted.

Metrics that matter

No single metric is enough. A high-recall system can still waste the context window; a precise system can still miss the controlling document.

Kevin wiki version

The wiki uses Knowledge Compilation to improve retrieval before search even starts. Instead of only retrieving raw source chunks, agents retrieve compiled pages whose current truth has already absorbed prior sources.

Retrieval quality here means:

• qmd search finds the canonical page by slug, alias, and common phrasing;
• _index.md routes the agent to the right section;
• related: and inline wikilinks expose adjacent pages;
• timelines preserve dated evidence;
• build-index and qmd embedding run after edits;
• high-traffic routers prevent orphan concepts;
• stale pages are rewritten, not merely appended.

This is why the wiki has both RAG System Architecture and this page. Architecture names the pipeline; retrieval quality names the evaluation target.

Choosing retrieval strategy

Visual retrieval

PixelRAG adds a separate retrieval mode: render the source, index screenshot tiles, and let a vision-language model read the retrieved pixels. This improves answerability when the relevant information lives in layout, tables, charts, diagrams, dashboards, or PDF pages that text parsers flatten or drop. It is not a universal replacement for text retrieval; it is the right mode when the visual arrangement carries the answer.

The evaluation boundary is rendered state. If the browser cannot see content because it is behind login, hidden in a collapsed section, blocked by a captcha/paywall, or still loading, screenshot retrieval will faithfully index the wrong state. Good visual retrieval therefore includes browser-state setup, wait conditions, and explicit page actions before capture. Source: PixelRAG, 2026-06-30

Design implications

Agent products should show retrieval evidence:

• top sources used;
• skipped near-matches when useful;
• freshness dates;
• citation confidence;
• missing-source warning;
• "open full context" affordance.

Search UX should optimize answerability, not just matching. The best result is the one that lets the agent take the next correct step.

Failure modes

• hiding retrieval behind an answer with no sources;
• ranking old docs above current compiled truth;
• using vector search for exact command names;
• using keyword search for conceptual paraphrases;
• measuring answer quality while retrieval silently fails;
• treating citations as decoration rather than proof.

---

Timeline

• 2026-06-30 | Added visual retrieval as a first-class retrieval strategy after deep-reviewing PixelRAG. The durable distinction: screenshot-tile retrieval improves layout-heavy answerability, but browser-state setup remains part of retrieval quality. Source: X bookmark artifact audit, 2026-06-30
• 2026-06-30 | Added LiteParse as the local document QA route: parse once, search bounded windows, and escalate to screenshots or visual retrieval only when text/layout extraction cannot answer. Source: X bookmark artifact audit, 2026-06-30
• 2026-06-19 | Created as the missing evaluation concept behind qmd/wiki search, RAG architecture, context engineering, and retrieval-augmented agents. Grounded against RAGAS and ARES. Source: User request; RAGAS; ARES; local qmd/RAG pages, 2026-06-19