Agent Memory System Architecture

The memory subsystem moves facts between volatile working context and durable long-term stores. Its quality is determined by extraction, consolidation, retrieval, and conflict handling.

The cognitive taxonomy lives in Agent Memory Patterns. This page is the system design.

Kevin Retrieval Stack

Kevin's current memory/retrieval stack has four distinct lanes:

The boundary matters. qmd answers "what is the compiled truth?" Hindsight answers "what did Hermes remember across runs?" Honcho answers "what model of Kevin or a peer is useful here?" Graphify answers "what touches what in this repo?" None of the runtime lanes replace the wiki. A runtime memory becomes durable only after it is written back to a page, skill, automation, config file, project AGENTS.md, or log.

Two Tiers

The architecture's job is not to remember everything. It is to move the right thing to the right tier at the right time.

Agent Company OS names the runtime version of this as a memory skill overlay: inject the relevant facts, examples, runbooks, and constraints for a worker's department without giving it the whole workspace. That is the practical boundary between useful long-term memory and context stuffing. Source: X/@DerekNee image, 2026-06-25

Storage Backends

• Document/wiki store: inspectable compiled truth and timelines.
• Vector store: fuzzy semantic recall.
• Keyword index: exact identifiers, commands, names, and rare terms.
• Graph layer: entities and relationships.
• Key-value store: exact settings, preferences, and state.
• Append log: replayable episodic history.

Production systems are hybrid. Kevin's wiki intentionally biases toward inspectable documents plus qmd retrieval, because the agent and Kevin both need to read the memory.

Latent Briefing adds a lower-level handoff path for multi-agent systems: instead of passing the orchestrator's whole reasoning trace as text, compact the relevant KV-cache state for the worker's task. That is not a replacement for durable memory; it is a runtime transfer optimization for "what the next agent needs right now." Use it as an inference/memory-system pattern, not as a wiki-storage pattern. Source: X/@RampLabs and technical source review, 2026-07-04

Write Path

Important invariants:

• Do not append duplicate facts when an existing page should be updated.
• Preserve evidence in timelines.
• Use aliases/frontmatter to resolve identity.
• Route recurring procedures to skills or automations.
• Rebuild indexes after broad writes.

Module Decomposition

The Are We Ready For An Agent-Native Memory System? paper provides a system-level decomposition that should be used when designing or auditing memory products:

Do not evaluate "memory" as one feature. Evaluate the module that is likely to fail. In Kevin's wiki, extraction and maintenance are the risky modules; storage is intentionally boring markdown, and retrieval is qmd plus graph navigation.

Read Path

1. Search catalog/qmd.
2. Read likely pages.
3. Follow related/backlinks if the task crosses domains.
4. Filter for current, source-backed claims.
5. Inject only the useful slice into the active context.

This read path is Context Engineering in action: select before stuffing.

Conflict Handling

Memory systems rot when they cannot represent changed facts. The wiki pattern handles this by separating compiled truth from timelines:

• Update compiled truth to the current best synthesis.
• Append a timeline entry for the new evidence.
• If an older claim was wrong, add a correction entry rather than silently erasing history.
• Merge duplicate entities instead of maintaining parallel pages.

Evaluation

Evaluate memory on four axes:

• Write precision: did the agent store the right fact in the right place?
• Dedup/consolidation: did it avoid parallel contradictory memory?
• Retrieval recall: can a later agent find it?
• Usefulness: did the retrieved memory change behavior?

For agent-native systems, add the paper's finer-grained measures:

• Representation fidelity: does the stored form preserve the detail needed later?
• Retrieval precision: did irrelevant memory stay out of the working context?
• Update correctness: did new evidence change the right record without corrupting old evidence?
• Long-horizon stability: does memory still help after many updates?
• Cost-performance: does maintenance cost scale with the value of the memory?

The paper's cost result is directionally useful: localized maintenance can beat global reorganization. That supports Kevin's page-by-page compiled-truth updates and targeted merge protocol over periodic whole-wiki rewrites. Source: arXiv 2606.24775v1, 2026-06-23

---

Timeline

• 2026-07-04 | Added Latent Briefing as the representation-level context handoff pattern for multi-agent memory: compact task-relevant KV-cache state instead of re-sending the full reasoning trace as text. Source: X/@RampLabs, 2026-04-10
• 2026-06-25 | Added the four-module data-management decomposition and evaluation axes from "Are We Ready For An Agent-Native Memory System?" Source: arXiv 2606.24775v1
• 2026-06-25 | Added Agent Company OS's memory skill overlay as the runtime pattern for injecting scoped memory into workers without broad context stuffing. Source: X/@DerekNee, 2026-06-25
• 2026-06-18 | Expanded with wiki-specific backend mapping, write/read paths, conflict handling, and evaluation axes. Source: User request, 2026-06-18
• 2026-06-02 | Page created. Captured working-vs-long-term tiering, vector/graph/KV/log backend mix, extract-consolidate-dedup write path, retrieve-rank-page read path, and durability implications. Source: compiled from Mem0, MemGPT/Letta, CoALA