Agent Memory Architectures: Vector, Graph & Episodic

TL;DR — Vector memory is fast and cheap but forgets relationships. Graph memory captures structure but adds 200-400ms per lookup. Episodic buffers keep temporal order but blow up your context window. Production agents in 2026 use all three in a layered architecture. Here’s how to wire them together without losing your mind.

The Problem Nobody Warned You About

You build an agent. It works great in demos. Then a user asks it about something they mentioned three sessions ago, and it stares back blankly like you’ve never met.

Choosing an agent memory architecture isn’t a prompt engineering problem — it’s a foundational design decision most teams get wrong. Most agent frameworks treat memory as an afterthought: a vector store you bolt on and hope for the best. But in 2026, the gap between “agent with memory” and “agent that actually remembers well” has become the primary differentiator between toys and production systems.

I’ve spent the last few months wiring memory into agents across different frameworks — Hermes, OpenClaw, LangGraph, custom builds. Here’s what I’ve learned about what actually works, what fails silently, and where the field is heading after Anthropic’s “Dreaming” announcement shook things up in May.

Three Memory Architectures, Three Failure Modes

1. Vector Memory (Semantic Recall)

The most common pattern. You embed conversation turns or extracted facts, store them in a vector database (Pinecone, Qdrant, Weaviate, pgvector), and retrieve by cosine similarity at query time.

How it works in practice:

# Simplified vector memory write
from openai import OpenAI

client = OpenAI(base_url="https://api.sandbase.ai/v1", api_key="...")

def store_memory(text: str, metadata: dict):
    embedding = client.embeddings.create(
        model="openai/text-embedding-3-small",
        input=text
    ).data[0].embedding
    
    vector_db.upsert(id=uuid4(), vector=embedding, metadata=metadata, text=text)

def recall(query: str, top_k: int = 5):
    q_embedding = client.embeddings.create(
        model="openai/text-embedding-3-small",
        input=query
    ).data[0].embedding
    
    return vector_db.query(vector=q_embedding, top_k=top_k)

Where it shines: Recency-biased recall. “What did we discuss about authentication?” → finds the relevant chunks fast. Latency is typically 50-150ms for a top-5 query on managed services.

Where it breaks:

• Multi-hop reasoning fails. “What was the name of the library that John recommended for the project Sarah mentioned?” requires connecting three entities. Vector similarity treats each fact as isolated — there’s no relational structure.
• Temporal confusion. A user says “I moved to Berlin” in January, then “I moved to Tokyo” in March. Vector recall might surface the Berlin fact if its embedding is closer to the current query. There’s no inherent “this supersedes that” logic.
• Scale degradation. A recent proof showed that embedding-based retrieval has a mathematical property: the same geometry that makes it work at small scale forces it to forget at large scale. The bigger the memory, the worse recall gets.

Real numbers: Mem0 reports 7-10 second response times in production workloads with large memory stores. That’s not retrieval latency — that’s the full pipeline including extraction, dedup, and re-ranking.

2. Graph Memory (Structural Recall)

Instead of embedding flat text, you extract entities and relationships into a knowledge graph. Zep’s Temporal Knowledge Graph is the most mature implementation; Mem0 added a graph layer in early 2026.

The core idea:

User → [works_at] → Acme Corp
User → [prefers] → Python
User → [deployed_to] → AWS us-east-1
Acme Corp → [uses] → Kubernetes

When the agent needs to answer “What infrastructure does my company use?”, it traverses the graph: User → works_at → Acme Corp → uses → Kubernetes. This multi-hop reasoning is impossible with pure vector similarity.

Where it shines:

• Relationship queries (“Who introduced me to that tool?”)
• Preference consistency (“Always use TypeScript” stays sticky even 50 sessions later)
• Contradiction detection (new facts can explicitly override old ones via temporal edges)

Where it breaks:

• Extraction is expensive and lossy. Turning natural language into structured triples requires an LLM call per conversation turn. That’s 200-400ms added latency and ~500-2000 tokens burned on extraction alone.
• Schema rigidity. You either predefine entity types (limiting what can be captured) or let the LLM decide (introducing inconsistency — is it “Python” or “python” or “Python 3.12”?).
• Cold start problem. Graph memory is useless until populated. The first 5-10 sessions feel no different from a stateless agent.

Real numbers: Zep reports 4-second average latency with graph lookups in production. Mem0’s hybrid (vector + graph) claims 91% faster responses than pure vector, but their benchmark context is specific to preference recall tasks.

3. Episodic Memory (Temporal Buffers)

This is the closest to how human memory actually works — full episodes (complete sessions or task sequences) stored chronologically, summarized progressively, and retrieved as narratives rather than isolated facts.

Anthropic’s “Dreaming” feature (launched May 6, 2026) is the highest-profile implementation. Between active sessions, Claude agents review past work, extract patterns, identify recurring mistakes, and rewrite their own memory. Harvey reported a 6x improvement in agent task-completion rates after enabling it for legal workflows.

The pattern:

Session 1 (full transcript) → [summarize] → Episode 1 (compressed)
Session 2 (full transcript) → [summarize] → Episode 2 (compressed)
...
Episodes 1-10 → [consolidate] → Meta-episode (higher-level patterns)

Where it shines:

• Temporal ordering is preserved. “Last time we tried X and it failed, so this time…”
• Narrative coherence. The agent can explain its own history and learning.
• Self-improvement. Patterns emerge from episode review that individual facts can’t surface.

Where it breaks:

• Context window bloat. Even compressed episodes eat tokens. Loading 20 session summaries at 200 tokens each = 4000 tokens of preamble before you’ve even started the current task.
• Summarization loss. Every compression step loses detail. The exact error message from session 7 might be exactly what you need in session 15, but it got summarized away.
• Consolidation cost. Anthropic’s Dreaming runs as a background process — it’s essentially an extra LLM call per session that you pay for but the user never sees.

The 2026 Production Pattern: Layered Memory

Nobody ships one memory type alone anymore. The dominant architecture looks like this:

Layer	Storage	Retrieval Latency	Use Case
Hot (short-term)	In-context window	0ms (already loaded)	Current session state
Warm (mid-term)	Markdown files or KV store	10-50ms	User profile, preferences, active project context
Cold (long-term)	Vector DB + Graph + Episodes	100-500ms	Historical recall, relationship queries, pattern detection

The flow on each turn:

1. Load Hot context (current conversation + system prompt)
2. Load Warm context (user.md, project notes — always included, ~500-1000 tokens)
3. If the query needs historical context → route to Cold layer
4. Cold layer does parallel lookups: vector similarity + graph traversal + episodic search
5. A Memory Router (small LLM call or classifier) picks which results are relevant
6. Inject selected memories into the context, generate response
7. Post-response: extract facts for graph, embed for vector, append to episode log

What This Looks Like in Practice

Hermes Agent implements this as their five-pillar system: user.md (warm), memory.md (warm/cold), skills (episodic patterns), and the soul file (identity). OpenClaw uses SOUL.md + layered memory files + vector-indexed long-term store. (If you’re weighing these two frameworks, see our Hermes Agent vs OpenClaw comparison for the full breakdown.)

The key insight both frameworks share: warm memory is always loaded, cold memory is retrieved on-demand. This keeps base latency low while still having deep recall available when needed.

# Hermes-style memory configuration
memory:
  warm:
    - user.md          # Always loaded. Facts about the user.
    - project.md       # Current project context. Swapped per workspace.
  cold:
    vector_store: qdrant
    graph_store: neo4j  # Optional, adds relationship queries
    episode_log: sqlite # Compressed session summaries
  consolidation:
    schedule: "0 3 * * *"  # Nightly consolidation (like Dreaming)
    model: anthropic/claude-sonnet-4  # Cheaper model for background work

Decision Framework: Which Architecture to Prioritize

Stop thinking “which one should I use?” and start thinking “which one should I build first?”

Start with warm Markdown files if:

• Your agent serves one user or a small team
• Memory needs are mostly preferences and project context
• You want zero infrastructure overhead
• You’re okay with manual curation (user edits their own memory file)

Add vector memory when:

• Conversation history exceeds what fits in context (~50+ sessions)
• Users ask “what did we discuss about X?” frequently
• You need fuzzy matching (“something about deployment… maybe kubernetes?”)
• You’re okay with 7-10 second latency for large stores (Mem0) or 4 seconds (Zep)

Add graph memory when:

• Your domain has real relationships (people → teams → projects → tools)
• Users ask multi-hop questions (“what does Sarah’s team use for CI?”)
• You need contradiction resolution (superseding old facts)
• You can afford the extraction cost (~500 tokens per turn)

Add episodic memory when:

• Your agent runs recurring workflows (daily standups, weekly reports)
• Self-improvement matters (the agent should get better at its job over time)
• Temporal ordering matters (“we tried X before and it didn’t work”)
• You’re building for long-term relationships (months/years)

The Latency Budget Reality

Here’s the uncomfortable truth: every memory layer you add costs latency. A typical budget breakdown:

Operation	Latency	Token Cost
Load warm files	5-10ms	500-1500 tokens (always)
Vector retrieval (top-5)	50-150ms	300-800 tokens
Graph traversal (2-hop)	200-400ms	100-400 tokens
Episode retrieval	50-100ms	200-600 tokens
Memory router decision	100-200ms	100-200 tokens
Total cold path	400-850ms	700-2000 tokens

That 400-850ms is on top of your LLM response time. For a chatbot where users expect sub-2-second responses, this is tight. For an async coding agent that runs for minutes, it’s negligible.

Practical optimization: Don’t hit cold memory on every turn. Use a lightweight classifier (or even keyword matching) to decide whether the current query needs historical recall. Most turns in a conversation are follow-ups that only need hot context.

What’s Coming Next

Three trends worth watching:

1. Memory-as-infrastructure is consolidating. Mem0, Zep, and Letta are the three standing players. Cloudflare launched Agent Memory. Expect 2-3 to survive, the rest to merge.

2. Consolidation becomes standard. After Anthropic’s Dreaming and Google’s Memory Bank at I/O 2026, “background memory processing” is no longer exotic. Every serious framework will have it by end of year.

3. The vector ceiling is real. That mathematical proof about embedding geometry degrading at scale isn’t going away. Hybrid architectures (vector + graph + episodic) aren’t a nice-to-have — they’re the only path that scales past ~100K memories without recall degradation.

FAQ

Q: Can I just use a bigger context window instead of building memory?

You can, until you can’t. Gemini’s 1M-token window or Claude’s 200K let you stuff a lot of history in. But at $3-15 per million input tokens, loading 500K tokens of history on every turn gets expensive fast. Memory systems exist to load the right 2K tokens instead of all 500K tokens.

Q: Is Mem0 or Zep better?

Different strengths. Mem0 excels at user preference recall and has better vector+graph hybrid search. Zep’s Temporal Knowledge Graph is stronger for multi-hop relationship queries and temporal reasoning. Both are 3-10 seconds for full pipeline. If you mostly need “remember what this user likes,” go Mem0. If you need “trace the chain of decisions that led to this outcome,” go Zep.

Q: How does Anthropic’s Dreaming compare to Hermes’ skill system?

Dreaming is passive consolidation — it reviews and curates. Hermes’ skills are active extraction — it writes reusable procedures. Dreaming makes the agent remember better. Skills make it act faster on recurring tasks. They’re complementary patterns.

Q: What’s the minimum viable memory for a production agent?

A user.md file loaded on every session. Seriously. A Markdown file with 20 lines of context about the user (“prefers Python, works on e-commerce platform, timezone UTC+8”) does more for perceived intelligence than a sophisticated vector store that’s 80% irrelevant noise.

Q: How do I test if my memory is actually helping?

Run the same 10-turn conversation twice: once with memory, once without. Score the responses for relevance and coherence (use an LLM as judge). If memory doesn’t improve the score by at least 20%, your retrieval is broken — you’re loading noise, not signal.