What is agent memory? How AI agents remember across sessions
AI agent memory is what an agent keeps outside its context window and reads back later. Short-term vs long-term memory, memory tools, and a Python demo.

Agent memory is the information an AI agent stores outside the model’s context window and retrieves later, so it can carry facts, decisions and progress across steps, context resets and separate sessions. The model itself remembers nothing between calls; every call starts from what’s in the prompt. Memory is the part of the harness that decides what gets written down, where it lives, and what gets loaded back into the prompt next time.
Why do AI agents need memory?
Because a language model is stateless. Each API call sees only the tokens you send, and the context window that holds them is finite and billed on every turn. An agent working a long task hits two walls. Within a session, the transcript grows until it overflows or the model starts losing details in it. Across sessions, everything is gone unless something saved it.
Anthropic’s Effective context engineering for AI agents (September 29, 2025) names the first wall context rot: as the token count grows, “the model’s ability to accurately recall information from that context decreases.” Its answer is structured note-taking, where “the agent regularly writes notes persisted to memory outside of the context window.” The example in the post is Claude playing Pokémon, keeping tallies like “Pikachu has gained 8 levels toward the target of 10” across thousands of steps and picking up where it left off after its context resets.
What are the types of agent memory?
The cleanest taxonomy comes from CoALA (Cognitive Architectures for Language Agents, Sumers, Yao, Narasimhan and Griffiths, September 2023), which borrowed it from cognitive science:
- Working memory “maintains active and readily available information as symbolic variables for the current decision cycle.” In practice: the current prompt.
- Episodic memory “stores experience from earlier decision cycles.” Logs of past runs, past conversations, what worked and what failed.
- Semantic memory “stores an agent’s knowledge about the world and itself.” Facts: the user’s stack, the team’s conventions, the customer’s plan.
- Procedural memory comes in two forms: implicit, in the model’s weights, and explicit, “written in the agent’s code.” Prompts, tool definitions, skills files.
CoALA also names the operations: retrieval reads long-term memory into working memory, and learning writes to long-term memory. Every memory system I’ve seen is some choice of what to write, when, and how to retrieve it.
Engineers usually collapse this into two buckets. Short-term memory is the conversation and scratchpad for the current task. Long-term memory is anything persisted: files, database rows or a vector index.
How does agent memory work in practice?
Three designs show the range.
Memory as an operating system. MemGPT (Packer et al., October 2023) treated the context window like RAM and external storage like disk. It proposed “virtual context management,” where the agent itself pages information between tiers with function calls, and tested it on documents larger than the context window and on multi-session chat.
Memory as a scored stream. Generative Agents (Park et al., April 2023) put 25 simulated characters in a sandbox town. Every observation went into a memory stream, the agents periodically wrote higher-level reflections, and retrieval scored each memory on three things: recency (exponential decay, factor 0.995 per game hour), importance (the model rates each memory from 1 to 10 when it’s stored) and relevance (cosine similarity of embeddings). Each score is min-max scaled to 0 to 1, and in their implementation all three weights are 1.
Memory as files. Anthropic’s memory tool gives Claude a /memories directory and six commands: view, create, str_replace, insert, delete and rename. It’s client-side: Claude asks for a file operation and your code runs it against whatever storage you choose. When the tool is enabled, the API adds an instruction to the system prompt telling Claude to view its memory directory before doing anything else, and to “ASSUME INTERRUPTION: Your context window might be reset at any moment.” Claude Code’s CLAUDE.md instructions file, and the persistent memory scope its subagents can be given, are the same idea inside a coding tool.
What does memory retrieval look like in code?
Here’s the Generative Agents retrieval function on a toy memory store. Relevance uses bag-of-words cosine similarity in place of real embeddings so it runs with the standard library; recency, importance, the 0.995 decay, min-max scaling and equal weights follow the paper.
# memory_retrieval.py: Generative Agents-style memory retrieval, stdlib only.
# score = recency + importance + relevance, each min-max scaled to [0, 1],
# all weights 1, recency decay 0.995 per hour (Park et al., 2023).
# Relevance here is bag-of-words cosine, standing in for embedding similarity.
import math, re
from collections import Counter
NOW = 500 # hours since the agent started
MEMORIES = [ # (text, hour last accessed, importance 1-10 as an LLM would rate it)
("User prefers answers in Python, not JavaScript", 20, 7),
("User's production database is Postgres 16 with pgvector", 120, 8),
("User said good morning", 495, 1),
("Deploy failed because the Postgres migration timed out", 480, 9),
("User asked for a recipe for dal", 490, 2),
("Team decided to keep embeddings in Postgres, not Pinecone", 300, 8),
]
def vec(text):
return Counter(re.findall(r"[a-z0-9]+", text.lower()))
def cosine(a, b):
dot = sum(a[k] * b[k] for k in a)
return dot / (math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values())) or 1)
def minmax(xs):
lo, hi = min(xs), max(xs)
return [(x - lo) / (hi - lo) if hi > lo else 0.0 for x in xs]
def retrieve(query, k=3):
q = vec(query)
rec = minmax([0.995 ** (NOW - t) for _, t, _ in MEMORIES])
imp = minmax([i for _, _, i in MEMORIES])
rel = minmax([cosine(q, vec(m)) for m, _, _ in MEMORIES])
scored = sorted(zip(MEMORIES, rec, imp, rel), key=lambda r: -(r[1] + r[2] + r[3]))
print(f"query: {query}")
for (text, _, _), r, i, v in scored[:k]:
print(f" {r + i + v:.2f} (rec {r:.2f} imp {i:.2f} rel {v:.2f}) {text}")
retrieve("Why is the Postgres deploy slow?")
retrieve("Write the fix in Python")
Output (Python 3.13.11):
query: Why is the Postgres deploy slow?
2.92 (rec 0.92 imp 1.00 rel 1.00) Deploy failed because the Postgres migration timed out
1.57 (rec 0.07 imp 0.88 rel 0.63) User's production database is Postgres 16 with pgvector
1.50 (rec 0.31 imp 0.88 rel 0.31) Team decided to keep embeddings in Postgres, not Pinecone
query: Write the fix in Python
2.39 (rec 0.92 imp 1.00 rel 0.47) Deploy failed because the Postgres migration timed out
1.75 (rec 0.00 imp 0.75 rel 1.00) User prefers answers in Python, not JavaScript
1.63 (rec 0.31 imp 0.88 rel 0.44) Team decided to keep embeddings in Postgres, not Pinecone
The first query works as you’d hope. The second one shows the trade-offs in the formula. The Python preference is the most relevant memory, but it’s old, so recency drags it to second place. The failed deploy wins on recency and importance even though its relevance comes mostly from the word “the”, which a real embedding model would mostly ignore. “Good morning” and the dal recipe never surface: low importance does its job. Tuning those three weights is most of the work in a memory system like this, and the right values depend on whether your agent’s world changes by the hour or by the quarter.
Agent memory vs RAG: what’s the difference?
The retrieval half looks the same: score stored text against a query, put the top results in the prompt. That’s RAG, and memory systems often use a vector database to do it.
The difference is who writes. In RAG, people write the corpus and the agent only reads. In agent memory, the agent writes as well: it decides what was worth keeping, updates facts that changed and deletes what’s stale. That write path is where memory systems succeed or fail. An agent that saves everything rebuilds the overflowing context it was trying to escape, one retrieval at a time. An agent that saves the wrong things carries a mistaken fact forward into every future session.
Where does agent memory go wrong?
Stale facts. A memory that was true in March is still retrieved in October. Store when each fact was written and prefer updates over appends; the memory tool’s str_replace exists for this.
Poisoned memory. If an agent reads a malicious web page or document and writes an instruction from it to memory, the injection now survives across sessions. Treat memory writes from untrusted input the way you treat prompt injection: limit what can be written, and review it.
Path traversal and privacy. The memory tool docs warn that a path like /memories/../../secrets.env can escape the directory if your handler doesn’t validate it, and recommend canonicalizing paths and rejecting traversal sequences. They also recommend size caps and expiring files that haven’t been read in a long time. From a data-platform and privacy-compliance background, this is the line I’d circle: agent memory is user data. It needs a retention policy, a delete path and per-user isolation from day one, not after the first deletion request.
Memory as a crutch. Long-lived memory hides bad context hygiene. If a session needs constant retrieval to function, the fix may be smaller tasks or subagents, not a bigger memory.
What I would do
- Start with files. A per-project or per-user notes file the agent reads at the start and updates at the end covers more use cases than a vector store, and you can read it yourself.
- Separate the types: a facts file (semantic), a progress log (episodic), and instructions (procedural). They change at different rates and deserve different write rules.
- Write on milestones, not every turn. Anthropic’s memory tool docs suggest updating a progress log at the end of each session and marking a feature complete only after verification.
- Add scored retrieval, recency plus importance plus relevance, only when the files get too big to load whole.
- Give memory a retention policy and a delete command before launch, and log every write so you can audit what the agent decided to remember.
FAQ
What is short-term vs long-term memory in AI agents?
Short-term memory is what’s in the context window for the current task: the conversation, tool results and scratch notes. Long-term memory is persisted outside the model, in files, a database or a vector index, and loaded back when relevant.
Do LLMs have memory?
No. A model’s weights don’t change between calls, and each call sees only the prompt you send. Products that “remember” you store notes outside the model and insert them into later prompts.
How is agent memory different from a bigger context window?
A bigger window holds more for one session, at a per-turn cost and with weaker recall as it fills. Memory persists across sessions and loads only what’s relevant, which keeps prompts smaller.
What is the Claude memory tool?
An Anthropic API tool (memory_20250818) that lets Claude view, create, edit, rename and delete files under a /memories directory. Your application runs the operations and controls the storage.
Sources
- Anthropic, Effective context engineering for AI agents, September 29, 2025
- Anthropic, Memory tool documentation (read 2026-10-31)
- Sumers, Yao, Narasimhan and Griffiths, Cognitive Architectures for Language Agents, arXiv, September 2023
- Packer et al., MemGPT: Towards LLMs as Operating Systems, arXiv, October 2023
- Park et al., Generative Agents: Interactive Simulacra of Human Behavior, arXiv, April 2023