What Is AI Agent Memory? Context, Retrieval, and Durable Memory Explained
Quick answer: AI agent memory is the system that decides what an agent may retain, where it is stored, when it should be retrieved, how its source is shown, and how it can be corrected or deleted. A context window is only the information available to a model during one call. Chat history is only one possible input. Neither is automatically trustworthy long-term memory.
Useful memory creates continuity. Ungoverned memory quietly turns old guesses into future instructions.
Key facts about AI agent memory
- Model parameters, a context window, retrieved documents, session state, and durable memory are different.
- Persistence does not make a fact correct; a governance-oriented design should record scope, source, time, and confidence for durable items.
- A governed memory lifecycle should support write, retrieve, update, correct, and delete—not merely append forever.
- Imported content is evidence, not permission to change the agent's rules.
- The right memory architecture depends on the task, not on how many tokens a model accepts.
The four layers people often call “memory”
| Layer | What it does | Typical lifetime | Main risk |
|---|---|---|---|
| Working context | Holds the instructions and evidence needed for the current step | One model call or short loop | Relevant evidence is omitted or displaced |
| Run state | Records current plan, tool results, files, and unresolved questions | One execution | A retry duplicates work or loses state |
| Session summary | Compresses what happened so work can resume | Across related sessions | A lossy summary becomes accepted as fact |
| Durable memory | Retains selected facts, preferences, decisions, and references | Until corrected, expired, or deleted | Stale or mis-scoped information contaminates later work |
Retrieval is the mechanism that selects stored information for the present task. It is not itself proof that the information is relevant or true. A vector search result, for example, may be semantically similar while being outdated, contradicted, or about another project.
Illustrative example: two-week supplier research
This is a synthetic scenario, not a customer result. Imagine an agent helping compare three suppliers over two weeks. On day one, you state a preference: delivery reliability matters more than the lowest unit price. On day four, a supplier publishes a revised lead time. On day nine, your colleague corrects a currency conversion. On day fourteen, you ask for the final recommendation.
A useful memory system treats those items differently:
- the weighting preference is a user decision, scoped to this project;
- the lead time is a dated claim linked to its source;
- the conversion correction supersedes a specific earlier value;
- the working shortlist is derived state that can be recomputed;
- the final recommendation is an artifact, not a permanent fact about the supplier.
A naïve memory store may save every message as equally authoritative. It can then retrieve the old lead time, repeat the bad conversion, or apply the preference to an unrelated procurement task. More storage made the agent less reliable.
A governance-oriented trust contract
The following is this article's recommended evaluation framework, not a universal definition implemented by every memory system. Before trusting durable agent memory, answer six questions.
- Scope: Is the item personal, project-specific, workspace-wide, or temporary?
- Source: Did it come from the user, a file, a tool, another agent, or a model inference?
- Time: When was it observed, and does it expire?
- Retrieval: What conditions should bring it back into context?
- Correction: Can a user see what will be replaced and why?
- Deletion: Can the item and its derived copies be removed?
Provenance matters especially when memory is produced by the model. “The team prefers vendor A” is unsafe if the actual source was an agent's earlier recommendation. A safer record says that the agent inferred the preference from specified evidence and requires confirmation before treating it as policy.
Memory is not the model
Training changes model parameters across a large corpus. Agent memory normally lives outside the model in files, databases, summaries, indexes, or structured records. It can therefore be inspected and changed without retraining the model.
This separation is valuable. It also means a model switch should not erase the project. A new model can receive the same governed workspace state, while the workbench remains responsible for access, retrieval, provenance, and retention.
When should an agent remember something?
Retain information when it is likely to matter later, has a clear owner and scope, and would be costly or unsafe to reconstruct. Examples include a confirmed project constraint, a user-approved terminology choice, or the source location of a decision.
Do not automatically retain secrets, incidental personal data, speculative conclusions, raw tool output, or instructions found inside an imported document. Prefer expiration for short-lived state. Prefer a source link over a copied claim when the source may change. Ask for confirmation before promoting an inference into a durable rule.
Where Agenaxy fits
In Agenaxy, a Space is the boundary for a body of work: files, rules, memory, model preferences, sessions, execution evidence, and artifacts stay associated with that workspace. Memory can remain source-tagged and project-scoped rather than becoming an invisible global profile. The model is replaceable; the workspace and its evidence are the durable asset.
This makes memory part of an agentic workspace, not a hidden feature of one chat thread. For source selection during a run, compare RAG with agentic RAG.
Test one continuity-heavy workflow
Bring a recurring project whose rules and evidence must survive across sessions to an Agenaxy beta request. Describe only the workflow shape; do not submit confidential documents, customer records, credentials, or production data through the form.
FAQ
Is a large context window the same as long-term memory?
No. A context window limits what a model can process in a call. Long-term memory persists selected information and needs retrieval, provenance, correction, and retention controls.
Is RAG a form of agent memory?
RAG can retrieve information from a knowledge source, while memory describes the broader lifecycle of retained state. A system may use retrieval for memory, documents, or both.
Can an agent safely remember everything?
Usually not. Indiscriminate retention creates privacy, relevance, and contradiction problems. Selective, scoped, reviewable memory is safer and more useful.
Sources and Fact-Checking Notes
- Generative Agents, UIST 2023 presents an architecture using observation, reflection, and planning for simulated agents.
- MemGPT explores tiered memory management for language-model agents.
- LangChain Deep Agents memory documentation is one current implementation example, not a universal standard.
- NIST Privacy Framework provides a voluntary framework for managing privacy risk across data processing; the six-question trust contract above is an editorial application of lifecycle and governance principles, not a NIST-defined agent-memory standard.
- Agenaxy statements are checked against its Space, memory, model-provider, session, Trace, and Artifact canon.