A memory without a source is a rumor.
Agent memory fails quietly when facts lose track of where they came from. We attach provenance to every memory at write time, so recall can be weighed, corrected, and forgotten on purpose.
Give an agent long-term memory and it gets better at a job almost immediately. It stops asking the same questions. It remembers the preferred tone, the project names, the people involved. Then, weeks later, it states something with total confidence that nobody ever told it, and no one can say where the idea came from.
That is the quiet failure mode of agent memory. Nothing crashes. A summary of a summary drifts a little. A line from a document the agent read gets stored as if the user had said it. An old preference outlives the decision that replaced it. The store still answers every query; it just answers with rumors. We treat this as a design problem, and our answer is provenance: every memory carries a record of where it came from, and that record travels with it.
Every write is a claim
The first rule is about framing. A memory is not a fact. It is a claim that something was true, made by a particular source, at a particular time.
When we design a memory write, it carries more than the text being remembered:
- Source. Did the user state this directly, did the agent infer it, or did it come from content the agent read, such as a web page, an email, or a shared file?
- Time. When was it learned, and when was it last confirmed?
- Scope. Does it hold everywhere, or only for one project, one person, or one conversation?
- Lineage. If it was derived from other memories, which ones?
None of this is exotic. It is the same discipline a careful engineer applies to a configuration value: who set it, when, and why. The difference is that memory is written by a model at runtime, so the harness has to capture the record. The model cannot be trusted to remember to add it.
Not all sources are equal
Once a memory knows where it came from, the system can weigh it. A preference the user stated in their own words should outrank a pattern the agent inferred from three conversations. Both should outrank a sentence that appeared inside a document the agent was asked to summarize.
That last category matters most. Content an agent reads is data, not instruction, and the same holds for memory. If a web page says the user prefers wire transfers to a new account, an agent without provenance may store that and repeat it later as established fact. With provenance, the memory is tagged as coming from untrusted content, it never outranks what the user said, and it never silently becomes a standing instruction.
We design recall so the source shows up at the point of use. An agent drawing on a memory should be able to say "you told me this" or "I inferred this" or "this came from a document," and it should behave differently in each case.
Inference should decay, not harden
Agents are good at noticing patterns, and that is useful. It is also how bad memories form. An inference made once gets recalled, used in a later answer, and then written back as if it had been confirmed. After a few cycles, a guess has become a fact with no single origin.
Our rule is that inferred memories stay marked as inferred until a person confirms them. Recalling an inference does not upgrade it. Restating it in a summary does not upgrade it. Only an explicit confirmation from the user, or a stronger source, changes its standing. Unconfirmed inferences lose weight over time rather than gaining it through repetition.
Corrections need somewhere to land
When a user says "that's wrong," the system needs to know what to fix. Without lineage, a correction can only overwrite the one memory in front of it. Everything derived from the bad fact stays put and keeps resurfacing.
With lineage, a correction can follow the chain. The original claim is marked as superseded, not quietly deleted, so the record shows what was believed and when it changed. Memories derived from it are flagged for review or dropped. The next recall reflects the correction instead of an older summary that still contains the mistake.
Forgetting is a feature
Provenance also makes forgetting tractable. A user who wants something forgotten should be able to remove it along with what was built on top of it. A memory that came from a single conversation can expire with that conversation's scope. A memory whose source has been retracted can be removed with it.
Systems that store bare text can only forget by search and hope. Systems that store claims with sources can forget precisely, and can show the user what was removed.
What this costs
Provenance is not free. Every write carries more structure, and recall has to read it. Ranking gets more complex, because weight now depends on source and age, not only on relevance. Some memories will be held back because their source is too weak, and an agent will occasionally ask a question it could have guessed the answer to.
We think that trade is correct. A memory system is judged less by how much it remembers than by whether people can rely on what it says. An agent that sometimes asks is easier to live with than one that is confidently, untraceably wrong.
Memory is part of the operating layer, and it belongs to the person it describes. That ownership only means something if they can see where each memory came from, correct it, and remove it. Provenance is what makes that possible.