An AI assistant that writes everything down and never forgets will eventually be wrong, and with full confidence. I saw this happen with my own assistant. For months I did not notice that the files it draws its knowledge from had stopped being true. Missing content was not the reason. Everything was written down, and nothing ever left.
I think this happens to any system that writes something down every day but never digests and never forgets. The problem never announces itself. You only notice when you look again.
My assistant's memory is over a hundred markdown files, which are plain text files. Each of them was true on the day I wrote it. Together they add up to a memory that is confidently wrong. I had once called that stack the filing cabinet. What the filing cabinet did while I was not looking inside, I only noticed late.
My assistant read the outdated plan first
I caught it in a recommendation. The assistant suggested running a local model on a particular machine. Months earlier we had ruled out exactly that machine for this job, because it was too small. That decision was in the files. The outdated plan from before it was in there too. The assistant read the outdated one first.
Once I started paying attention, I saw it everywhere. The assistant retold one decision with a status I had corrected twice since. One topic was spread across fifteen files. The answer depended on which of them happened to be read first. None of it was broken. There was no error and no warning. Every single memory had been true on the day I wrote it.
I call this memory rot. What gets written down turns wrong over time, and the system never pushes back. It is not a bug, it is the default. A system that writes something down every day, never digests and never forgets, will eventually be wrong, precisely because it records so diligently.
The first scan found 64 problems
I built myself a small scanner and pointed it at my own setup. The first number made me uncomfortable. Roughly 5,300 tokens were loading into every single chat before I had typed a word. A token is a small unit of text. So this is background text the assistant reads automatically in every conversation. It is the always-loaded part of the memory, and I had never seen it listed in one place.
The first full scan found 64 problems. Some files were explicitly marked as rejected and were still influencing the assistant's answers. Three files had fallen out of the master list. You could only find them if you already knew they existed. On top of that came near-duplicates and dead links. And the same file sat in two places, carrying two versions of the truth.
My favourite finding disproved an assumption I did not know I held: that the newest file is the right one. Twice, the richer and correct version was sitting in the older copy. So a file's date tells you nothing about whether it is right. The content does.
The obvious fixes all fail at the same point
I tried three of them, one after another.
Saving less does not help. Saving is how the system learns who you are. If you let it record less, you end up with an assistant that stays generic.
Deleting old things automatically does not help either. Whether something is stale depends on what it means, not on how old it is. My preferences from March still hold. A project status from last week may not.
Leaving the sorting to the model while it reads fails as well. That is exactly what it cannot do, and it should not have to. A model has to be able to trust its own memory.
A memory needs a metabolism
The mental model that finally worked for me is biological. A memory system has to do three things with equal standing: take in, digest, let go. Taking in means writing things down. Digesting means merging and correcting entries. Letting go means archiving what is outdated. My system could only do the first.
In practice this runs on three schedules. In every chat I write freely while following a few rules. For example, I update the existing file instead of creating a second one next to it. When a fact changes, I correct it in place rather than writing the new version somewhere else.
Every week a deterministic scan runs, checking against the same fixed rules each time. It finds problems in the file structure: orphaned files, dead links, duplicates and files marked as stale. Every month there is a short ritual. In it I distil finished episodes into durable knowledge and then archive them.
Two rules carry most of the weight. First, nothing is ever deleted. Superseded files get marked and move into an archive with its own master list. That way the history stays findable without steering anything. Second, every fact gets exactly one owner file, and everything else only points there. When something changes, there is one place to correct instead of fifteen.
A single number shows me whether the system is only taking in. It counts the tokens that load into every chat before I type anything. If it grows month over month, the system is taking in without digesting. It measures size, though, not whether the content is right.
A correction now stays corrected
The cleanup itself took three cleanup chats with the assistant. The scanner flags problems, I decide what to change, and the assistant makes the changes. That took it from 64 problems to zero. On top of that I deliberately marked around sixty findings as fine, each with a written reason. The weekly scan has kept it there since.
The more interesting change does not fit into a chart. My assistant no longer gets things confidently wrong that we corrected long ago. When I set a fact straight today, it stays straight. The correction lands in the one owner file, and the old statement gets marked where it stands.
I wrote the whole approach up as a small field guide. It comes with the scanner as a readable reference implementation, around 700 lines of Python. Both are on GitHub: memory-metabolism. The guide explains the approach. The code is there so you can check that I am not just making claims.
Bottom line
If you run an agent with memory, I would suggest two things. First, I would audit before cleaning anything up. Just knowing what loads into every chat and what it costs changes how you write memory in the first place.
Second, I would expect knowledge to go stale from day one. That is not a personal failing. It starts earlier than you think. On my setup it took about three months.
In the end, one principle stays with me. Remembering, digesting and forgetting need equal standing, each on its own schedule. A system that only remembers will, after enough months, be wrong, about you as well.
Do you know how many tokens your agent loads into every chat before you type anything? And when did you last check whether what it loads is still true?
Try it yourself
The scanner needs Python 3.12 and uv. The repo ships with a demo store. That way you can try it before pointing it at your own setup:
git clone https://github.com/adrianfoehl/memory-metabolism
cd memory-metabolism/memgc
uv sync
uv run memgc audit --config ../examples/demo/memgc.yaml # what loads into every chat, and what does it cost?
uv run memgc scan --config ../examples/demo/memgc.yaml # stale, duplicated, no longer findable
The scanner only finds candidates. What can actually go is a human decision.