My AI Remembers Everything. That Turned Out to Be the Problem.

In the last post I said something I was quietly proud of: my setup keeps writing things down. What I decided, where a project stands, what I care about. More than a hundred plain markdown files, with an index at the front. The filing cabinet, I called it.

This is the post about what the filing cabinet did next.

The moment I noticed

A few weeks ago the assistant recommended running a local model on a machine we had explicitly ruled out for that months earlier. Too small, we had decided. The decision was in the files. So was the old plan from before the decision. It read the old plan first.

Once I saw it, I saw it everywhere. It described a decision using a version of events that had been corrected twice since. One topic was covered in fifteen files, and the answer depended on which file it happened to read first. Nothing was broken. No error, no warning. Every single memory had been true on the day it was written.

That is the part worth sitting with. I call it memory rot: it is not a bug, it is the default. A system that writes something down every day, and never digests and never forgets, ends up confidently wrong. Not despite the diligent note-taking. Because of it.

What I found when I finally looked

I built a small scanner and pointed it at my own setup. First honest number: about 5,300 tokens were being loaded into every single session, before I typed a word. That is the always-on part of the memory, and I had never seen it listed in one place.

The first full scan found 64 problems. Files explicitly marked "cancelled" that still steered answers as active knowledge. Three files the index had simply forgotten, unreachable unless you knew they existed. Near-duplicates. Dead links. The same file in two places with two different versions of the truth.

My favorite finding, because it broke an assumption I didn't know I had: the newest file is not the truest file. Twice, an older copy held the richer, correct version. File dates lie. Content decides.

Why the obvious fixes don't work

I tried the obvious things first.

Save less? Saving is how the system learns who you are. A memory on a write-diet is just an assistant that stays generic.

Delete old stuff automatically? Whether a fact is stale is a question of meaning, not of age. My preferences from March are still true. A project status from last week might not be.

Let the model figure it out when it reads? It can't, and that is by design. The model trusts its memory. That is what memory is for.

Memory needs a metabolism

The mental model that finally worked for me is biological. A memory system needs to do three things, and they need equal rank: take things in, digest them, and let go. My setup had only ever eaten.

In practice that became three clocks. Every session: write freely, but follow a few writing rules, like "update the existing file instead of creating a sibling" and "when a fact changes, correct it where it stands, don't just write the new version somewhere else." Every week: a deterministic scan that catches the mechanical drift, orphans, dead links, duplicates, stale markers. Every month: a short ritual where finished episodes get distilled into durable knowledge and then archived.

Two rules carry most of the weight. Nothing is ever deleted; superseded files get marked and moved to an archive that keeps its own index, so history stays findable without steering anything. And every fact gets exactly one home file; everything else just points there. When something changes, there is one place to correct instead of fifteen.

And one metric tells me if it works: the always-loaded token count. If it grows month over month, the system is eating without digesting.

What changed

Cleaning up took three review sessions with the assistant itself: the scanner finds, I decide, it applies. From 64 findings to zero, with about sixty deliberately dismissed as fine, each with a written reason. Since then the weekly scan has held the line.

The more interesting change is the one I can't graph. The assistant stopped being confidently wrong about things we had already corrected. When I fix a fact now, it stays fixed, because the correction lands in the fact's one home file and the old statements get marked where they stand.

I wrote the whole approach up as a small field guide, with the scanner as a readable reference implementation, about 700 lines of Python. It is on GitHub: memory-metabolism. The guide is the actual product; the code is there so you can check I'm not hand-waving.

What I would tell anyone running an agent with memory

Audit before you clean. Knowing what loads into every session, and what it costs, changes how you write memory in the first place.

Expect rot. It is not a sign you did something wrong. It is what happens by default, and it starts earlier than you think; my setup took about three months.

And the principle I keep coming back to: remembering, digesting and forgetting need equal rank, on different clocks. A system that only remembers will, given enough months, remember you wrong.