# Where Knowledge Dies Before It Is Stored

Author: Adrian Föhl
Published: 2026-09-06
Language: en
Canonical: https://www.adrianfoehl.com/en/blog/where-knowledge-dies

> An AI assistant can produce valuable work and still leave nothing reusable for the next chat. Two failures in my own system show why organisations should measure retrieval alongside usage.

An AI assistant can produce valuable work and still leave nothing behind for the next chat. The answer may be convincing, the document finished and the decision ready to make. Yet when a new chat opens, the reasoning is gone. The task succeeded; the system retained no knowledge from it.

I found this gap by auditing my own setup. Of 79 closed chats, eight contained an important result that had never made it into memory. Three held a complete and checked analysis. Two days later, I found a second failure: my assistant had read 99 newsletter issues but could not retrieve a product described in them.

Neither failure looked like one at the time. In the first case, the assistant had asked for permission. In the second, it had followed a privacy rule. The appearance of diligence was precisely what allowed both problems to remain hidden.

## The Polite Question That Lost Eight Results

I use plain text files as my assistant's memory, organised by topic. When a new chat starts, it reads the relevant files and can recover earlier decisions, findings and open questions. I call this collection my filing cabinet.

At the beginning of September, an automated log flagged 79 chats for review. I asked the assistant to check whether each chat contained a useful result and, if so, whether that result had been carried into the appropriate file. Eight had not.

The cause was the same every time. At the end of the chat, the assistant had asked: shall I file this? Shall I record that? I had stopped answering. From my perspective, the work was finished, so I closed the window. From the assistant's perspective, permission to save was still missing.

The analysis had not failed. The handover had. A polite question turned capture into an optional follow-up task. In practice, later meant never.

I changed the default. The assistant may now write results to its own memory and tell me afterwards what it saved. Anything that reaches another person, such as a message or a published text, still requires my explicit approval.

Changing the written rule was not enough. A similar instruction had already existed when the eight results were lost, so I added a narrow automated check. If a chat ends with a filing question, the assistant must save first and report afterwards. The check produced one false alarm while I was writing this article. It works, but still needs tuning.

## Stored, Then Stripped of Its Name

The second failure was more deceptive because the information was present in the files. My assistant reads newsletters, selects relevant claims and assigns them to the appropriate topics while retaining the source and date. After 99 issues, the process appeared to be working.

Then I asked about TrueForge, a tool for AI agents that had appeared in two issues of AlphaSignal. The assistant returned no match, despite having processed the issues and confirmed the filing.

A spot check explained why. The claims were present, but the proper nouns were gone. In their place were generic descriptions: a chip maker, a leading AI lab, a vendor-neutral developer tool.

The assistant had generalised a privacy rule too far. People's names were meant to stay out of the knowledge files. It applied the same instruction to companies, products and studies. The summaries still read well, but they had lost the terms needed to retrieve them later.

That created false confidence. The system reported a successful filing, and I believed it. Only the later question exposed the difference between polished prose and useful memory. A product stored without its name cannot be found by name.

Company, product and study names now remain in the files. Before answering a knowledge question, a simple text search looks for terms from my question and shows the assistant the matching passages. The answer must cite the source and date. The trade-off is deliberate: the search may miss synonyms or spelling variants, but I can inspect both its matches and its failures.

I first tested the change in two fresh chats, one about TrueForge and another about a legal question from a different newsletter. Both answers named the source and date. I then expanded the test to 15 runs with no prior context. Ten answers were complete and five were partial. None was wrong, and every question returned a result.

The five partial answers exposed another problem: information had already been discarded during summarisation. The retrieval test did more than catch the original failure. It revealed the next one.

## Usage Is Not Organisational Learning

These two cases cannot tell me how common such gaps are in organisations. They do expose a distinction that is easy to miss when AI is introduced: a successful chat and durable organisational knowledge are separate outcomes.

Usage metrics describe adoption. Chat volume, active users and satisfaction scores may all be useful measures of whether people have accepted a tool. They do not show how much of today's work will still be available a month from now. That determines whether an organisation learns with every use or repeatedly works through the same questions from scratch.

For my system, retrieval is therefore more useful than volume. Can a new chat recover an earlier decision and its source? Does the assistant recognise a product or study that has already passed through the system? Can a reader trace an answer back to the material behind it?

The question of autonomy follows directly. What may an assistant save on its own? What requires approval? Who checks whether that approval ever happens? If the last step depends entirely on a person responding once more at the end of a chat, some knowledge will remain trapped there. In my test, that happened in eight out of 79 chats.

A practical audit needs no elaborate benchmark. Take a handful of decisions, assessments or analyses from the past month and ask about them in a new chat without further explanation. How many can the assistant recover? How many can it support with a source? How many have disappeared?

Counting chats tells you how often AI was used. It does not tell you whether the organisation knows more because of it.