The memory of AI agents

What would happen if every morning you woke up having forgotten everything that happened yesterday and on every other day, and all you had to go on were the things you’d left around the house and a few notes you’d written down?

That’s basically what happens with today’s AI. When we open a new prompt, the only context it has is what you tell it at that moment, plus any default instructions it may have. These days we try to solve this with some fairly simple memory systems that store a few basic facts the agent considers important.

Why not store everything? Well, in general it’s an optimization problem, but also one of scale and space. Storing everything an AI agent does from the very first moment you start interacting with it means storing a lot of information: not just your prompt, but what it read along the way, the tools it used, its answer, the files it produced, the solutions it applied. Besides being a crazy amount of space, it isn’t optimal, because how do we search all that information for what’s actually useful?

This is where the need for modern processing and agentic processes starts to show: processes that digest the information and keep only an extract. Often it’s enough to ask it to save a summary of what matters, or of what needs to be remembered in future sessions. Those text files are still hard to process, so we invented RAG techniques, or semantic search, so that an agent can later search by related context and doesn’t have to look for a literal word, but for something with a similar meaning in the information… I’ll go into more technical detail on how this works another time, but all you need to know is that it works pretty well.

But of course, when we compare it with our brain, this whole mechanism is still a measly little pebble next to the huge mountain that is our brain. Yes, even the brain of the person with the worst memory does far, far more. This is where biology and brain processes come in. I’ve read a bit about all this and learned a few things:

• At night, while we sleep, our brain processes what happened during the day and starts a process of learning and remembering. How?
• Did I have to recall something from before that helped me with today’s tasks? Yes? Then positive reinforcement: that memory is useful.
• Did thinking about something hurt me, stress me out, or make me feel bad? Yes? Bad, negative memory… sink it into the deepest corner of memory.
• Is there something I did that’s the same as always and didn’t spark the slightest interest? Forget it outright. That’s why we often don’t remember every detail of what we did yesterday, only the most important parts.
• Then there’s associative memory… our memories are… multimodal. AI keeps getting better, but the one we chat with reads and writes text; its memories are text, nothing more. We, on the other hand, perceive images, sounds, lots of things that can also be stored in our brain… That’s why we remember something lovely when a certain song was playing, our first dance, or how a date went just by catching the scent of the perfume they were wearing. Our brain takes in not only text and speech, but sounds, images, concepts… and it can link them to other things: dates, people, friends, feelings.

The better we get at simulating some of these things in the information the AI processes, the more powerful our agent will be when it comes to working, because it will remember lots of things that are useful for its tasks.

Of course, the information being remembered may include sensitive or personal data. Does anyone want big corporations to know these things? I don’t think anyone does… That’s why they put a lot of effort into building processes that strip this out of the memories. But wouldn’t it be more useful to us if it kept them? Probably, yes, but no company wants to be on the hook for fines over disclosing personal information.

This is where personal memory systems come into play: they store the information on our own computer, or in a private git repository (which lets us sync it with other computers).

That’s why I first built Remembrances and then Pando. The first one was the initial attempt, and it worked very well, but as I wrote above, our brain does a lot of things while it works… and there are many processes running in the background to give us the right memories for the task we’re about to do at any given moment. We needed to build our own AI agent if we wanted to work underneath all the processes the AI runs, so we could automatically feed any prompt with the right context and avoid having an agent go looking for it through MCP tools.

The more I work on memory systems, the more amazing it is to see how our brain still operates on scales infinitely larger and superior to any current AI. The amount of information it processes and stores is enormous… OK, it can’t hold all the world’s knowledge, but it can hold everything that has happened in my life, what I’ve seen, smelled, heard, learned, and sometimes even forgotten…