Overhaul

Intro should have hopefully made the timeline here and how this system was iterated on, in these next sections I am going to break down each component then the converged design. The sections pre convergence were written before when I thought I had finished and was running bench, I left them in here though since everything is similar and relevant at some point across the productivity and character surfaces.

The first change is in these ledgers, the conversation logs. Before there were a few issues; firstly on selection it was only choosing one subtopic at a time so if knowledge was needed from several it could be missed. Furthermore choosing the storage location each turn gets messy with asides, tangents, continuations etc. knowledge could be folded into the wrong subtopic. Also these subtopic ledgers were formed from many rows; bits may not be resolved and may not make sense on future retrieval. On top of this the short term continuity is weak, I passed in the recent 3 verbatim turns as a counter measure before but it was a fragile solution; it may have worked for the bench but it does not translate cleanly to real world chats.

This had been replaced so now stage 3 (the storage stage) just compresses the turn, still lossless containing all details but more token efficient - like caveman representation. There are two scenarios in which I allow lossy compression; the first is when the AI response contains recomputable general knowledge as this can be represented as the key points so that the future response is clear if pulled on but we don't bloat the ledger rows. The second is a new mechanism I added for scenarios where we might want iteration and exact precision, e.g. AI writing an essay, giving a plan, creating a schedule etc. in this case stage 3 can choose to create a document so this can be carried forward then the ledger turn can just include a reference to this document.

In an ideal world I would not store anything from the AI side unless there is signal from the user queries that they engaged in that part of the response. The user query is always ground truth, it came from the user you know it is important, however the majority of the AI response is throwaway and so shouldn't be stored unless the user confirms, pushes back, expands etc. The issue with this is that LongMemEval-S bench tests AI side recall when there are no salience signals from user and so points were dropped here. I removed this but it is one of the reasons I am not a fan of this bench but SOTA score is needed for marketing piece so I accepted this tradeoff. This decision changes in convergence below.

The rolling ledger then builds until a threshold, once it reaches a certain size a background job is queued to harvest the head rows of the ledger (the older rows) and process into a better format keeping the tail (the recent rows) intact. This allows short term continuity to hold regardless of conversation and keeps this context bounded since the live ledger tail is always inserted as context. This ledger and the harvest both get removed for productivity in convergence below, though they still run in the character app.

The harvest job is a collection of LLM calls each serving a distinct purpose which runs in the background as to not affect the user experience. Since only the head of the ledger is harvested, more turns can be completed while this job runs and they can just be added to the tail allowing overflow on the threshold since once the harvest completes the head is removed from the ledger restoring it below the threshold.

Harvest works like this: extract facts and supporting detail (traces) from the head slice of ledger rows, organise these facts and traces into topics as well as link to entities (will explain later).

If you take any information like dialogue, documents or anything else and think about the most atomic way to store it you land on facts and traces; your claims and your supporting info around it. This covers the gap of ledger rows with resolutions across turns since these facts and traces are derived from the head slice rather than per turn, your fact states what it is and the trace contains all the extra relevant detail like evolutions, discussion etc. This moves to per turn in convergence below.

There is still a gap here in that what if this extract contains continuations or mentions to something from earlier, especially when near the boundaries. I solved this for the majority of cases which then introduced its own edge which again I solved, a common theme when implementing this architecture - the more I solve the more I realise is unsolved but it's part of what got me so addicted to this project. I want to fully solve this issue since it is my one major blocker to applying this to document ingestion, however I also pushed back shipping by a couple weeks already trying to solve every new edge I could think of and so I had to park it but I will solve this soon. I'll touch more on why this affects the doc ingestion in that section later down if you are interested.

Organise is then going to choose topics to store these facts and traces in as well as a new layer I added for entities. While topics cover most cases, there are still scenarios in which knowledge can be scattered e.g. ‘how many projects did I undertake this year’. A key word search could be implemented for these types of queries but that is weak, a grouping layer is needed instead which is where entities come in. They live alongside topics as an optional extra link in order to group these related facts.

Now we have a nice clean storage layer however this still won't scale infinitely if implemented wrong. Keeping every call and component bounded was a core focus as I took this architecture from paper to code ensuring that input tokens across the calls in the pipe stay stable no matter the chat length. I break down what that means for cost and scaling in the bench section below.

The last major layer is retrieval, the call that runs before we pass off to the answering model. This chooses topics and entities to load into context. I did mess around with an intermediate filter call to select only relevant facts, cutting down context even more making the system even more efficient however it just wasn't reliable on the cheaper models. I tried every trick I could think of but another issue was always exposed and so I reverted back to just dumping everything within a topic which is why it's so important to keep these bounded and not let them grow uncontrollably.

As a chat gets larger, the number of entities and topics can balloon quite easily hurting retrieval, so I implemented an active vs cold storage solution (active being the recently used topics and entities, cold the rest) where everything is still retrievable but it ensures this call stays bounded token wise.

This stage also has additional responsibility in deciding what images, docs, chats or web sources to load in. More on chat attachments in the product section, but it's just selective retrieval here again, once attached to a chat it makes no sense to pass in these attachments to all subsequent queries as standard chat apps do and so only load in when necessary, not much more to say on these.

Spent a solid couple weeks on documents while iterating in these past 6 weeks, but ultimately landed on a simple solution until I solve that resolution edge mentioned before in extract. This comes in here for speed of ingestion but also for correctness. Docs are not like chat turns where there are clear separation between turns and while there is structure, it is not guaranteed to be clean and parseable. Since docs are also larger than chat turns, you are going to want to process all chunks in parallel for ingest to finish in a timely manner, and so you need a way to resolve the continuations and pronouns etc.

Part of the reason I haven't solved this issue is because this updated memory design I only landed on a couple weeks ago. So then I only made the connection that I could apply this memory to doc surface like a week ago and that would be a whole thing in itself so I deferred so I could ship.

My current solution for docs is just offload the work to cheaper models; if a doc is small enough can throw it all into the answering model and it's cheap since selectively retrieved so not a consistent cost. If a doc is larger, fire parallel calls to process a chunk at a time and since chunks are still quite large, a ref map (a map of references into the chunk) is used to break it down further. This lets the model output just the refs that might be needed to answer the question which are then joined and passed into the answering model to crystallise an answer from. From here adding in edits wasn't too hard as if you can find something you can edit it, however it did introduce a bunch of edges which were not fun to solve. While this did hold through tests, this is not as polished as I would like however as a solo dev I am restricted by my time and output. Even with 4 chats at going once there is only so much I can do but I will continue to refine this and then wire in the proper memory architecture. Since the full doc is processed every query here when relevant, it is more costly long run vs if I applied the memory architecture you pay once at ingest then it's simpler selective retrieval much cheaper.

Was using grok 4.1 fast non reasoning before happily until they deprecated it and so had to pivot, blessing in disguise though as I have now moved onto Qwen3 235B A22B Instruct 2507. Not only is this model open weight giving way more flexibility on providers, it is also much cheaper at $0.1 / million tokens in and out vs Grok which was $0.2 / million in and $0.5 / million out.

This does of course come with the tradeoff that it makes more mistakes, it is less reliable in capturing all detail and so the majority of my time implementing is working on prompts and implementing guardrails and recovery measures to ensure the memory stays solid.

Designing this architecture was one thing, but then implementing was a whole different beast. Iterating on prompts is some of the most draining work I have ever completed, going in circles, every tweak regresses 3 others then a new edge surfaces from the non deterministic nature. I could write another 3 pages on all the different design choices within, with some going over all my iterations but finally I have gotten to a spot where I am happy with it.

It's all great having a memory system but you need some way to get all your previous chats into this system otherwise you have to start fresh and there are three methods for this.

Method 1: if you have all your source turns via a data export or other methods then you can simply ingest through stage 3 and harvest. This is how I did the bench; run the turns through stage 3 then can run harvest and it all gets placed. This was the pre convergence flow though, after convergence you ingest each turn straight into facts and traces with no separate harvest step. This method has its issues regardless. Firstly it is slow since it has to be sequential within a chat. Then the cost for richer chat histories can get large even when running through cheaper models, not to mention the friction of getting your data export which is something most people can't be bothered to do, and so while this is the best method for completeness, it doesn't really work in a commercial product. Again here resolution issue is a blocker if you want to parallelise within chat to speed up so this one remaining piece unlocks a lot of different options since cost is one thing, but waiting days for a full chat import is a whole different issue in itself.

Method 2: A simpler method; a well written prompt asking for the detail you want. The source chat outputs and then we can parse the output and fill in the memory for this chat. Obviously a lot of specifics will be dropped here, however the key details are retained which I found to be enough when testing and it's much cheaper and easier, simple copy and paste and so I went for this in app.

Method 3: This is a hybrid approach where you get the best of both worlds by using method 1 for the recent history then method 2 for all older history. This way you have everything in memory and it eliminates cost concerns since ingesting recent history is very cheap as you will see when I break down ingestion costs on the LongMemEval-S dataset below in the results section. Any chat that hasn't been interacted within last 90 days widest, but could even reduce this to like 30 days or narrower, realistically is unlikely to be revisited and the user won't even remember all details from it so key points would be enough and then active chats are ready to be continued properly via method 1. I will integrate this option into app at some point, most likely when I solve the resolution problem so can be done in apt time, but I haven't hooked this up yet.

Before I break down the bench results I actually made some more changes to the system, I wrote this blog while doing the bench as I thought it was done but while doing bench some gaps were revealed and some things didn't make sense to me so I converged on something better. I wouldn't call this a redesign its just an improvement in my execution.

In this convergence, the biggest change is that I removed the harvest. The original idea was write into ledger rows for short term continuity then harvest periodically however this led to detail drop and other issues and it doesn't really make sense thinking about it now.

First issue with this is double writing, we are taking a source turn then converting it into two formats, on a cheap model this is bound to introduce failures and the ledger layer really is redundant; how does writing as a chat log differ from straight into facts and traces? To the answering model its the same, just context, so I removed the ledger layer and went straight into facts and traces. My other reason for doing harvest was I was worried about cross turn resolution, something over many turns you want to be able to resolve everything so that all facts/traces can read standalone on future recall but this concern was misplaced. The answering model is already gonna resolve in its answer and so this really wasn't an issue I needed to worry about going straight into facts/traces.

The next issue with harvest, even though it was bounded in how many tokens it sees, we are still putting multiple rows in then asking for facts/traces while preserving all detail which, again, makes no sense. Per turn is going to be way more reliable in preserving detail as its way more scoped and on top of that, the labelling of topics and entities are way more reliable per turn. Again in my head I was worried that labelling would degrade on per turn and it needed many turns to properly label but this is the same as the resolutions it's not an issue and was me just overthinking things.

Then there was also the fact that in the old there was this unharvested ledger which didn't show in the memory panel which was not ideal. This was bad for display and also for sharing I had to slap on a measure to extract, but now since we go straight into facts and traces the memory panel is always synced which was another nice win of this convergence.

Finally, I didn't split AI side from user side, I merged in facts/traces storing them as if equal weight. I mentioned earlier that if it was up to me I wouldn't store the AI side at all without salience signal and was just doing for the bench however I was thinking about this wrong as well. I shouldn't have been thinking about dropping AI side on storage, it is retrieval where this dropping should happen. Any AI side information which is interacted with from user would then get into the user turn of that query and so we don't need to bother detecting salience signal in a separate mechanism, it was always right there. I still use the fact/trace format but I changed how I store here, utilising the fact for the user turn distillation and the trace for the AI side giving a clean split. This then allows on retrieval the option to either include AI traces or just pass through user side, for the majority of queries just user side suffices and it is obvious when AI side is being referred to from query so this decision is reliably made. This brought my context passed to answering model down significantly, as you will see in breakdown in bench results sections below, and this also allows for panel display to be much cleaner and less noisy.

LongMemEval-S is the standard benchmark for memory systems and what you'll see referenced in every other paper. 500 questions across ~57M tokens of conversation data, broken into these categories:

For the bench, and in production, Qwen3 235B A22B Instruct 2507 handles ingestion and retrieval. The answering models I ran were gpt-4o, the official benchmark model, along with gemini-3.1-pro and gemini-3-flash. The answerer only ever sees a lean retrieved slice rather than the full history so it can be swapped without blowing up cost.

c137 v1 scores are from the last design, v2 is our new converged design so you can see the jump in scores on flash and pro, with flash now outscoring our old pro score. Our 4o score is lower than previous however this is because I didn't get around to refining the 4o prompt to get a higher score, I could have tuned to push into 84 range I reckon but not something I am bothered by since it is still on par with oracle (the score 4o gets when only the turns with answers in are passed as context). Also they differ by 2 points on raw score so within variance.

If you have read Mastra's blog you will know that they headlined 95% with a score of 94.87% on gpt 5 mini and their pro and gpt 4o scores are different than listed in my leaderboard. This is because those scores are not from the official metric of score out of 500 as a percentage, Mastra averaged their percentage scores across the categories which inflates it since some categories are easier and have fewer questions so a high score there boosts the average. Not hating, it's fair play, my scores using their same metric were lower than my raw scores otherwise I probs would have headlined the same thing. They didn't give their 3 flash raw score in blog so I left it in marked as inflated since it doesn't cleanly divide into 500 and my raw gemini score is better than it anyway.

While my pro is 6 points higher than Mastra which is on the ends of the variance, their 5 mini score is only 2 points below my pro so even without their inflated figure the two systems are effectively on par. You can also argue that they used 3 pro I used 3.1 pro so its not apples to apples, that is because 3 pro is deprecated I can't even use via API and you can still see my gemini 3 flash preview which is apples to apples outperforming all others.

I'm going to compare to Mastra in my results here and my token and cost efficiency since they are the only other system not using an embedding approach, and also since they next highest after my v2.

My retrieval accuracy was also 98.8%, meaning that out of 500 questions only 6 failed to deliver the appropriate context to the answering model. The rest of the fails for each model were just how they interpreted the question and used the context given. Most of these on pro were really hard edge cases with counting and temporal calculations and fixing those led to regressions elsewhere, leading to 94% being the consistent ceiling I could achieve. I'm sure I could have just thrown Opus or 5.5 to get a higher score but then that actually wouldn't be apples to apples.

Here is where the real win is, not only do I have top raw score, my token efficiency is far better than anyone else on the list. Embedding systems are less efficient since they are blind grabbing, it depends on how many results you take but I think they mostly will average somewhere in 40-50k token range for context passed into answering model. Mastra reports an average context window size of ~30k tokens for the entire run so more efficient than embedding models. I averaged ~2.6k.

Great headline figure but that is the best of my averages, let me break it down properly so I am not misleading you. This is probs best with a table, the memory block is context we passed in everything else is the static prompt, the majority of that is cacheable but there is a tail block as gemini benefits from that just for the harder questions where it wasn't following instructions to a T.

The mean is higher than median here by almost double and the max is very high, this is the assistant traces being included and this fired on 89 out of the 500. Assistant traces don't really explain the max 26k, they normally came out to around 10-15k tokens, so that 26k max and other higher contexts are on vague questions, the flavour of 'recommend me things I would like' and so recall just fires off random topics and entities. I'll tighten this in future but not too deep right now.

The other big win here is when we split the user from AI side as I mentioned before. Here is median, mean, max tokens from before we split compared to after and our token savings from this split:

Another thing I should flag is that I did not pass recent facts/traces for short term continuity, something I do have in prod. I left out here because it doesn't affect scores given that LongMemEval-S only tests recall; it doesn't need continuity, so in prod would be an extra ~4k tokens of context.

Since retrieval is fresh every time, the tokens injected into the answering model does not scale with turns, it scales with the memory density on a single topic area or entity. I have put in measures to cap the tokens a single topic or entity can reach with a splitting mechanism once they reach a certain size ensuring average context never blows up. This combined with the user/AI split means that chats using this architecture can effectively run on infinitely and the cost stays effectively flat, as well as the speed since every llm call in pipe is bounded; context never blows up anywhere.

On top of this, I have the active cold split I mentioned before which further helps keep everything bounded and scalable. Then if I see it becoming an issue in prod, I can always implement a graceful forgetting mechanism to keep scalability, as I have signals stored for how much info is retrieved, stored, updated. All that makes it easy to implement and pretty low risk of deleting data the user might refer to later on. This data can also be distilled before we drop it to get key insights out of it so we just don't store in as much detail. There are so many measures I can, and will eventually, take, so scaling in my eyes is pretty much solved.

Obviously the token efficiency saves cost on the answering model, allowing more usage for the same price on these frontier models, however the model selection is also a huge cost win. Our v1 scores which we ran on grok 4.1 fast non reasoning, that cost about $150 for ingestion of everything, v2 cost me $60. This is also down from my ingestion with harvest on v2 which cost me about $90 dollars so another win as a result of the convergence. Mastra's blog says they use gemini 2.5 flash for ingestion which is $0.3 per mil tokens in $2.5 per mil tokens out, taking raw blended cost Qwen is 7x cheaper. Realistically though, it's more like 4x cheaper since the majority of tokens go to input in which flash is very similar to Qwen in pricing.

The other big win with Qwen, as I mentioned earlier, is its a fully open weight model. Not only does this give me so much flexibility with providers, but it opens doors for self hosting if I wanted to, when scale allows I can go distributed over serverless cutting costs and then it also means privacy can be controlled much better. If I am running on my own hardware, it opens up so many more surfaces I can apply to, especially for B2B, without having to worry about if the provider is misusing the data.

As I said for my last results, I am closed source with no API so my results are not reproducible and there is no way for you to verify that I did not cheat this bench. I am going to look at implementing stuff for those in future but it is not the right use of my time right now, I have prepared a bench viewer where you can view all 500 questions in my chat ui, grouped into projects for each category type. The fails are tagged with [FAIL] in the title and there is a show fails only button to filter to just those. You can also search the search bar for a specific question id. Then I also included the tokens passed into memory block for each question in the top strip and you can open the memory panel for any question to see how it would be displayed in chat. This is as much data as I am willing to share to protect my work here, so if you still think I have cheated the bench or it's not a legit run fair enough, I can't put down your skepticism at the end of the day.

All I can say is that I see no merit in that which is why I have tried to be as honest as possible in this blog, I don't care for fudging results for a marketing piece as I think this is a solid project and I don't want to tarnish that with falsified results. As for how I ran each model, I froze the stage 2 prompts then replayed through other models, for 4o I changed the static as they were very gemini tuned so I removed some of my extra clauses, however the dynamic context stayed identical.

Edit: Ingestion and retrieval are still not reproducible, however I created an open source repo which contains the full prompts for each question, you can plug in your key, run and grade on the official grader script to ensure no fabrication in there. Every scaffold I added to the prompt was abstracted for question flavour but never used specifics from the bench as to not game it, this obviously is still not full reproducibility but it's as much as I am willing to share.

You can view all the questions here:

Edit: I added stage 2 prompt dumps for every question so you can see exactly what the model saw to answer with. My live prod prompts are different since these prompts are very gemini tuned and I offer any model so clashes, this is why I removed some scaffolds for 4o and also why there are a lot of tail instructions not giving optimal caching. I hope this along with memory panel and all questions is enough to fight any doubt around my results.

I would like to use this section to actually retract some of my issues I said I had with the bench in the previous blog, as a memory recall test it is actually very well made and covers everything it really needs to cover over a good period of time. The only real extension I would make would be to expand the length, test more turns and maybe throw in some mid session recall test questions rather than one at the end, so that the recall as chats develop can be measured. Other than that, I don't really see any gaps in it and it has been very useful in refining my iterations in this system, I am going to probably make a bench for each surface I apply this memory to for a real measure of its capabilities, but that is down the line.

The memory revamp is one aspect, but the application is just as important and I optimised for the wrong surface. I had tunnel vision on my end goal which made the app confusing and not usable.

With the end goal being a voice assistant, that needs universal memory; you don't have chats when you are talking to an Alexa, for example, it should all be continuous. However, in a chat app you do have chats and it's the structure people are familiar with, removing that was a mistake. Before I had a general chat where you can dump ideas then auto organising projects, this does sound optimal however it introduced lots of edges and reliability issues and also was hard to explain. I had a long onboarding with a help guide modal and lots to explain, the product was bad.

I bounded the memory to single chats for this to give a familiar interface, so now you can just have chats that scale infinitely and do not slow down when context gets large or forces you to clear and start a new chat, two big pain points I felt whenever using chat apps. There is no context degrading as well in longer chats where recall dips as detail gets buried in the middle, as well as the cost decreasing in long run since input tokens stay consistent rather than growing linearly with turns, so just all round better chats.

I did keep in universal memory but bounded within projects, so any chat within a project can naturally pull from all sibling chats or attachments, anything within the project. This allows, for example, all backend in one chat, frontend in another chat, marketing in another and then cross referencing between them, clean split with ability to recall without needing to go into each chat and asking for a summary to pass through. Everything in the app is now intentional, to fix pain points that I encounter rather than stringing on features for the sake of it.

I also configured chats as attachable, just like a doc or an image. This means outside of a project other chats can still be easily mentioned, this works because of the structure we detailed before, it is not just dumping the entire chat, the other chat topics and entities are simply included in the retrieval allowing for selective retrieval but with changeable scope.

With all this put together chats can now be infinite length, maintain accurate recall and stay consistently cheap and fast no matter the length. No limit on doc and image attachments in a chat, info from other chats can be pulled and projects can be created for it all to be natural pulling without having to mention.

On top of this in the new design, memory is directly editable as facts and traces are cleanly sorted and are what is passed into the model, also making chats shareable as this is a clean format and citations and references to attachments are written into them.

I focused before too much on making the site unique, one of a kind and discarded usability for this. There was too much going on, too much neon and colour; it was an eye-sore. I toned it down and also made sure everything got a place, on the landing in particular I was adding sections just because they looked cool when they added no value. I also introduced some negative space around the outside giving a panel effect but this also made it feel more spacious to my eye which seems counter intuitive but it worked for me as the old felt very cramped. The product decisions I made also helped me cut down all the different bits I had before, making the interface much simpler as well and easier to understand, here are some before and after screenshots:

I also added a light mode variant:

Finally, I created the character surface for this memory system which can be found here at meisan.ai. I reworked the underlying structure a bit so that it worked better for what we need in a character app which is more of in story feel not losing context rather than as precise recall as the productivity variant requires.

I removed topics, I felt they were redundant here and I moved entities to the first class storage principle. There are 3 modes to character app, chat mode which is one on one AI companion style, roleplay which is multi character casts or in a story as a character, then story mode which is full third person narration.

While chat mode may work for topics, I decided it was misplaced here since the majority of detail is going to be temporary, these are conversational chats and so they don't need that level of retrieval. Roleplay and story mode are the same, for those two pulling the relevant entities are enough to retrieve the majority of the time.

The shape this app takes is entities which have attributes and then event logs. These follow the same fact and trace format and the rolling ledger, harvest that all still lives from pre convergence, it is just the structure which we extract into which changed. Pre convergence design actually does make more sense here since a character surface doesn't need as precise recall, it needs to stay in story and in character. Also the issues outlined with the original harvest system do not apply here since the ledger logs are copied verbatim into trace anyway for events. Attributes on entities are durable facts about the entity and are always injected into stage 2 when the entity is selected by retrieval step. The rest of the info is then in these event logs which are a fact covering the key events that happened and then the trace containing all the detail.

I rate this event axis as well and it fits nicely into productivity surface and so I am planning on carrying that over at some point, however not blocking and so I deferred for now.

This worked during testing as the majority of recall needed in this surface is either going to be durable entity attributes or they will contain some event info which we can then fetch the event from. I tested out some topic iterations but none fit as cleanly as this model for this surface, as it is then very easy to edit in panel. By passing in the event facts but not their supporting detail unless retrieved, the answering model can always retain full awareness of the entire stories and facts are lean without their traces allowing this to scale.