From the source
When an AI agent helps you work through a complex task across hundreds of messages, what happens when it runs out of memory?
The answer determines whether your agent continues productively or starts asking "wait, what were we trying to do again?"
We built an evaluation framework to measure how much context different compression strategies preserve.
After testing three approaches on real-world, long-running agent sessions (debugging, PR review, feature implementation, CI troubleshooting, data science, ML research), we found that structured summarization retains more useful information than alternative methods from OpenAI and Anthropic, without sacrificing compression efficiency.
This post walks through the problem, our methodology, concrete examples of how different approaches perform, and what the results mean for building reliable AI agents.
The problem Long-running agent sessions can generate millions of tokens of conversation history.
That far exceeds what any model can hold in working memory.
The naive solution is aggressive compression: squeeze everything into the smallest possible summary.
…
