Imagine you had a great tutor, but they forgot the material you covered and your prior knowledge after every session. The work you’d have to do to get the tutor up to speed at every session may outweigh the learnings from the tutor itself! That describes the issues of working with LLMs that don’t have a memory system - given that these machines are by default stateless, agents without memory infrastructure lead to disconnected sessions, require heavy prompting work, and are just not as effective as they could be.
Which begets the market for agent memory infrastructure. From a pure technical standpoint, these products are a base vector database/graph database under the hood with additional workflows and abstractions on top. They intelligently store past chat histories (oftentimes using an LLM to discern what’s worth keeping), connect dots between facts mentioned and disparate chat histories to build a profile on the user that strengthens over time, and have search/traversal capabilities such that an agent is able to retrieve this important information at runtime.
A few things on my mind when thinking about this market:
1) Are we in for a new unit of abstraction, a new data structure, around memory? Letta calls it memory blocks, Zep AI (YC W24) calls it a context block, cognee calls it a DataPoint. Irrespective of naming, they approximate the same thing: an abstraction upon data snippets that breaks it down into modular, manageable, purposeful chunks that can be persisted. This makes memory programmable at design time vs forcing all the context assembly to happen at runtime.
2) What is the closest historical comparable to the nascent agent memory space? The data geek in me likens this to the data lakehouse space: just as the lakehouse boils down to object storage with core abstractions on top to enable querying and governance, memory systems are vector/graph/relational DBs (for metadata/provenance) at the base layer with memory-specific abstractions and workflow on top. But similarly, if one is to think of these products as database wrappers, it bodes a parallel to the original system of record products. What if memory systems are intelligent, horizontal systems of records for agents? Instead of being optimized for UI/UX like classic system of records are, what if they’ll be optimized for agent recall and AX (agent experience)?
Still very early here but great to see the lessons learned from those at the agentic frontier. If you’re a developer/AI engineer in New York working on similar problems, I would love to meet - coffee on me!
👋 I’m a Researcher at Work-Bench, a Seed stage enterprise-focused VC fund based in New York City. Our sweet spot for investment at Seed correlates with building out a startup’s early go-to-market motions.
