How Redis Agent Memory stacks up

On LongMemEval—a widely-used public benchmark for long-term agent memory—Redis's combined Remis + Instruct strategy scores 86.5% task-averaged accuracy. Among the production systems we reproduced on a common GPT-4o backbone, this is the top result. It comes within half a point of the best managed service, Oracle, which uses a stronger answer model, and Redis reaches this score at a fraction of the cost of the other top systems.

We also found that published memory results often don't hold up when rerun through the same judge. Our Vertex Memory Bank test scored 42.4%, well below the roughly 70% reported in the paper the service is based on. The setup of the released product differs from the one described in the paper.

An AI agent that forgets everything between sessions can’t be a useful long-term assistant. The obvious fix—keep the entire conversation history in the prompt—breaks down fast: context windows are bounded, long contexts are expensive and slow, and model accuracy degrades once the prompt runs into the hundreds of thousands of tokens. Re-reading a year of chat history on every turn is wasteful when 99% of it is irrelevant to the question being asked.

A memory system is the out-of-band store that solves this. It lets an agent recall facts, events, and preferences from prior interactions across turns, sessions, and weeks without holding every token in context. Every memory system, regardless of how it works internally, comes down to two operations:

• Ingest (write). Take conversation sessions and store them by embedding, extracting, chunking, summarizing, indexing, or graph-building. Runs once per session; can be slow and done in the background.

• Query (read). Given a question, retrieve the relevant context and answer. Runs on every user turn; must be fast and cheap.

Everything else, vector database, knowledge graph, LLM extractor, is implementation detail beneath that interface. The interesting question is which implementation actually recalls the right thing when it matters.

LongMemEval (Wu et al., ICLR 2025) is one of the most widely-used public benchmarks for this. It poses 500 questions over multi-session conversations, where each question comes with a “haystack” of prior sessions (about 24 on average) that the system must have remembered. A standardized GPT-4o binary judge scores each answer for correctness, which makes results more comparable across systems. We report task-averaged accuracy—the mean across the six question types—so that rare but hard categories aren’t drowned out by common ones.

The six question types. single-session-user, single-session-assistant, single-session preference, knowledge-update (a fact changed over time), multi-session (evidence spread across conversations), and temporal-reasoning (ordering and dates). A good memory sys tem has to handle all six, not just the easy recall cases.

How we reproduce a system. For each of the 500 examples we (1) reset the memory store, (2) in gest all haystack sessions, (3) query with the test question, and (4) send the (question, predicted answer, gold answer) triple to the official judge. Every system uses gpt-4o as the answer LLM and the same judge prompt, and we capture token usage on both the ingest and query paths so we can report real dollar costs. Every reproduced number in this document can be re-run from our evaluation harness.

Memory systems sit on a spectrum from raw evidence to compressed knowledge. The further toward compression, the more LLM work happens at ingest time and the more opinionated the stored format, but the more raw detail is lost.

Redis built and benchmarked three strategies that span this spectrum. The combination of the two stronger ones is what ranks highest among our reproduced runs.

• Instruct (LLM-extracted). An extractor reads each session and the current memory store, then creates, updates, or deletes individual facts. This can consolidate changes across sessions. Strength: deduplicated, contradiction-aware facts. Weakness: anything the extractor skips, such as narrative detail, exact quotes, timestamps, is gone.

• Remis (hybrid semantic RAG). Skips extraction entirely. It splits each session at topic-shift breakpoints, indexes every chunk with both a dense vector and a BM25 lexical signal, fuses the two scores (0.7 · cosine + 0.3 · BM25), and expands the top-k chunks with their neighbors. No LLM runs at ingest, so it’s fast (1.4 s/session) and cheap. Strength: nothing is thrown away. Weakness: noisier context.

• Remis + Instruct (combined). Run both at ingest; at query time, retrieve from both and hand the answer LLM a single labeled context block: a Conversation excerpts section (raw Remis chunks) and an Extracted memories section (Instruct facts). The two strategies fail on different question types, so their union has higher coverage than either alone—no router, no rerank, one answer call.

Figure 1: Each Redis strategy on the same 500-example LongMemEval split, same gpt-4o answer backbone, same judge. 
Combining raw evidence (Remis) with curated facts (Instruct) beats either alone.

We placed all three Redis strategies on a single leaderboard alongside every third-party system we could reproduce, plus published-but-unverified numbers for context. Among the production systems we reproduced on a common gpt-4o backbone, Remis + Instruct posts the highest task averaged accuracy, effectively level with Oracle’s managed service (86.0%, but measured on a stronger answer model). The only published numbers above it come from two research systems we have not independently reproduced.

Figure 2: LongMemEval task-averaged accuracy. Red = a Redis strategy. Purple = a third-party system we reproduced on our own harness through the official judge. Gray = self-reported by the vendor and independently verified.

Headline accuracy hides the more interesting story: why combining the raw-evidence Remis layer with the extracted Instruct facts helps. Breaking the score down by question type shows that Remis and Instruct have uneven, complementary coverage on their own, and that their union lifts the weaker categories without giving up ground on the strong ones.

Figure 3: Accuracy by question type for Remis alone, Instruct alone, and the combined Remis + Instruct pipeline. The two single strategies trade off across categories, while the combination matches or beats both on every type in this run—most sharply on verbatim assistant recall (72% → 100%) and on the two hardest types, multi-session and temporal.

The pattern is consistent: raw chunks preserve exact numbers, quotes, and dates that an extractor would compress away, while extracted facts consolidate scattered evidence and resolve contradictions. Neither is enough on its own; together they cover the board.

In a real deployment the memory layer is hit on every user turn, not just once. We model the holistic per-session cost as ingest $/sess + 5.13 × query $/Q (5.13 is the average number of user turns per session in LongMemEval), and plot it against accuracy. Redis’ strategies sit on the efficient part of that curve: comparable or better accuracy at a fraction of the cost of the other high-scoring systems.

Figure 4: Accuracy versus total memory cost per session (log scale). Redis’ best result lands at $0.07/session — roughly 5x cheaper than Mastra OM and 3.5x cheaper than emergence-fast at similar or better accuracy. “>” marks lower-bound costs where a vendor’s ingest spend isn’t visible to us.

Cost is dominated by the query side, not ingest: at five turns per session, the per-turn answer-LLM calls outweigh a one-time extraction pass. That makes a cheap, accurate query path—which is what Remis + Instruct provides—a major factor in production economics.

A useful reminder came from re-running managed cloud services through the same judge: the figures vendors (or, in Google’s case, the paper behind the service) publish and the numbers we measured often differed—sometimes by a lot, and in both directions.

Figure 5: Reported figures versus what we measured on the same benchmark and judge. Our Vertex Memory Bank run came in about 28 points below the ~70% reported in the research paper its design draws on (not a figure Google publishes for the product); Oracle landed about 8 points under its blog number; Amazon’s AgentCre came in above its self-reported figure once we enabled all three strategies. The takeaway isn’t that anyone is wrong—setups differ—it’s that it’s worth measuring under your own conditions before relying on a published number.

Why the gaps? Managed services are effectively black boxes—the extraction prompts and retrieval pipeline are vendor-controlled, and a system tuned for one kind of workload won’t necessarily fit LongMemEval’s question mix. In Vertex Memory Bank’s case the extractor favors stable persona facts and misses many one-off details, counts, and dates, so a large share of answers simply aren’t in memory at query time. And a research paper that controls the whole pipeline is not the same thing as the shipping product behind an API.

All accuracy is task-averaged on the 500-question LongMemEval small split. Red rows are Redis strategies; yellow are third-party systems we reproduced through the official judge; gray are self-reported and not verified. Total $/sess assumes 5.13 user turns per session and counts only LLM answer prompt + completion at list pricing; rows marked “>” are lower bounds where a vendor’s server-side ingest cost is not visible to us. All reproduced rows use gpt-4o as the answer LLM, except Oracle’s run (gpt-5.5 xhigh)—so its 86.0% is not directly comparable to the gpt-4o rows. Google Vertex’s ∼70% is the figure from the academic paper its design draws on, not a number Google reports for the shipping product. “Reported” figures are self-reported by each system’s authors unless noted and are not independently verified. Sources: Redis Applied AI Research evaluation decks, May 2026; per-question-type breakdowns reconcile to the task-averaged figures shown here.