LoCoMo is a widely reported benchmark for long-term conversational memory, but published scores mix memory quality with choices of answer prompt, judge and baseline layout that are rarely reported together. We measure how much these choices move a score, and evaluate Becos, a team memory system for coding agents, as a black box under all of them: ten conversations (1,540 questions), two answer models (gpt-4o-mini, gpt-4.1-mini), two answer prompts and three LLM judges, against a whole-conversation baseline rerun under identical conditions. Changing only the judge moves a score by up to 7.2 points and changing only the answer prompt by up to 9.3. A blind human check of 150 answers finds the judges reliable when they agree, and the primary judge's errors on disputed answers working against Becos rather than for it. Becos answers from a context with 90% fewer characters than the whole conversation. Its configuration was developed on LoCoMo itself, so there is no held-out set. With gpt-4o-mini answers it scores clearly higher than the whole conversation (79.7 vs 73.0 with the short prompt, paired 95% interval for the difference +3.8 to +9.5; 88.8 vs 77.5 with a step-by-step prompt). With gpt-4.1-mini answers the two score about the same (83.2 vs 83.1; 92.5 vs 92.1). We release every answer and verdict and the scoring script.
Figures & tables
Figure 1: Experiment overview. Each question is answered from one of three contexts, by one of two answer models with one of two answer prompts, and every answer is graded by three judges.
Figure 2: Becos (circles) against the whole conversation (squares) on all 1,540 questions, for each answer model and prompt (rows) and each judge (panels). Numbers give Becos’s difference in points, computed before rounding.
Answer model
Answer prompt
Context
Judge
4o-mini
4.1-mini
DeepSeek
gpt-4o-mini
short
No memory
19.5
24.5
–
gpt-4o-mini
short
Whole conversation
73.0
75.5
68.2
gpt-4o-mini
short
Becos
79.7
83.6
78.6
gpt-4o-mini
7-step
Whole conversation
77.5
75.6
70.8
gpt-4o-mini
7-step
Becos
88.8
87.5
83.2
Table 1: All ten conversations (1,540 questions). LLM-judge score (%) by answer model, answer prompt, context and judge. The no-memory row was not judged by DeepSeek.
All ten conversations (1,540)
conv-47 to conv-50 (655)
Answers
Prompt
Extra correct
Difference
95% interval
Becos
Whole
gpt-4o-mini
short
+104
+6.8
[+3.8, +9.5]
78.8
70.5
gpt-4o-mini
7-step
+173
+11.2
[+9.2, +13.1]
89.3
77.6
gpt-4.1-mini
short
+1
+0.1
[-2.3, +2.2]
83.2
82.3
gpt-4.1-mini
7-step
+5
+0.3
[-0.6, +1.1]
93.4
93.3
Table 2: Becos minus the whole conversation, gpt-4o-mini judge. “Extra correct” counts the additional questions judged correct. The 95% interval is a paired bootstrap that resamples whole conversations (20,000 replicates), conditional on the saved answers and verdicts; with ten conversations it is approximate. The last two columns are the four conversations least used during development (Section 3.3 ).
Table 4: Per conversation, gpt-4o-mini judge. “Whole” is the whole conversation.
Answers
System
LLM score
Judge
4o-mini
Mem0 [ 3 ]
61.3
4o-mini
4o-mini
Zep [ 3 ]
58.5
4o-mini
4o-mini
LangMem [ 3 ]
51.3
4o-mini
4o-mini
Full context [ 3 ]
72.3
4o-mini
4o-mini
Nemori [ 3 ]
73.0
4o-mini
4o-mini
Whole conversation (ours, short)
73.0
4o-mini
Table 5: Overall LLM-judge score next to published results on all ten conversations. Published numbers are from Nemori’s Table 2 [ 3 ] (judge gpt-4o-mini) and from [ 5 ] (judge and answer prompt not stated). Our rows use the gpt-4o-mini judge. Matching the judge model does not make these controlled comparisons: answer prompts and other settings may differ.
Large language models (LLMs) are increas- ingly deployed as long-horizon conversational agents, motivating growing interest in mem- ory systems. However, existing benchmarks primarily evaluate memory through QA-style probing rather than in-situ conversational usage. We introduce LOCOMO-CONV, a conversa- tional memory benchmark derived from Lo- CoMo with four query styles: dialog, implicit, counterfactual, and composed. Across five rep- resentative memory systems, we evaluate both retrieval recall and end-to-end response qual- ity. Our experiments show that conversational framing exposes substantial retrieval gaps over- looked by QA benchmarks, especially on im- plicit and composed queries, which multi-facet query rewriting narrows for raw-turn mem- ory but not abstractive memory. We further find that strong retrieval does not fully trans- late into response quality, and that implicit queries exhibit silent grounding, where mem- ory improves contextual grounding without ex- plicitly surfacing the gold fact. These results point to reasoning-based memory elaboration as a promising direction, and we release aux- iliary supportive_memory annotations captur- ing conversationally useful context beyond the original gold evidence.
Large language model assistants are increasingly expected to retain and reason over information accumulated across many sessions. We introduce EngramaBench, a benchmark for long-term conversational memory built around five personas, one hundred multi-session conversations, and one hundred fifty queries spanning factual recall, cross-space integration, temporal reasoning, adversarial abstention, and emergent synthesis. We evaluate Engrama, a graph-structured memory system, against GPT-4o full-context prompting and Mem0, an open-source vector-retrieval memory system. All three use the same answering model (GPT-4o), isolating the effect of memory architecture. GPT-4o full-context achieves the highest composite score (0.6186), while Engrama scores 0.5367 globally but is the only system to score higher than full-context prompting on cross-space reasoning (0.6532 vs. 0.6291, n=30). Mem0 is cheapest but substantially weaker (0.4809). Ablations reveal that the components driving Engrama's cross-space advantage trade off against global composite score, exposing a systems-level tension between structured memory specialization and aggregate optimization.
The ability to handle long-term memory in LLMs is becoming increasingly critical, yet existing benchmarks remain English-centric and rely on aggregate retrieval metrics, failing to capture interactions between long-range context, temporal information, and reasoning. To address this, we introduce RUMBA (Russian User Memory BenchmArk) - a new benchmark for long-term conversational memory that provides a fine-grained taxonomy of memory-centric question types and a unified methodology accounting for semantic type, session scope, temporal reasoning, and the explicitness of temporal expressions. RUMBA consists of timestamped user-assistant dialogues with QA pairs requiring retrieval, combination, and reasoning across sessions. While designed for Russian, we also provide an aligned English subset under the same methodology. We evaluate contemporary memory systems and long-context models, and show how RUMBA serves as a diagnostic tool to analyze model behavior across benchmark slices and identify strengths and failure modes of different memory mechanisms.
Elizaveta Shevtsova, Inna Glebkina, Mark Baushenko +2