Chain on Demand: Deferred Evidence Organization for Long-Term Conversational Memory
Authors: Di Lu, Zechen Li, Hao Xue
Organizations: University of New South Wales, Sydney · Hong Kong University of Science and Technology (Guangzhou) · Hong Kong University of Science and Technology
Long-term conversational memory faces a timing mismatch: write-time summaries and graphs organize history before a future query is known, while independent retrieval can allocate a limited evidence budget to redundant or fragmented turns. We introduce Chain on Demand (CoD), a query-time evidence organization framework over an unchanged turn-level memory. Given a broad candidate pool, CoD constructs a temporary evidence chain by allocating a fixed anchor budget according to query relevance, lexical diversity relative to selected anchors, and newly covered query terms, followed by deterministic restoration of each selected anchor's same-session neighborhood. The resulting chain is used only for the current query, while the underlying conversational memory remains unchanged. In a paired comparison over all 1,986 LoCoMo questions, CoD achieves 82.2% accuracy on 1,540 non-adversarial questions and 76.3% on all 1,986 questions. On LongMemEval-S, one fixed configuration that indexes both user and assistant turns obtains 86.6% across all 500 questions. These results show that long-term memory can preserve the original conversation while deferring evidence organization until the information need is known.
Figures & tables
Design
No generated semantic memory
Fixed query procedure
Set-aware allocation
Source-context restoration
LoCoMo dialog RAG ( Maharana et al., 2024 )
✓
✓
✗
✗
SeCom ( Pan et al., 2025 )
✗
✓
✗
✗
SetR † ( Lee et al., 2025 )
–
✓
✓
✗
CueMem ( Wang et al., 2026 )
✗
✓
✗
✓
ReFind ( Li et al., 2026a )
✓
✗
✗
✓
Chronos ( Sen et al., 2026 )
✗
✗
✗
✓
Table 1: Query-time evidence construction in representative systems. “No generated semantic memory” excludes query-independent summaries, cues, and events; sparse or dense indexes do not count. “Fixed query procedure” excludes multi-round agent search. “Set-aware allocation” evaluates each candidate against the evolving selected set. “Source-context restoration” adds original neighboring turns after selection. The columns describe mechanisms rather than an ordinal ranking. † denotes document-level RAG.
Figure 1: Chain on Demand. (a) Query-time set-aware allocation selects anchor locations from a fixed candidate pool, followed by deterministic same-session reconstruction. (b–c) An adaptive gap-directed acquisition variant iteratively updates a temporary evidence state and acquires additional anchors within the same candidate pool. The end-to-end results use the fixed-budget pipeline; the adaptive variant is evaluated separately with selection-level diagnostics.
Method
Overall ↑
Multi
Open
Single
Temp
Full Text
73.83
68.79
56.25
86.56
50.16
RAG
63.64
55.32
47.92
70.99
56.39
LangMem
55.45
52.10
41.65
62.80
43.25
Mem0
36.49
30.85
34.38
38.41
37.07
A-MEM
64.16
56.03
31.25
72.06
60.44
MemoryOS
58.25
56.74
45.83
67.06
40.19
Table 2: Cross-protocol LoCoMo context with GPT-4o-mini (%). CoD is evaluated on all 1,540 non-adversarial questions. Multi, Open, Single, and Temp denote multi-hop, open-domain, single-hop, and temporal reasoning.
Method
Temp
Multi
Know
User
Asst.
Pref.
Overall ↑
Full Text
31.58
45.45
76.92
87.14
89.29
36.67
56.89
RAG
39.85
48.48
67.95
90.00
98.21
53.33
60.90
LangMem
15.79
20.30
66.67
60.00
46.43
60.00
37.20
Mem0
40.15
46.21
70.12
81.43
41.07
60.00
53.51
A-MEM
47.36
48.87
64.11
92.86
96.43
46.67
62.20
MemoryOS
32.33
31.06
48.72
80.00
64.29
30.00
44.66
Table 3: Cross-protocol LongMemEval-S context with GPT-4o-mini (%). The CoD row comes from one fixed configuration that indexes all turns and is applied to all 500 questions. Temp, Multi, Know, User, Asst., and Pref. denote temporal reasoning, multi-session reasoning, knowledge update, single-user recall, single-assistant recall, and preference.
Method
Acc. ↑
Turns
Words
Dense-60
51.0
60.0
1,268
Dense-200
57.0
200.0
4,587
Dense + Temporal
73.0
202.87
4,566
CoD
77.0
202.82
4,575
Table 4: Evidence construction on the LoCoMo diagnostic (%, N=100 ).
Figure 2: Analysis of query-time evidence construction on the 100-question diagnostic subset. (a) Temporal reconstruction improves over independent retrieval and random expansion at similar evidence volume. (b) Combining diversity and coverage with relevance improves set-aware allocation over the relevance-only baseline. (c) Gold evidence is missed both during candidate retrieval and anchor allocation. (d) Iterative evidence acquisition improves evidence coverage in the selection-only diagnostic.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Evaluation set
N
Role in the paper
LoCoMo diagnostic
100
Configuration and mechanism analyses
LoCoMo non-adversarial
1,540
Aggregate benchmark result
LoCoMo all questions
1,986
Matched allocator and aggregate result
LongMemEval-S
500
One fixed user–assistant indexing configuration
Appendix
Table 5: Evaluation scope. The LoCoMo benchmark reports overlap with the development diagnostic subset; they are not disjoint held-out estimates.
Setting
Value
Dense retrieval
text-embedding-3-small ; Qdrant
Candidate pool / anchor budget
200 / 60 turns
Allocator weights (λ,β)
(0.06,0.10)
Same-session neighborhood
Two turns per side
Answer model / judge
GPT-4o-mini / GPT-4o-mini
Temperature / maximum output
0 / 512 tokens
Appendix
Table 6: Configuration used for the main CoD evaluations.
Comparison
Held fixed or matched
Question isolated
Relevance-only vs. CoD
Memory, questions, candidate pool, anchor budget, reconstruction, model, judge
Does set-aware allocation improve answers?
Dense + Temporal vs. CoD
Candidate pool, anchor budget, reconstruction procedure, approximate context volume
Does allocation help beyond local reconstruction?
Random vs. temporal expansion
Same relevance anchors; approximate turn count
Does local context help beyond more turns?
Token-matched random vs. temporal
Same relevance anchors; approximate evidence-token volume
Does local context help beyond more text?
Appendix
Table 7: Purpose of the controlled comparisons. Approximate matching is stated explicitly; only the allocator comparison over all 1,986 questions is the benchmark-wide matched test.
Method
Retrieval
Selection
Reconstruction
Total
Dense-60
21
–
–
21
CoD (original)
30
7046
108
7183
CoD (optimized)
25
115
122
262
Appendix
Table 8: Mean query-time memory-construction latency in milliseconds per query over 100 diagnostic questions. Stage means are rounded independently; Total is measured end to end. Caching changes the implementation but not the scoring rule.
Dense-60
CoD
Retrieved turns
60.0
202.82
Retrieved words
1,268
4,575
Prompt tokens
4,206
13,218
Completion tokens
129.4
111.1
Total tokens
4,335
13,330
Appendix
Table 9: Per-query API usage statistics over the 100 diagnostic questions. Prompt-token counts include the surrounding evaluation prompt and are therefore not equivalent to retrieved-evidence tokens.
Indexing
N
Accuracy
User-only
56
19.6
Dual-role
56
78.6
Appendix
Table 10: Role coverage on 56 assistant-memory questions (%).
Condition
Accuracy
Δ
Reconstruction and volume controls
Top-300 Raw
63.0
–
Random Expansion
67.0
–
Token-matched Random Expansion
68.4
–
Temporal Reconstruction
73.0
–
Allocator ablations
Appendix
Table 11: Reconstruction controls and allocator ablations on the fixed 100-question diagnostic set (%). Token-matched Random Expansion reports the mean over seeds 42–46. All allocator variants share candidates, budget, and reconstruction.
Outcome
Count
Fraction
Gold items
142
100.0
Selected anchor
84
59.2
Candidate, not anchor
41
28.9
Absent from candidates
17
12.0
Appendix
Table 12: Gold-evidence status before reconstruction (%).
Allocation rule
Accuracy
Relevance-only
73.0
Protected coverage
72.0
Protected D+C
73.0
CoD (unprotected R+D+C )
77.0
Gold-informed reference
78.0
Appendix
Table 13: Allocation controls on 100 diagnostic questions (%). The gold-informed reference is not deployable.