A conversational agent that remembers has to decide what to keep. Retrieval-augmented generation keeps the transcript and searches it; the alternative is to compact each turn into atomic facts and search those. We compare the two on LoCoMo, with both arms answering all 1,985 questions in one window, against the same stores, with the same generator and judge, and both predictions on one row, which licenses McNemar's exact test on the 512 questions where they disagree. On the four categories comparable work reports, compaction scores 71.4 per cent against 64.6, a lead of 6.8 points (p = 5.5e-08) at 34 per cent fewer prompt tokens. On all five, including the adversarial set comparable work drops, the lead falls to 2.0 and stops being significant, because the baseline leads adversarial by 14.6. That loss has a structure. Our lead falls as the reference answer demands more source wording, 9.6 points ahead at one or two reference words and 3.1 behind past eight, with token overlap moving the same way, so the judge is not the cause. Compaction keeps what a fact asserts and discards how it was said, so it wins where the answer is a value and loses where it is a phrasing. We also report four negative results with their power calculations.
Figures & tables
Figure 1: The two paths through the system. Write path across the top, read path across the bottom, the two storage tiers in the middle. The dotted return arrow is reinforcement: reading a fact rewrites it.
Category
n
NMAFC
RAG
Lead
Won/lost
McNemar p
Context
single-hop
281
50.9 [45.1, 56.7]
45.6 [39.8, 51.4]
+5.3
51/36
0.133
980 vs 1,486
temporal
321
71.7 [66.5, 76.3]
46.1 [40.7, 51.6]
+25.5
101/19
1.1e-14
943 vs 1,422
multi-hop
96
57.3 [47.3, 66.7]
44.8 [35.2, 54.7]
+12.5
15/3
0.0075
961 vs 1,410
open-domain
841
79.8 [76.9, 82.4]
80.3 [77.4, 82.8]
-0.5
71/75
0.804
967 vs 1,461
adversarial
446
35.9 [31.6, 40.4]
50.4 [45.8, 55.1]
-14.6
38/103
4.1e-08
960 vs 1,486
scored (4)
1,539
71.4 [69.1, 73.6]
64.6 [62.2, 66.9]
+6.8
238/133
5.5e-08
964 vs 1,454
Table 1: Paired LoCoMo result, 1,985 questions, both arms in one window against the same stores. Intervals are Wilson 95 per cent. Discordant counts are questions won and lost. Context is mean retrieved tokens.
Reference length
n (all 5)
Lead, all 5
p
n (scored 4)
Lead, scored 4
p
1 to 2 words
683
+6.0
0.0027
532
+9.6
1.2e-05
3 to 4 words
498
+0.8
0.781
375
+7.5
0.0023
5 to 8 words
510
+2.5
0.305
404
+8.2
0.0017
9 or more words
294
-6.1
0.057
228
-3.1
0.410
Table 2: The paired lead as a function of reference-answer length. The split is on the reference, so it is fixed before either arm runs.
Figure 2: Left, the paired lead by reference-answer length, on all five categories and on the scored four. Right, mean token-overlap F1 against the reference by category, which has no model in the loop.
Parameter
Value
Role
lambda_core_anchor
0.0
core anchors never decay
lambda_active_context
0.05
base decay rate per turn
lambda_ephemeral
0.69
base decay rate per turn
eta
0.15
consolidation constant
beta
0.0
clustering protection, ablation control
w_prune
0.1
demotion threshold
Table 3: Shipped configuration. Every parameter that differs from zero or from a library default is listed. Values marked as ablation controls are inactive at their defaults and exist so that the corresponding mechanism can be switched off and attributed.
Figure 3: Decay under the shipped constants. Left, the three memory types at k=0 . Centre, the effect of retrieval count on an active-context fact. Right, one fact through three retrievals; each resets the weight to 1 and lowers the rate for the next interval.
Paired questions
Expected discordant
Smallest detectable lead
108
27
9.4 points
214
54
6.7 points
322
80
5.5 points
830
208
3.4 points
1,539
385
2.5 points
1,985
496
2.2 points
Table 4: Smallest lead detectable by a two-sided paired test at α=0.05 , at the discordance rate of 0.25 observed here.
Figure 4: Discordant questions per category. Bars to the right are questions only NMAFC answered correctly, bars to the left only the baseline. Agreement counts and exact p-values are on the right. The information in the agreements is zero, which is the argument for pairing.
Conversation
n
NMAFC
RAG
Lead
Won/lost
p
conv-26
199
68.3
58.8
+9.5
42/23
0.025
conv-30
105
57.1
55.2
+1.9
16/14
0.856
conv-41
193
66.3
60.6
+5.7
33/22
0.177
conv-42
259
56.4
59.5
-3.1
30/38
0.396
conv-43
242
60.3
61.6
-1.2
27/30
0.791
conv-44
158
64.6
58.9
+5.7
25/16
0.211
Table 5: Paired accuracy by conversation, all five categories. Won and lost are the discordant counts, and p is McNemar exact. No single conversation is significant after any correction for ten comparisons, which is the expected result at these sample sizes given the minimum detectable effects above.
Figure 5: Accuracy by category with Wilson 95 per cent intervals, plotting Table 1 . Significance markers are McNemar exact on the paired discordant counts, so they do not follow from whether the intervals overlap.
Figure 6: Left, the open-domain accuracy and context of three retrieval widths against the baseline. Right, all five categories; each arrow runs from the baseline’s position to ours, so an arrow pointing up and to the left is a win on both axes.
Figure 7: Latency percentiles on a log scale, and the distribution clipped at ten seconds. The arms separate only above the ninetieth percentile. Both maxima near 93.8 seconds are the provider’s retry ladder rather than either system’s work.
Question
Reference
NMAFC
RAG
What is Melanie excited about in her adoption process?
creating a family for kids who need one
inspired by her friend’s positive experience
making a family for kids who need one
What did Caroline and her family do while camping?
explored nature, roasted marshmallows, and went on a hike
Caroline did not go camping with her family
explored nature, roasted marshmallows, hiked
What motivated Melanie to pursue counseling?
her own journey and the support she received, and how counseling improved her life
Melanie is not pursuing counseling
her own journey and the support she got
What inspired Melanie’s painting for the art show?
visiting an LGBTQ center and wanting to capture unity and strength
nature and sunsets
the LGBTQ center visit
How did Caroline’s son handle the accident?
He was scared but reassured by his family
Caroline does not have a son mentioned in the facts
He was scared but reassured and explained he would be OK.
What does Jon’s tattoo symbolize?
Freedom and expressing himself through dance
No information available
freedom, dancing without worrying what people think
Table 6: Adversarial questions the baseline answered and we did not. Rows one to four show wording drift, where our answer is defensible and shares almost nothing with the reference. Rows five and six show the other failure, where retrieval missed and our arm declined.
Question
Reference
NMAFC
RAG
When did Melanie paint a sunrise?
2022
2022
last year
When is Melanie planning on going camping?
June 2023
June 2023
next month
When did Caroline give a speech at a school?
The week before 9 June 2023
early June 2023
the week before 25 May 2023
How long ago was Caroline’s 18th birthday?
10 years ago
ten years ago
Not mentioned in the excerpts.
When did Caroline have a picnic?
The week before 6 July 2023
late June or early July 2023
yesterday
When did Caroline go to the LGBTQ conference?
10 July 2023
10 July 2023
two days ago on 4 July 2023
Table 7: Temporal questions we answered and the baseline did not. The baseline’s answers are frequently the relative phrase from the retrieved chunk, unresolved.
Mechanism
n
Variant
Shipped
Won/lost
p
list-shape gate, wide retrieval
214
36.4
36.9
9/10
1.0
short-phrase clause, all
1,984
63.4
63.4
72/72
1.0
short-phrase clause, adversarial
445
41.1
35.7
41/17
0.0022
short-phrase clause, temporal
321
67.6
71.7
2/15
0.0024
exact-wording clause, both
767
49.5
50.8
30/40
0.282
exact-wording clause, adversarial
446
36.1
35.9
28/27
1.0
Table 8: Three of the four mechanisms of Section 5.6, with the paired counts behind each. Won and lost are discordant questions counted for the variant against the shipped setting, and p is McNemar exact. The weight signal is excluded because it was measured as answer reachability rather than as a paired accuracy test.
File
Contents
data/locomo_all.json
1,985 rows, one per question, both arms’ predictions, correctness, retrieved-context size and latency
data/od_ship.json
841 open-domain rows at the shipped retrieval width
data/od_ground.json
841 open-domain rows with the grounding term active
data/od_wide.json
841 open-domain rows at ten hydrated turns
data/lg_ship.json
214 rows where the list-shape detector fires, shipped width
data/lg_wide.json
214 rows where the list-shape detector fires, wide
Table 9: The files released with the paper for recomputing every number and figure in it. The scripts share no state and none of them contacts a model.