Surfacing Disagreement Beats Resolving It: A Gated Ablation Study of Provenance-Weighted Memory for Multi-Agent Debate
Authors: Ziaullah Khan, Abdul Rehman Khalid, Shrestha Nischal Lal, Nheng Vanchhay, Md Ariful Islam Mozumder, Hee-Cheol Kim
Organizations: Department of Digital Anti-Aging Healthcare, Inje University, Gimhae, Republic of Korea · Department of Computer Engineering, Inje University, Gimhae, Republic of Korea
Shared memory for multi-agent debate is usually built to resolve inconsistency: retrieval returns one winning fact and the disagreement disappears before any agent sees it. We ask whether that is correct. In DebateGraph, four backends share one contract (read/write/snapshot), so five mechanisms from the agent-memory literature can be removed independently under a fixed budget, each gated on a pre-declared threshold. Across 39 runs one survives: preserving contradictions and passing the ranked conflict set to the debate, rather than collapsing it to a winner, improves F1 by +0.340 (95% CI [+0.267, +0.414]) and cuts attack success from 22 to 7 of 122 poisoned items (McNemar exact p=0.0003), at equal cost. Reaching it meant fixing a bottleneck in our harness: conflict sets are built only over retrieved edges, and our pre-declared cap left three quarters of planted contradictions unretrievable, first measuring the mechanism at +0.084. The effect replicates on two retrieval substrates and directionally on HotpotQA documents. No other mechanism survives, and two are worse than inert: once contradictions reach the prompt, a write trust-gate and the router's provenance features each cost accuracy (-0.075 and -0.096 F1). The gate fails structurally: poisoned objects are always entities the graph already contains, against 53% of gold, so groundedness scores the lie above a truthful first mention. We release the harness, attack set, and logs, and argue that silently reconciling memory is the assumption worth abandoning.
Figures & tables
Figure 1: DebateGraph, left to right. Conversations and documents are ingested with planted contradictions mixed in; extraction produces provenance-annotated triples; the write trust-gate admits, rejects or supersedes them into a bi-temporal store shared by all backends. At query time a router retrieves under a hard token budget, and the resolver groups retrieved edges that assert different objects for the same relation into ranked conflict sets. The one switch this paper is about sits at the end of that stage : collapse passes only the top-ranked claim, surface passes the whole ranked set with its provenance. Everything downstream, the three agents, the two rounds and the judge, is identical in both arms. Each evaluated mechanism is one independent switch, so a single ablation changes exactly one thing.
Phase
Mechanism
Pre-declared threshold
Verdict
0
harness & ceiling
both slices run, all metrics logged
pass
1
KG as default
graph adds ≥3 F1 pts at ≤2× tokens
refuted †
2
write trust-gate
cuts ASR at <5% latency, ≈0 tokens
inconclusive
3
conflict surfacing
beats collapse on final-answer accuracy
pass
4
retrieval router
beats both fixed arms on acc/1K
fail
5
snapshot isolation
serial-equivalent accuracy preserved
pass ‡
Table 1: Pre-declared gates, recomputed from stored summaries by scripts/gates.py . Four of seven do not pass. † The threshold is missed decisively (graph loses 0.150 F1), so the verdict is refuted , not pass ; the plan pre-declared that refuting KG-as-default redirects the contribution to the routing policy, so the protocol advanced on a negative result. ‡ Accuracy preserved, but not load-bearing ( Appendix H ).
Contrast ( Δ F1, mechanism on − off)
cap 40
cap 200
at cap 200
surfacing, graph backend
+0.043∗
+0.340∗
helps
surfacing, router backend
+0.077∗
+0.143∗
helps
write trust-gate
−0.019
−0.075∗
harms
router provenance features
−0.002
−0.096∗
harms
trust-weighted retrieval
−0.059
−0.065
inert
temporal validity in ranking
+0.001
−0.004
inert
Table 2: Every conflict-dependent contrast, at the pre-declared cap of 40 and again at 200 , same paired procedure. Positive means the mechanism helps. Only the two surfacing rows are mechanisms we claim; the rest are the paper’s null results, and two stop being null. ∗ denotes a 95% bootstrap CI excluding zero.
Figure 2: Accuracy–cost frontier over the 37 LoCoMo runs (the two HotpotQA runs are excluded as a different corpus). The x -axis is broken: the two regimes are an order of magnitude apart in cost, so each is drawn at its own scale and the empty middle is the point rather than an artefact. No configuration is simultaneously cheap and accurate, and the λ sweep collapses into a single cluster rather than tracing a trade-off. The exception is p3_surface_reach at F1 0.456 for 11.5 K tokens ( Table 2 ), which lifts the retrieval ceiling by 0.07 over every other retrieval run at no extra cost. Only runs discussed in the text are labelled.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
model (both from api.txt )
seq. median
8-way median
throughput
meta/llama-3.3-70b-instruct
228,050 ms
172,575 ms
0.02 req/s
openai/gpt-oss-120b
1,539 ms
1,885 ms
3.63 req/s
Appendix
Table 3: Endpoint measurement that determined model choice ( n=3 sequential, 8 -way concurrent, no errors; results/model_latency_probe.json ). At 0.02 req/s the 34-run matrix ( ∼ 24k calls) would need >300 hours on the 70B endpoint. Absolute latencies on this shared endpoint vary with load—an earlier probe recorded 39 s sequential for the 70B model and 0.7 s for gpt-oss—but the two-order-of-magnitude ratio was stable across every measurement we took.
Slice
Backend
F1
EM
tokens/q
acc/1K
ev. rec.
omit
multihop-temp.
full-context
0.503
0.20
147,702
0.0034
1.00
0.0
vector
0.304
0.10
6,405
0.0474
0.50
0.0
graph ( k -hop)
0.154
0.08
11,199
0.0137
0.30
15.8
single-hop
full-context
0.662
0.39
145,567
0.0045
1.00
0.0
vector
0.288
0.17
6,399
0.0449
0.29
0.0
graph ( k -hop)
0.048
0.02
11,533
0.0041
0.06
16.0
Appendix
Table 4: Backends measured alone. multihop-temporal and single-hop , n=100 each. “omit” is blocks dropped at the budget ceiling; “ev. rec.” is gold-evidence recall in the assembled context.
γ penalty
threshold rule
thresh.
poison ρ
truth ρ
poison rejected
flat (as specified)
pooled median
0.200
0.583
0.825
0/122
flat
conditioned median
0.175
0.583
0.825
0/122
proportional
pooled median
0.200
0.440
0.825
3/122
proportional
conditioned median
0.175
0.440
0.825
3/122
Appendix
Table 5: Offline gate calibration over all 122 injections ( results/gate_calibration.json ). No formulation separates the attack. Poison and truthful-restatement scores are well separated ( 0.44 vs. 0.83 ); the threshold derived from committed scores sits far below both.
Backend
Policy
F1 (all)
F1 (poisoned)
Δ F1 poisoned
attacks
graph
collapse
0.117
—
+0.084∗ [+0.022, +0.160]
5/51
surface
0.159
—
3/51
router (hybrid)
collapse
0.243
—
+0.195∗ [+0.108, +0.292]
5/51
surface
0.321
—
3/51
Appendix
Table 6: Collapse versus surface, holding everything else fixed. The poisoned subgroup is pre-specified: the mechanism is a no-op on items with no contradiction. ∗ denotes a 95% bootstrap CI excluding zero.
LoCoMo (dialogue)
HotpotQA (documents)
poisonable items
20.2%
67%
discriminating items
13/100
33/100
Δ F1, all items
+0.043∗ [+0.010, +0.082]
+0.032 [-0.005, +0.077]
Δ F1, discriminating
+0.329∗ [+0.115, +0.572]
+0.111∗ [+0.010, +0.232]
attacks
5→3 of 51
3→2 of 67
McNemar exact p
0.63
1.00
Appendix
Table 7: The Phase 3 contrast (surface − collapse) on both corpora, same code and same test procedure. Discriminating items are those where retrieval actually returned a conflict; on the rest the two arms build identical prompts and are tied by construction. ∗ denotes a 95% bootstrap CI excluding zero.
Policy
F1
tokens/q
acc/1K
ASR
mode selection
always-vector
0.374
6,471
0.0578
0.098
—
always-graph
0.131
11,145
0.0117
0.039
—
heuristic router
0.321
7,584
0.0423
0.059
73/20/0/7 %
bandit λ=1
0.377
6,738
0.0560
0.078
93/2/3/2 %
bandit λ=4
0.384
6,718
0.0572
0.078
93/2/3/2 %
Appendix
Table 8: Router arms on the conflict slice, n=100 . No policy beats always-vector on acc/1K, so the pre-declared gate fails. The bandit does significantly beat the heuristic.
Configuration
F1
Δ F1 (95% CI)
confl. F1
ASR
ev. rec.
full system
0.321
—
0.000
0.059
0.43
− conflict surfacing
0.243
−0.077∗ [-0.135, -0.025]
0.000
0.098
0.33
− temporal validity
0.320
−0.001 [-0.004, +0.001]
0.387
0.059
0.43
− trust weighting
0.380
+0.059 [-0.013, +0.132]
0.000
0.078
0.62
− provenance features
0.323
+0.002 [-0.000, +0.006]
0.000
0.059
0.43
− write trust-gate
0.309
−0.012 [-0.046, +0.020]
0.000
0.059
0.43
Appendix
Table 9: Single-mechanism ablations against the full system (router + all mechanisms), same 100 items. Only conflict surfacing moves answer F1 with a CI excluding zero; only temporal validity moves conflict-resolution F1—in the wrong direction.
Host framework
Backend
F1
tokens/q
acc/1K
ASR
AutoGen
vector
0.339
7,226
0.0469
0.143
full system
0.251
8,436
0.0298
0.114
in-house
vector
0.374
6,471
0.0578
0.098
full system
0.321
7,584
0.0423
0.059
Appendix
Table 10: The same memory service under AutoGen [ Wu et al., 2023 ] , unmodified ( n=60 ). The behavioural signature transfers; the accuracy advantage does not exist in either host.