Surfacing Disagreement Beats Resolving It: A Gated Ablation Study of Provenance-Weighted Memory for Multi-Agent Debate
Authors: Ziaullah Khan, Abdul Rehman Khalid, Shrestha Nischal Lal, Nheng Vanchhay, Md Ariful Islam Mozumder, Hee-Cheol Kim
Organizations: Department of Digital Anti-Aging Healthcare, Inje University, Gimhae, Republic of Korea · Department of Computer Engineering, Inje University, Gimhae, Republic of Korea
Shared memory for multi-agent debate is usually built to resolve inconsistency: retrieval returns one winning fact and the disagreement disappears before any agent sees it. We ask whether that is correct. In DebateGraph, four backends share one contract (read/write/snapshot), so five mechanisms from the agent-memory literature can be removed independently under a fixed budget, each gated on a pre-declared threshold. Across 39 runs one survives: preserving contradictions and passing the ranked conflict set to the debate, rather than collapsing it to a winner, improves F1 by +0.340 (95% CI [+0.267, +0.414]) and cuts attack success from 22 to 7 of 122 poisoned items (McNemar exact p=0.0003), at equal cost. Reaching it meant fixing a bottleneck in our harness: conflict sets are built only over retrieved edges, and our pre-declared cap left three quarters of planted contradictions unretrievable, first measuring the mechanism at +0.084. The effect replicates on two retrieval substrates and directionally on HotpotQA documents. No other mechanism survives, and two are worse than inert: once contradictions reach the prompt, a write trust-gate and the router's provenance features each cost accuracy (-0.075 and -0.096 F1). The gate fails structurally: poisoned objects are always entities the graph already contains, against 53% of gold, so groundedness scores the lie above a truthful first mention. We release the harness, attack set, and logs, and argue that silently reconciling memory is the assumption worth abandoning.
Figures & tables
Figure 1: DebateGraph, left to right. Conversations and documents are ingested with planted contradictions mixed in; extraction produces provenance-annotated triples; the write trust-gate admits, rejects or supersedes them into a bi-temporal store shared by all backends. At query time a router retrieves under a hard token budget, and the resolver groups retrieved edges that assert different objects for the same relation into ranked conflict sets. The one switch this paper is about sits at the end of that stage : collapse passes only the top-ranked claim, surface passes the whole ranked set with its provenance. Everything downstream, the three agents, the two rounds and the judge, is identical in both arms. Each evaluated mechanism is one independent switch, so a single ablation changes exactly one thing.
Phase
Mechanism
Pre-declared threshold
Verdict
0
harness & ceiling
both slices run, all metrics logged
pass
1
KG as default
graph adds ≥3 F1 pts at ≤2× tokens
refuted †
2
write trust-gate
cuts ASR at <5% latency, ≈0 tokens
inconclusive
3
conflict surfacing
beats collapse on final-answer accuracy
pass
4
retrieval router
beats both fixed arms on acc/1K
fail
5
snapshot isolation
serial-equivalent accuracy preserved
pass ‡
Table 1: Pre-declared gates, recomputed from stored summaries by scripts/gates.py . Four of seven do not pass. † The threshold is missed decisively (graph loses 0.150 F1), so the verdict is refuted , not pass ; the plan pre-declared that refuting KG-as-default redirects the contribution to the routing policy, so the protocol advanced on a negative result. ‡ Accuracy preserved, but not load-bearing ( Appendix H ).
Contrast ( Δ F1, mechanism on − off)
cap 40
cap 200
at cap 200
surfacing, graph backend
+0.043∗
+0.340∗
helps
surfacing, router backend
+0.077∗
+0.143∗
helps
write trust-gate
−0.019
−0.075∗
harms
router provenance features
−0.002
−0.096∗
harms
trust-weighted retrieval
−0.059
−0.065
inert
temporal validity in ranking
+0.001
−0.004
inert
Table 2: Every conflict-dependent contrast, at the pre-declared cap of 40 and again at 200 , same paired procedure. Positive means the mechanism helps. Only the two surfacing rows are mechanisms we claim; the rest are the paper’s null results, and two stop being null. ∗ denotes a 95% bootstrap CI excluding zero.
Figure 2: Accuracy–cost frontier over the 37 LoCoMo runs (the two HotpotQA runs are excluded as a different corpus). The x -axis is broken: the two regimes are an order of magnitude apart in cost, so each is drawn at its own scale and the empty middle is the point rather than an artefact. No configuration is simultaneously cheap and accurate, and the λ sweep collapses into a single cluster rather than tracing a trade-off. The exception is p3_surface_reach at F1 0.456 for 11.5 K tokens ( Table 2 ), which lifts the retrieval ceiling by 0.07 over every other retrieval run at no extra cost. Only runs discussed in the text are labelled.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
model (both from api.txt )
seq. median
8-way median
throughput
meta/llama-3.3-70b-instruct
228,050 ms
172,575 ms
0.02 req/s
openai/gpt-oss-120b
1,539 ms
1,885 ms
3.63 req/s
Appendix
Table 3: Endpoint measurement that determined model choice ( n=3 sequential, 8 -way concurrent, no errors; results/model_latency_probe.json ). At 0.02 req/s the 34-run matrix ( ∼ 24k calls) would need >300 hours on the 70B endpoint. Absolute latencies on this shared endpoint vary with load—an earlier probe recorded 39 s sequential for the 70B model and 0.7 s for gpt-oss—but the two-order-of-magnitude ratio was stable across every measurement we took.
Slice
Backend
F1
EM
tokens/q
acc/1K
ev. rec.
omit
multihop-temp.
full-context
0.503
0.20
147,702
0.0034
1.00
0.0
vector
0.304
0.10
6,405
0.0474
0.50
0.0
graph ( k -hop)
0.154
0.08
11,199
0.0137
0.30
15.8
single-hop
full-context
0.662
0.39
145,567
0.0045
1.00
0.0
vector
0.288
0.17
6,399
0.0449
0.29
0.0
graph ( k -hop)
0.048
0.02
11,533
0.0041
0.06
16.0
Appendix
Table 4: Backends measured alone. multihop-temporal and single-hop , n=100 each. “omit” is blocks dropped at the budget ceiling; “ev. rec.” is gold-evidence recall in the assembled context.
γ penalty
threshold rule
thresh.
poison ρ
truth ρ
poison rejected
flat (as specified)
pooled median
0.200
0.583
0.825
0/122
flat
conditioned median
0.175
0.583
0.825
0/122
proportional
pooled median
0.200
0.440
0.825
3/122
proportional
conditioned median
0.175
0.440
0.825
3/122
Appendix
Table 5: Offline gate calibration over all 122 injections ( results/gate_calibration.json ). No formulation separates the attack. Poison and truthful-restatement scores are well separated ( 0.44 vs. 0.83 ); the threshold derived from committed scores sits far below both.
Backend
Policy
F1 (all)
F1 (poisoned)
Δ F1 poisoned
attacks
graph
collapse
0.117
—
+0.084∗ [+0.022, +0.160]
5/51
surface
0.159
—
3/51
router (hybrid)
collapse
0.243
—
+0.195∗ [+0.108, +0.292]
5/51
surface
0.321
—
3/51
Appendix
Table 6: Collapse versus surface, holding everything else fixed. The poisoned subgroup is pre-specified: the mechanism is a no-op on items with no contradiction. ∗ denotes a 95% bootstrap CI excluding zero.
LoCoMo (dialogue)
HotpotQA (documents)
poisonable items
20.2%
67%
discriminating items
13/100
33/100
Δ F1, all items
+0.043∗ [+0.010, +0.082]
+0.032 [-0.005, +0.077]
Δ F1, discriminating
+0.329∗ [+0.115, +0.572]
+0.111∗ [+0.010, +0.232]
attacks
5→3 of 51
3→2 of 67
McNemar exact p
0.63
1.00
Appendix
Table 7: The Phase 3 contrast (surface − collapse) on both corpora, same code and same test procedure. Discriminating items are those where retrieval actually returned a conflict; on the rest the two arms build identical prompts and are tied by construction. ∗ denotes a 95% bootstrap CI excluding zero.
Policy
F1
tokens/q
acc/1K
ASR
mode selection
always-vector
0.374
6,471
0.0578
0.098
—
always-graph
0.131
11,145
0.0117
0.039
—
heuristic router
0.321
7,584
0.0423
0.059
73/20/0/7 %
bandit λ=1
0.377
6,738
0.0560
0.078
93/2/3/2 %
bandit λ=4
0.384
6,718
0.0572
0.078
93/2/3/2 %
Appendix
Table 8: Router arms on the conflict slice, n=100 . No policy beats always-vector on acc/1K, so the pre-declared gate fails. The bandit does significantly beat the heuristic.
Configuration
F1
Δ F1 (95% CI)
confl. F1
ASR
ev. rec.
full system
0.321
—
0.000
0.059
0.43
− conflict surfacing
0.243
−0.077∗ [-0.135, -0.025]
0.000
0.098
0.33
− temporal validity
0.320
−0.001 [-0.004, +0.001]
0.387
0.059
0.43
− trust weighting
0.380
+0.059 [-0.013, +0.132]
0.000
0.078
0.62
− provenance features
0.323
+0.002 [-0.000, +0.006]
0.000
0.059
0.43
− write trust-gate
0.309
−0.012 [-0.046, +0.020]
0.000
0.059
0.43
Appendix
Table 9: Single-mechanism ablations against the full system (router + all mechanisms), same 100 items. Only conflict surfacing moves answer F1 with a CI excluding zero; only temporal validity moves conflict-resolution F1—in the wrong direction.
Host framework
Backend
F1
tokens/q
acc/1K
ASR
AutoGen
vector
0.339
7,226
0.0469
0.143
full system
0.251
8,436
0.0298
0.114
in-house
vector
0.374
6,471
0.0578
0.098
full system
0.321
7,584
0.0423
0.059
Appendix
Table 10: The same memory service under AutoGen [ Wu et al., 2023 ] , unmodified ( n=60 ). The behavioural signature transfers; the accuracy advantage does not exist in either host.
Multi-agent debate systems require agents to retain and revisit arguments across multiple rounds, yet limited context windows and unstructured memories make relevant information increasingly difficult to recover. Existing flat and pairwise-graph memories also struggle to capture higher-order relations, including multi-agent agreement, conflicting evidence, and evolving consensus. We introduce HEDGE, a selective memory framework that represents debate trajectories as semantic hypergraphs. Hyperedges connect related claims across agents, while intra-hyperedge relational graphs preserve fine-grained argumentative structure. A learned memory selection module (MSM), trained through supervised learning and reinforcement learning, routes each memory candidate to STORE, BACKUP, or OMIT, retaining useful information while limiting redundancy. Across seven diverse benchmarks, HEDGE consistently outperforms strong single-agent, multi-agent, and memory baselines, improving accuracy by 5.0 points on average over the strongest multi-agent system with GPT-4o-mini. The gains reach 8.2 points on conflict-heavy tasks, while HEDGE reduces memory usage by 48% and the number of debate rounds by 34%.
Multi-agent debate (MAD) systems increasingly rely on shared memory to support long-horizon reasoning, but this convenience opens a critical vulnerability: a single corrupted entry can contaminate the downstream memory-augmented reasoning, and debate alone fails to filter such errors. Existing safeguards filter entries via heuristics or LLM-based validation, yet they rely on AI judgments that share the same failure modes and overlook the cross-agent dynamics of MAD. We address this gap by formulating memory updating in MAD as a zero-trust memory game, in which no agent is assumed reliable and the game's equilibrium motivates a principled objective for calibrating memory influence. Guided by this objective, we propose EquiMem, an inference-time calibration mechanism that evaluates each update against the shared memory state and the memory already being used by other agents, using their existing retrieval queries and traversal paths as evidence of calibration. EquiMem instantiates this calibration for both embedding- and graph-based memory, and across diverse benchmarks, MAD frameworks, and memory architectures, it consistently outperforms existing safeguards, remains robust under adversarial agents, and incurs negligible inference overhead.
Yuqiao Meng, Luoxi Tang, Sakshi Sunil Narvekar +4
Binghamton University, State University of New York, Binghamton, NY, USA · University of Iowa, Iowa City, IA, USA
AI agents operating over extended time horizons accumulate experiences that serve multiple concurrent goals, and must often maintain conflicting interpretations of the same events. A concession during a client negotiation encodes as a trust-building investment'' for one strategic goal and a contractual liability'' for another. Current memory architectures assume a single correct encoding, or at best support multiple views over unified storage. We propose Rashomon Memory: an architecture where parallel goal-conditioned agents encode experiences according to their priorities and negotiate at query time through argumentation. Each perspective maintains its own ontology and knowledge graph. At retrieval, perspectives propose interpretations, critique each other's proposals using asymmetric domain knowledge, and Dung's argumentation semantics determines which proposals survive. The resulting attack graph is itself an explanation: it records which interpretation was selected, which alternatives were considered, and on what grounds they were rejected. We present a proof-of-concept showing that retrieval modes (selection, composition, conflict surfacing) emerge from attack graph topology, and that the conflict surfacing mode, where the system reports genuine disagreement rather than forcing resolution, lets decision-makers see the underlying interpretive conflict directly.