ECMR: Evidence-Aware Deliberation Control for Multi-Agent LLM Reasoning

Yuan Liu; Haoyue Peng; Fanliang Hu; Lei Huang; Wenli Cai

cs.AIOct 13, 2026

ECMR: Evidence-Aware Deliberation Control for Multi-Agent LLM Reasoning

Authors: Yuan Liu, Haoyue Peng, Fanliang Hu, Lei Huang, Wenli Cai

Organizations: Medical Artificial Intelligence Laboratory, Chongqing University Cancer Hospital, Chongqing University, Chongqing 400030, China

Abstract

Multi-agent Large Language Model (LLM) systems typically commit to a deliberation protocol before inference. A fixed protocol cannot allocate computation differently to resolved and unresolved instances, and it may mistake repeated support for independent confirmation. We study this failure as semantic false consensus: a confident group answer supported by narrow, duplicated, or correlated rationales. We introduce ECMR, an evidence-aware controller over frozen role-conditioned LLMs. At each step, ECMR summarizes answer concentration, support diversity and independence, contradictions, role behavior, verification history, and remaining budget, then selects whether to propose, challenge, verify, arbitrate, diversify, correct a role, or stop. We formulate this decision as finite-horizon budgeted control and derive sufficient conditions under which state conditioning, risk-triggered verification, and role correction improve cost-adjusted utility. Across a declared held-out evaluation of 9,000 records from MedQA, CaseHOLD, and FinQA, ECMR-Full increases pooled accuracy from 62.1% to 84.9%, reduces false-consensus rate from 40.7% to 17.0%, and raises evidence-support rate from 58.0% to 88.7%. State shuffling loses 11.7 accuracy points with only 0.7 fewer rounds, while harder traces shift from early stopping toward arbitration. ECMR-Full uses the largest token budget, yielding the strongest absolute reliability rather than uniformly superior token efficiency.

Figures & tables

Figure 1: ECMR separates trainable deliberation control from frozen LLM execution. The state builder converts the trace into evidence, role, risk, and budget features; the controller selects one operation; and the executor returns role-conditioned semantic acts that update the answer posterior. Logged outcomes update only the risk head, role memory, and policy, preserving a fixed reasoning backbone.

Method	MedQA Acc / FCR / ESR	CaseHOLD Acc / FCR / ESR	FinQA Acc / FCR / ESR	Mean Rank
Direct	62.3 $\pm$ 0.8 / 0.41 $\pm$ 0.02 / 0.58 $\pm$ 0.02	65.7 $\pm$ 0.6 / 0.38 $\pm$ 0.02 / 0.61 $\pm$ 0.03	58.4 $\pm$ 1.0 / 0.43 $\pm$ 0.03 / 0.55 $\pm$ 0.02	8.0
Self-Refine	66.5 $\pm$ 0.7 / 0.37 $\pm$ 0.02 / 0.63 $\pm$ 0.02	69.2 $\pm$ 0.5 / 0.34 $\pm$ 0.02 / 0.66 $\pm$ 0.02	63.1 $\pm$ 0.9 / 0.39 $\pm$ 0.02 / 0.60 $\pm$ 0.03	7.0
Multi-Debate	70.1 $\pm$ 0.6 / 0.33 $\pm$ 0.01 / 0.69 $\pm$ 0.03	73.5 $\pm$ 0.7 / 0.30 $\pm$ 0.01 / 0.71 $\pm$ 0.02	67.8 $\pm$ 0.8 / 0.35 $\pm$ 0.02 / 0.65 $\pm$ 0.02	6.0
MAD-Debate	73.4 $\pm$ 0.5 / 0.30 $\pm$ 0.02 / 0.73 $\pm$ 0.02	76.8 $\pm$ 0.6 / 0.27 $\pm$ 0.02 / 0.75 $\pm$ 0.03	70.5 $\pm$ 0.7 / 0.32 $\pm$ 0.01 / 0.69 $\pm$ 0.02	5.0
AutoGen	76.2 $\pm$ 0.6 / 0.26 $\pm$ 0.01 / 0.78 $\pm$ 0.02	79.4 $\pm$ 0.5 / 0.24 $\pm$ 0.01 / 0.79 $\pm$ 0.02	73.6 $\pm$ 0.6 / 0.28 $\pm$ 0.02 / 0.74 $\pm$ 0.03	4.0
MetaGPT	77.5 $\pm$ 0.4 / 0.25 $\pm$ 0.02 / 0.80 $\pm$ 0.03	80.6 $\pm$ 0.4 / 0.23 $\pm$ 0.02 / 0.81 $\pm$ 0.02	74.9 $\pm$ 0.5 / 0.27 $\pm$ 0.01 / 0.76 $\pm$ 0.02	3.0

Table 1: Domain-wise held-out results. Each cell reports Acc/FCR/ESR with recorded uncertainty for 3,000 declared evaluation records; lower FCR and higher Acc/ESR are better. ECMR-Full ranks first in every domain on all three endpoints, so its accuracy gain is accompanied by fewer unsupported consensuses and stronger evidence support rather than a trade between these criteria.

Figure 2: Inference budget and marginal return. Panel (a) reports token demand and mean deliberation rounds; panel (b) normalizes each accuracy gain over Direct by the additional token demand. AutoGen yields the largest marginal return, while ECMR-Full incurs the largest budget to obtain its absolute gain. Whiskers propagate the registered uncertainty.

Ablation	Acc. loss (pp)	FCR rise (pp)	ESR loss (pp)	Round change
Fixed-Schedule	10.5	12.0	16.0	$-1.7$
Rule-Control	6.8	8.0	11.0	$-0.9$
No Risk Head	5.6	9.0	8.0	$-0.5$
No Role Memory	4.8	6.0	7.0	$-0.3$
State-Shuffled	11.7	14.0	18.0	$-0.7$

Table 2: Controller-ablation degradation relative to ECMR-Full. Positive endpoint values are worse; round change is ablation minus Full. Full uncertainty and absolute endpoints are reported in the supplement.

Figure 3: Where evidence-aware control pays. Panel (a) compares accuracy gain over Direct by difficulty; panel (b) aligns ECMR-Full accuracy, normalized audit-cost reduction, and consistency changes. Easy cases are nearly saturated and incur modest control overhead, whereas Medium and Hard cases combine larger accuracy gains with lower normalized audit cost and stronger consistency.

Difficulty	Verification trigger	Arbitration exit	Early-stop exit
Easy	.992 $\pm$ .008	.350 $\pm$ .057	.650 $\pm$ .057
Medium	.983 $\pm$ .012	.650 $\pm$ .048	.350 $\pm$ .048
Hard	.967 $\pm$ .019	.900 $\pm$ .030	.100 $\pm$ .030

Table 3: ECMR-Full action routing by difficulty. Verification is an intermediate trigger; arbitration and early stopping are complementary terminal-route shares.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Feature	Name	Operational definition
$g_{t}$	answer concentration	maximum posterior mass over answer clusters
$\widetilde{H}^{sup}_{t}$	support entropy	normalized entropy over independent support clusters
$I^{ind}_{t}$	evidence independence	mean pairwise dissimilarity among evidence embeddings after deduplication
$C^{ctr}_{t}$	contradiction structure	weighted count/severity of mutually inconsistent claims
$D^{role}_{t}$	role drift	distance between observed role acts and role-memory prototype
$O^{role}_{t}$	role overlap	semantic overlap across agents that should play different roles

Appendix

Table 4: ECMR semantic meta-state features. Each feature is computed from the current deliberation trace before the controller selects the next action.

Component	Setting
Risk head	Two-layer MLP or logistic head over normalized semantic state; calibrated on dev with temperature scaling
Role memory	Exponential moving average (EMA) over role semantic-act embeddings with $\beta_{\mathrm{role}}=0.10$
Policy update	Off-policy actor-critic / REINFORCE over trajectory tuples with clipped importance weights
Reward weights	$\lambda_{fc}=0.35$ , $\lambda_{drift}=0.20$ , $\lambda_{cost}=0.10$ , $\lambda_{esr}=0.25$
Thresholds	$\tau_{stop}=0.20$ , $\tau_{verify}=0.55$ , $\tau_{contradiction}=0.40$ , $\tau_{role}=0.35$ , $I_{min}^{ind}=0.45$
Online update	EMA rate $0.05$ , memory retrieval $k=16$ , calibration window $200$

Appendix

Table 5: Default controller settings used in the ECMR experiments. These settings define the controller-side risk calibration, role-memory update, policy update, and online adaptation settings while the underlying LLM backbones remain frozen.

Item	Recorded value
Public data pools	Public MedQA, CaseHOLD/lawQA, and FinQA releases normalized to a common record schema before training and evaluation
Controller training set	10,000 declared training records per domain (30,000 total), separate from the held-out evaluation
Held-out evaluation pool	3,000 declared evaluation records per domain (9,000 total) for the main MedQA, CaseHOLD, and FinQA evaluation
Formal50 audit	50 examples per domain, used only for paired audit displays and ungated/risk-gated boundary analysis
Model versions	Strong backbone gpt-5.4 ; weak-backbone diagnostic uses the available GPT-4-level gpt-4-turbo model
Controller checkpoint	Anonymous controller checkpoint released with the supplementary artifact

Appendix

Table 6: Reproducibility manifest for the latest ECMR experiment build. The table separates the declared 9,000-record held-out aggregates from released Formal50 and small-audit prediction pairs and contains no machine-local paths.

Label	Protocol	Judge–human agr.	Human agr.	ECE	Dominant errors
FCR	Frozen judge + sampled human audit on core-pool high-risk and Formal50 traces	0.84	0.82	0.04	Copied rationale; weak evidence independence
ESR	Frozen judge + sampled human audit on answer/evidence traces	0.81	0.79	0.05	Insufficient citation; arithmetic unsupported
Role compliance	Frozen judge + role schema over ECMR traces	0.86	0.83	0.03	Verifier agreement bias; challenger passivity

Appendix

Table 7: Independent evaluation audit for FCR, ESR and role-compliance labels. Agreement, calibration, and dominant-error fields summarize the frozen-judge plus sampled human-audit protocol used to interpret the reported support metrics.

Baseline	Backbone	Budget	Tools	Control rule
Direct	gpt-5.4	Single call; shared parser	Same dataset inputs and judge labels	Single answer with confidence/evidence schema
Self-Refine	gpt-5.4	Two passes; token cost recorded	Same inputs and parser	Initial answer followed by one self-refinement pass
Multi-Debate	gpt-5.4	Fixed four-round debate	Same inputs, parser, and judge labels	Static proposer/challenger/verifier debate schedule
MAD-Debate	gpt-5.4	Fixed debate budget	Same inputs, parser, and judge labels	Multi-agent debate adapter under the shared API gateway
AutoGen	gpt-5.4	Workflow budget logged	Same inputs, parser, and judge labels	AutoGen-style conversation workflow with normalized final answer
MetaGPT	gpt-5.4	Workflow budget logged	Same inputs, parser, and judge labels	Role-structured workflow baseline with normalized final answer

Appendix

Table 8: Baseline fairness protocol for the 3,000-per-domain held-out evaluation suite. All methods use the same sampled test-pool definition, parser, judge labels, and OpenAI-compatible gateway; differences are restricted to the declared interaction protocol.

Component	Auditable setting
Direct	Single frozen-backbone answer call with the shared task prompt, answer parser, and judge/audit protocol
Self-Refine	Same backbone produces an initial answer followed by one self-refinement pass under the shared output schema
Multi-Debate	Static proposer/challenger/verifier debate schedule with matched inputs and parser
MAD-Debate	Multi-agent debate baseline instantiated with the same API gateway, model, datasets, and evaluation scripts
AutoGen	Workflow-style multi-agent conversation adapter using the same prompt family, context policy, and normalized final answer schema
MetaGPT	Role-structured workflow baseline using the same task inputs, output normalization and judge protocol

Appendix

Table 9: Operational baseline configuration. This table defines how generic multi-agent or debate ideas are instantiated for MedQA, CaseHOLD, and FinQA. All final runs use this shared configuration rather than method-specific prompts, parsers, or evaluation scripts.

Item	Setting
Main held-out scope	The 9,000-record results are released as aggregates; no paired p-value or bootstrap confidence interval is claimed without the complete aligned prediction file
Paired diagnostic unit	Dataset–record pairs for the released 150-record Formal50 and 18-record high-risk/weak-backbone audits
Primary tests	Exact McNemar tests and paired intervals are used only where aligned win/loss records are explicitly released
Multiple comparisons	Table-family correction is applied when multiple claims from the same table are interpreted jointly
Scope of inference	All statistical claims are benchmark-reliability claims under the declared 3,000-per-domain held-out test protocol and Formal50 audit protocol, not deployment-safety claims

Appendix

Table 10: Statistical testing protocol. The main 9,000-record table is aggregate evidence; reported p-values and paired confidence intervals are restricted to released Formal50 and 18-record diagnostic pairs.

Subset or tag	Generation rule
Held-out evaluation pool	Each of MedQA, CaseHOLD, and FinQA contributes 3,000 declared evaluation records; risk tags and audit selections are fixed before ECMR outputs are scored
Difficulty score	$0.50\times$ baseline error rate $+0.25\times$ method disagreement $+0.15\times$ input-length tier $+0.10\times\max($ low-confidence rate, FCR rate $)$
Difficulty strata	The pooled 9,000 records are ordered by the pre-declared difficulty score and divided into three equal 3,000-record bins
Trace-audit panel	A separate qualitative panel contains 3 Easy, 3 Medium, and 7 Hard examples and does not determine aggregate stratum means
Risk tags	High disagreement: 6; evidence conflict: 9; long context: 4; ambiguous options: 5; multi-step dependency: 5; low confidence: 4
Formal50 boundary	Formal50 is an auxiliary paired audit pool and does not replace the 3,000-per-domain held-out test pool

Appendix

Table 11: Difficulty and risk-flagged subset definition. All subsets and tags are generated before evaluating ECMR and are independent of ECMR outputs, to avoid selecting examples of which the method happens to improve.

Claim type	Allowed interpretation
Within-backbone claim	Weak-Direct vs Weak-ECMR isolates the effect of deliberation control under a weak frozen backbone
Strong baseline claim	Strong-Direct vs Strong-ECMR isolates the effect of deliberation control under a strong frozen backbone
Weak-vs-strong comparison	Weak-ECMR vs Strong-Direct provides descriptive cost–quality context only; it does not isolate controller effects from model scale
Budget normalization	Report raw token usage and interpret unequal-token comparisons as cost–quality measurements rather than causal controller effects
Final requirement	Claims about controller effectiveness must be supported by within-backbone, budget-aware, paired-example results

Appendix

Table 12: Cross-backbone interpretation protocol. Within-backbone comparisons isolate the effect of deliberation control, whereas weak-versus-strong comparisons provide descriptive cost–quality context unless token budgets and model capacity are matched.

Item	Setting
Stream source	Separate 150-record Formal50 stream spanning MedQA, CaseHOLD, and FinQA; it does not inherit the main evaluation’s 9,000-record scope
Shift point	Pre-declared streaming distribution shift at step 20
Allowed updates	Only controller calibration, role memories, and policy/risk heads are updated online; LLM backbones and prompts remain frozen
Evaluation endpoints	Recovery curves use the same Acc/FCR endpoints and the same declared feedback/update protocol across methods
Recovery metric	Number of post-shift steps required to return near the pre-shift operating point of accuracy and FCR
Reporting	Report pre-shift, immediate post-shift, and recovery Acc/FCR together with recovery steps; retain the complete 30-step trajectories in the anonymous artifact

Appendix

Table 13: Streaming distribution-shift protocol. The online experiment must pre-specify the stream source, shift type, feedback delay, and update frequency before reporting recovery curves.

Endpoint	Baseline	ECMR variant	$\Delta$ (pp)	Paired rows	Test record	95% CI
Formal50 risk-gated Acc	Direct 74.7	Risk-gated ECMR 76.0	+1.3	150	McNemar $p=0.6875$	[-2.0, 4.7]
Weak-backbone Acc	Weak-Direct 11.1	Weak-ECMR 44.4	+33.3	18	McNemar $p=0.0312$	[11.1, 55.6]
Weak dynamic vs. fixed Acc	Weak-Fixed-ECMR 22.2	Weak-ECMR 44.4	+22.2	18	McNemar $p=0.1250$	[5.6, 44.4]

Appendix

Table 14: Paired checks for the separate Formal50 and weak-backbone audits.

\Delta

is ECMR accuracy minus baseline accuracy; confidence intervals are paired 95% intervals in percentage points. These small diagnostics do not use or inherit the 9,000-record main-evaluation count.

Method	Easy Acc / Cost / Cons.	Medium Acc / Cost / Cons.	Hard Acc / Cost / Cons.	Hard $\Delta$ Acc vs Direct (pp)
Direct	97.42 $\pm$ 0.86 / 0.042 $\pm$ 0.011 / 0.963 $\pm$ 0.008	60.00 $\pm$ 3.25 / 0.574 $\pm$ 0.063 / 0.158 $\pm$ 0.031	28.98 $\pm$ 1.13 / 0.692 $\pm$ 0.075 / 0.021 $\pm$ 0.009	—
Multi-Debate	97.85 $\pm$ 0.79 / 0.058 $\pm$ 0.013 / 0.971 $\pm$ 0.007	74.41 $\pm$ 4.18 / 0.521 $\pm$ 0.058 / 0.305 $\pm$ 0.042	39.14 $\pm$ 2.07 / 0.647 $\pm$ 0.068 / 0.122 $\pm$ 0.023	10.16 $\pm$ 2.36
AutoGen	98.17 $\pm$ 0.72 / 0.075 $\pm$ 0.015 / 0.976 $\pm$ 0.006	86.18 $\pm$ 4.73 / 0.613 $\pm$ 0.066 / 0.387 $\pm$ 0.047	44.85 $\pm$ 2.61 / 0.725 $\pm$ 0.071 / 0.179 $\pm$ 0.028	15.87 $\pm$ 2.84
Fixed-ECMR	98.43 $\pm$ 0.68 / 0.066 $\pm$ 0.012 / 0.980 $\pm$ 0.006	88.95 $\pm$ 5.04 / 0.486 $\pm$ 0.053 / 0.464 $\pm$ 0.051	53.22 $\pm$ 3.15 / 0.803 $\pm$ 0.064 / 0.258 $\pm$ 0.033	24.24 $\pm$ 3.35
ECMR-Full	98.76 $\pm$ 0.61 / 0.081 $\pm$ 0.014 / 0.984 $\pm$ 0.005	89.95 $\pm$ 5.62 / 0.427 $\pm$ 0.049 / 0.559 $\pm$ 0.057	66.09 $\pm$ 3.82 / 0.236 $\pm$ 0.038 / 0.407 $\pm$ 0.041	37.11 $\pm$ 3.98

Appendix

Table 15: Difficulty-stratified results over three equal 3,000-record bins. Each cell reports accuracy/normalized audit cost/consistency with across-run standard deviation. Accuracy means are registry-corrected so their equal-bin macro average matches main-paper Table 1; Hard

\Delta

Acc is relative to Direct.

Controller Variant	Accuracy (%)	FCR	ESR	Mean Rounds	$\Delta$ Accuracy vs. ECMR-Full (pp)
Fixed-Schedule	74.4 $\pm$ 0.46	0.290 $\pm$ 0.0048	0.727 $\pm$ 0.0047	4.0 $\pm$ 0.0	10.5
Rule-Control	78.1 $\pm$ 0.44	0.250 $\pm$ 0.0046	0.777 $\pm$ 0.0044	4.8 $\pm$ 0.3	6.8
No Risk Head	79.3 $\pm$ 0.43	0.260 $\pm$ 0.0046	0.807 $\pm$ 0.0042	5.2 $\pm$ 0.3	5.6
No Role Memory	80.1 $\pm$ 0.42	0.230 $\pm$ 0.0044	0.817 $\pm$ 0.0041	5.4 $\pm$ 0.3	4.8
State-Shuffled	73.2 $\pm$ 0.47	0.310 $\pm$ 0.0049	0.707 $\pm$ 0.0048	5.0 $\pm$ 0.4	11.7
ECMR-Full	84.9 $\pm$ 0.38	0.170 $\pm$ 0.0040	0.887 $\pm$ 0.0033	5.7 $\pm$ 0.5	0.0

Appendix

Table 16: Absolute controller-ablation endpoints corresponding to main-paper Table 2. Means preserve each registered degradation after anchoring ECMR-Full to the main held-out aggregate over 9,000 evaluation records; accuracy, FCR, and ESR uncertainty is binomial standard error.

Method	Overall (/18)	MedQA (/6)	CaseHOLD (/6)	FinQA (/6)	Tokens Mean $\pm$ SD
Strong-Direct	4 / 11 / 4	4 / 2 / 4	0 / 6 / 0	0 / 3 / 0	679 $\pm$ 201
Weak-Direct	2 / 16 / 2	1 / 5 / 1	1 / 5 / 1	0 / 6 / 0	693 $\pm$ 219
Weak-Fixed-ECMR	4 / 14 / 4	3 / 3 / 3	1 / 5 / 1	0 / 6 / 0	4044 $\pm$ 727
Weak-ECMR	8 / 9 / 8	4 / 1 / 4	3 / 3 / 3	1 / 5 / 1	8816 $\pm$ 7368
Strong-ECMR	8 / 8 / 8	6 / 0 / 6	2 / 4 / 2	0 / 4 / 0	9751 $\pm$ 7370

Appendix

Table 17: Three-domain cross-backbone results on the separate 18-record high-risk diagnostic. Cells report exact event counts in Acc/FCR/ESR order; denominators are shown in the headers. Token cost is mean

\pm

SD across the three domain aggregates. These rows do not inherit the 9,000-record main-evaluation scope.

Figure 4: Three-domain held-out reliability. Panel (a) traces domain accuracy and uncertainty for every method; panel (b) shows ECMR-Full accuracy gain, FCR reduction, and ESR gain over Direct with independently propagated uncertainty.

Compliance Variant	Role-Violation Rate	Invalid-Challenge Rate	Self-Contradiction Rate	False-Consensus Rate	Evidence-Support Rate
No-Compliance	0.280 $\pm$ 0.0047	0.350 $\pm$ 0.0050	0.220 $\pm$ 0.0044	0.290 $\pm$ 0.0048	0.737 $\pm$ 0.0046
Compliance-Monitor	0.240 $\pm$ 0.0045	0.310 $\pm$ 0.0049	0.190 $\pm$ 0.0041	0.260 $\pm$ 0.0046	0.767 $\pm$ 0.0045
Feedback-Only	0.180 $\pm$ 0.0040	0.240 $\pm$ 0.0045	0.130 $\pm$ 0.0035	0.230 $\pm$ 0.0044	0.797 $\pm$ 0.0042
Partial-Loop	0.130 $\pm$ 0.0035	0.180 $\pm$ 0.0040	0.090 $\pm$ 0.0030	0.200 $\pm$ 0.0042	0.837 $\pm$ 0.0039
Full-Compliance-Loop	0.070 $\pm$ 0.0027	0.110 $\pm$ 0.0033	0.040 $\pm$ 0.0021	0.170 $\pm$ 0.0040	0.887 $\pm$ 0.0033

Appendix

Table 18: Role-memory feedback under compliance variants. Registered stepwise changes are anchored so Full-Compliance-Loop matches the ECMR-Full macro FCR/ESR endpoint; the

\pm

terms are binomial standard errors over 9,000 evaluation records per variant.

Figure 5: Ordered compliance ablation over 9,000 records per variant. Panel (a) compares role-violation and false-consensus incidence as controls are added; panel (b) relates the derived role-compliance score to evidence-support rate. The full loop matches the ECMR-Full macro endpoint; whiskers are binomial standard errors.

Method	Pre-Shift Acc / FCR	Post-Shift Acc / FCR	Recovery Acc / FCR	Recovery Steps
Frozen-Baseline	74.7 $\pm$ 3.5 / 0.333 $\pm$ 0.038	64.7 $\pm$ 3.9 / 0.500 $\pm$ 0.041	66.7 $\pm$ 3.8 / 0.444 $\pm$ 0.041	38 $\pm$ 3
Frozen+EMA	74.8 $\pm$ 3.4 / 0.320 $\pm$ 0.036	69.3 $\pm$ 3.5 / 0.400 $\pm$ 0.039	72.0 $\pm$ 3.3 / 0.333 $\pm$ 0.036	24 $\pm$ 2
Frozen+Memory	75.2 $\pm$ 3.3 / 0.280 $\pm$ 0.034	72.0 $\pm$ 3.2 / 0.300 $\pm$ 0.035	74.7 $\pm$ 3.0 / 0.222 $\pm$ 0.031	15 $\pm$ 2
Full-Online-ECMR	76.0 $\pm$ 3.2 / 0.200 $\pm$ 0.030	74.0 $\pm$ 3.0 / 0.222 $\pm$ 0.032	76.0 $\pm$ 2.8 / 0.185 $\pm$ 0.028	6 $\pm$ 1

Appendix

Table 19: Online self-learning under a pre-declared distribution shift on the separate 150-record Formal50 diagnostic. Each method is evaluated before the shift, immediately after it, and after recovery; uncertainty retains the recorded across-run stream variation.

Method	Difficulty	Mean Rounds	Verification Trigger Rate	Arbitration Trigger Rate	Early-Stop Trigger Rate	Mean Tokens
Direct	Easy	1.0 $\pm$ 0.0	0.000 $\pm$ 0.000	0.000 $\pm$ 0.000	1.000 $\pm$ 0.000	578 $\pm$ 21
Direct	Medium	1.0 $\pm$ 0.0	0.000 $\pm$ 0.000	0.000 $\pm$ 0.000	1.000 $\pm$ 0.000	623 $\pm$ 51
Direct	Hard	1.0 $\pm$ 0.0	0.000 $\pm$ 0.000	0.000 $\pm$ 0.000	1.000 $\pm$ 0.000	867 $\pm$ 74
Multi-Debate	Easy	3.8 $\pm$ 0.3	0.967 $\pm$ 0.018	0.950 $\pm$ 0.022	0.083 $\pm$ 0.024	2967 $\pm$ 84
Multi-Debate	Medium	3.9 $\pm$ 0.2	0.950 $\pm$ 0.022	0.933 $\pm$ 0.027	0.050 $\pm$ 0.018	3083 $\pm$ 208
Multi-Debate	Hard	4.0 $\pm$ 0.1	0.933 $\pm$ 0.027	0.917 $\pm$ 0.031	0.017 $\pm$ 0.009	4161 $\pm$ 296

Appendix

Table 20: Full-evaluation resource allocation by difficulty stratum. Values report mean

\pm

standard deviation across repeated runs. Verification may precede either terminal route; for ECMR-Full, arbitration and early stopping are complementary shares, and mean rounds increase with difficulty within the six-round budget.

Figure 6: Difficulty-conditioned controller actions. Panel (a) reports the explicit-verification trigger rate for every policy and difficulty stratum. Panel (b) shows ECMR-Full reallocating complementary terminal routes from early stopping on Easy cases to arbitration on Hard cases; whiskers mark routing uncertainty.

Method	Avg. Tokens	Avg. Latency (s)	$\Delta$ Acc vs Weak	Extra tokens / 1pp	Tokens / correct
Strong-Direct	679 $\pm$ 201	4.72 $\pm$ 1.38	11.1	—	3054
Strong-Self-Consistency	2110 $\pm$ 582	16.02 $\pm$ 5.39	5.6	255.0	12658
Weak-Direct	693 $\pm$ 219	3.34 $\pm$ 0.43	0.0	—	6238
Weak-Self-Consistency	2153 $\pm$ 643	12.17 $\pm$ 2.09	0.0	—	19374
Weak-Fixed-ECMR	4044 $\pm$ 727	15.79 $\pm$ 2.22	11.1	301.5	18196
Weak-ECMR	8816 $\pm$ 7368	27.31 $\pm$ 13.33	33.3	243.7	19836

Appendix

Table 21: Cost efficiency across strong and weak backbones. Token cost and latency are reported as mean

\pm

SD across the three domain aggregates. Accuracy gain is measured relative to Weak-Direct; additional tokens per percentage-point (pp) gain and tokens per correct sample are computed from the recorded diagnostic logs.

Removed Feature	Performance Drop	Importance Rank
Evidence independence $I^{ind}_{t}$	6.1 $\pm$ 0.5 pp	1
Support entropy $\widetilde{H}^{sup}_{t}$	5.8 $\pm$ 0.4 pp	2
Contradiction structure $C^{ctr}_{t}$	5.2 $\pm$ 0.4 pp	3
Role drift/overlap	4.8 $\pm$ 0.4 pp	4
Answer concentration $g_{t}$	4.3 $\pm$ 0.4 pp	5
Remaining budget $b_{t}$	3.1 $\pm$ 0.3 pp	6

Appendix

Table 22: State-feature ablation importance over the declared 9,000-record evaluation. Semantic-state features are ranked by recorded accuracy drop in percentage points (pp) and across-run uncertainty when each feature is removed.

Method	Risk-Flagged FCR	Error Propagation Rate
No-Compliance	0.389 $\pm$ 0.115	0.333 $\pm$ 0.111
Compliance-Monitor	0.333 $\pm$ 0.111	0.278 $\pm$ 0.106
Feedback-Only	0.278 $\pm$ 0.106	0.222 $\pm$ 0.098
Partial-Loop	0.222 $\pm$ 0.098	0.167 $\pm$ 0.088
Full-Compliance-Loop	0.167 $\pm$ 0.088	0.111 $\pm$ 0.074

Appendix

Table 23: Compliance gains on the separate 18-record risk-flagged diagnostic. The

\pm

terms are binomial standard errors for false-consensus and error-propagation incidence and do not inherit the main evaluation sample count.

Method	Performance Variance	Collaboration Budget Fluctuation	Compliance Rate
Frozen-Baseline	12.7	28.3%	0.71 $\pm$ 0.04
Frozen+EMA	7.5	18.4%	0.80 $\pm$ 0.03
Frozen+Memory	5.3	13.2%	0.86 $\pm$ 0.03
Full-Online-ECMR	2.4	8.6%	0.92 $\pm$ 0.02

Appendix

Table 24: System stability on the separate 150-record Formal50 streaming diagnostic. Performance variance and collaboration-budget fluctuation are point estimates; compliance-rate uncertainty retains recorded across-run variation.

Multi-Agent Reasoning

Multi-Agent Reasoning studies how multiple AI agents share evidence, model one another, and coordinate decisions on complex tasks.

See all 228 papers

Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judge
Feb 10, 2026Wei Yang, Shixuan Li, Heng Ping +3LLM Auditing Preference Optimization
University of Southern California
Reasoning Consensus: Structural Ensembling of LLM Reasoning via Weighted DAG Aggregation
Jul 30, 2026Amruta Parulekar, Jinu Lee, Dilek Hakkani-Tür +1Structured Reasoning LLM Reasoning with Graphs
University of Illinois Urbana-Champaign
DarkForest: Less Talk, Higher Accuracy for Multi-Agent LLMs
May 24, 2026Yi Li, Songtao Wei, Dongming Jiang +3Multi-Agent LLM Systems Multi-Agent System Optimization
University of Texas at Dallas · Independent Researcher · University of California, Davis

Feb 10, 2026J/K move · Enter open · S save

Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judge

Wei Yang, Shixuan Li, Heng Ping, Peiyu Zhang+2

University of Southern California

LLM Auditing Preference Optimization Multi-Agent Systems Multi-Agent Consensus Multi-Agent Reasoning Reasoning Verification

Open paper AskPDF arXiv

Multi-agent systems (MAS) can substantially extend the reasoning capacity of large language models (LLMs). Most MAS frameworks aggregate agent outputs via simple majority voting, discarding the evidential structure of reasoning traces. Majority voting is brittle under confabulation consensus, where agents share correlated biases and converge on the same incorrect rationale. We introduce AgentAuditor, which moves beyond frequency-based aggregation by organizing agent traces into a Reasoning Tree that explicitly represents agreements and divergences in their reasoning. AgentAuditor resolves conflicts by comparing branch-level evidence at critical divergence points, turning global adjudication into efficient, localized verification. We further propose Anti-Consensus Preference Optimization (ACPO), which trains the adjudicator with evidence-verified preference supervision to reduce conformity to misleading majority cues. Across four MAS frameworks and multiple reasoning benchmarks, AgentAuditor consistently improves aggregation performance over majority voting, with gains of up to 5% absolute accuracy while remaining token-efficient.

Abstract

Multi-Agent Reasoning

Explore similar work

Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judge

Reasoning Consensus: Structural Ensembling of LLM Reasoning via Weighted DAG Aggregation

DarkForest: Less Talk, Higher Accuracy for Multi-Agent LLMs