DiscoUQ: Structured Disagreement Analysis for Uncertainty Quantification in LLM Agent Ensembles

Bo Jiang

cs.CLMar 21, 2026

DiscoUQ: Structured Disagreement Analysis for Uncertainty Quantification in LLM Agent Ensembles

Authors: Bo Jiang

Organizations: Temple University

Abstract

Multi-agent LLM systems, where multiple prompted instances of a language model independently answer questions, are increasingly used for complex reasoning tasks. However, existing methods for quantifying the uncertainty of their collective outputs rely on shallow voting statistics that discard the rich semantic information in agents' reasoning. We introduce DiscoUQ, a framework that extracts and leverages the structure of inter-agent disagreement -- both linguistic properties (evidence overlap, argument strength, divergence depth) and embedding geometry (cluster distances, dispersion, cohesion) -- to produce well-calibrated confidence estimates. We propose three methods of increasing complexity: DiscoUQ-LLM (logistic regression on LLM-extracted structure features), DiscoUQ-Embed (logistic regression on embedding geometry), and DiscoUQ-Learn (a neural network combining all features). Evaluated on four diverse benchmarks (StrategyQA, MMLU, TruthfulQA, ARC-Challenge) with a 5-agent system using Qwen3.5-27B, DiscoUQ-LLM achieves an average AUROC of 0.802, outperforming the best baseline (LLM Aggregator, 0.791) while being substantially better calibrated (ECE 0.036 vs. 0.098). The learned features generalize across benchmarks with near-zero performance degradation and provide the largest improvements where they are most needed: in the ambiguous "weak disagreement" tier where simple vote counting fails.

Figures & tables

	StrategyQA		MMLU		TruthfulQA		ARC-Chal.		Average
Method	AUC	ECE	AUC	ECE	AUC	ECE	AUC	ECE	AUC	ECE
B1: Vote Count	.647	.095	.841	.114	.798	.037	.796	.089	.770	.084
B2: Vote Entropy	.647	.338	.837	.205	.798	.096	.794	.185	.769	.206
B3: Verb. Conf.	.681	.205	.720	.143	.733	.122	.760	.042	.724	.128
B4: Self-Consist.	.647	.338	.837	.205	.798	.096	.794	.185	.769	.206
B5: Embed Centr.	.657	.244	.845	.186	.807	.155	.778	.056	.772	.160

Table 1: AUROC (higher is better) and ECE (lower is better) across four benchmarks. Bold : best per column. DiscoUQ -LLM achieves the highest average AUROC (0.802) and the lowest average ECE (0.036). Bootstrap 95% CIs and significance tests are in Appendix A .

Figure 1: AUROC by disagreement tier across benchmarks. DiscoUQ methods show the largest improvements in the strong and weak tiers where vote counting provides the least information.

Figure 2: Expected calibration error (ECE) across benchmarks. Lower is better. All DiscoUQ methods achieve substantially lower ECE than baselines.

Figure 3: Accuracy-coverage curves across benchmarks. DiscoUQ methods maintain higher accuracy at all coverage levels, particularly on StrategyQA and TruthfulQA.

	Str.	MMLU	TQA	ARC	Avg.
M1: DiscoUQ -LLM
Cross-BM	.679	.870	.856	.801	.802
Same-BM	.706	.870	.867	.766	.802
$\Delta$	-.027	+.001	-.011	+.035	-.001
M3: DiscoUQ -Learn
Cross-BM	.696	.892	.870	.839	.824

Table 2: Cross-benchmark generalization (AUROC). Cross-BM: trained on 3 benchmarks, tested on the held-out one. Same-BM: within-benchmark 5-fold CV.

\Delta=\text{Cross}-\text{Same}

Configuration	AUROC
M1: LLM Features
Full (9 features)	.706
$-$ majority confidence lang.	.683 ( $-$ .023)
$-$ reasoning complexity	.699 ( $-$ .007)
$-$ vote confidence	.699 ( $-$ .007)
$-$ divergence depth (late)	.702 ( $-$ .004)

Table 3: Ablation on StrategyQA. For M1, majority confidence language is the most important individual feature. For M2, PCA variance ratio contributes the most beyond vote confidence.

Figure 4: Cost (extra LLM calls) vs. performance (average AUROC) for all methods. M1 offers the best cost-performance tradeoff.

Method	Extra calls	Tok/q	AUROC
B1: Vote Count	0	0	.770
M2: DiscoUQ -Embed	0	0	.773
M1: DiscoUQ -LLM	1,624	2,799	.802
B6: LLM Agg.	2,750	1,387	.791
B3: Verb. Conf.	13,750	623	.724
M3: DiscoUQ -Learn	15,374	3,422	.790

Table 4: Computational cost vs. AUROC. M1 achieves the best AUROC with only 1,624 extra calls (structure analysis for non-unanimous questions only). M2 requires zero extra LLM calls.

	StrategyQA		TruthfulQA
	Homo	Hetero	Homo	Hetero
Disagree. rate	53.0%	45.6%	54.3%	49.2%
B1: Vote Count	.647	.675	.798	.881
M1: DiscoUQ -LLM	.706	.687	.867	.904

Table 5: Homogeneous vs. heterogeneous agent teams. Heterogeneous teams show lower disagreement rates but higher B1 performance, suggesting that cross-model disagreement is more informative.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Bench.	Method	AUROC	ECE	Brier	AUPRC	Cov@90	Cov@95	AUACC
StrategyQA	B1: Vote Count	.647 [.603,.693]	.095	.187	.818	.016	.001	.813
	B2: Vote Entropy	.647 [.603,.693]	.338	.319	.818	.016	.001	.813
	B3: Verb. Conf.	.681 [.638,.724]	.205	.224	.852	.339	.009	.849
	B4: Self-Consist.	.647 [.603,.693]	.338	.319	.818	.016	.001	.813
	B5: Embed Centr.	.657 [.610,.704]	.244	.244	.830	.003	.003	.823
	B6: LLM Agg.	.670 [.622,.718]	.186	.194	.852	.000	.000	.836

Appendix

Table 6: Complete results with bootstrap 95% CIs for AUROC. Bold : best per benchmark-metric. DUQ = DiscoUQ.

Figure 5: Feature importance from ablation studies. Majority confidence language and reasoning complexity are the most important LLM structure features.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 649 papers

Aug 12, 2026

Beyond Single-Turn Confidence: Trajectory-Adapted Uncertainty Quantification for LLM Agents

Uncertainty quantification (UQ) methods for language models are typically evaluated on single-turn outputs, where uncertainty is attached to one generated answer. For LLM agents, however, the unit of observation is an interactive trajectory, where the model can ask clarifying questions, call tools, update state, and make intermediate decisions whose errors propagate to the final outcome. We study whether three common families of single-turn UQ methods transfer to this setting. Across five LLMs and four multi-turn tool-use datasets from BFCL-v4 and

τ^2

-bench, we evaluate white-box scorers based on action-token probabilities, black-box consistency scorers based on resampled trajectories, and reflexive scorers based on model self-assessment of the trajectory. We find that transfer is often useful but uneven. Token-probability scores are highly sensitive to the choice of aggregator used across turns, reflexive scores provide the strongest low-cost baseline in most evaluated settings, and black-box self-consistency is often the strongest UQ family, with trajectory-equivalence and action-set consistency typically ranking highest among its variants. These results suggest that UQ methods developed for single generations should be revalidated at the trajectory level, with careful attention to the consistency measurement, aggregator choice, and computational budget.

Dylan Bouchard, Mohit Singh Chauhan

Confidence Estimation in Language Models LLM Agent Evaluation LLM Uncertainty Estimation Tool-Using Language Agents

Ask PDF

Jun 9, 2026

Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate

Evaluating reasoning quality in multi-agent LLM systems is challenging, especially for open-ended tasks without reference answers. We investigate whether intrinsic confidence signals, token-level log-probabilities from decoding, can predict reasoning quality as assessed by LLM-as-judge evaluation. Using a debate-based essay scoring framework, we compare confidence proxies against rubric-based judge scores across two ASAP essay sets. We find that early-token confidence, particularly within the first few generated tokens, is consistently the strongest predictor of reasoning quality, outperforming full-sequence statistics. Analysis of log-probability trajectories shows that the opening phase of generation is the most heterogeneous and therefore most informative. We also observe a systematic asymmetry between agent roles, with stronger alignment between confidence and quality for supportive reasoning than for adversarial critique. These results suggest that early decoding dynamics provide a lightweight and effective signal for estimating reasoning reliability in multi-agent LLM systems.

Ali Keramati, Justin Cheok, Jacob Horne +1

University of California, Irvine

LLM Agent Evaluation Multi-Agent Debate LLM Uncertainty Estimation

Ask PDF

Feb 12, 2026

When Agents Disagree With Themselves: Behavioral Consistency as an Uncertainty Signal for LLM Agents

Running the same LLM agent on identical inputs yields 2.3-4.2 distinct action sequences per 10 runs; this behavioral variance constitutes a training-free, black-box uncertainty signal that instantiates selective classification and distribution-free calibration for agentic systems. Across 8,000 runs of four models on 200 HotpotQA questions, consistent tasks (at most 2 unique paths) achieve 82-87% accuracy while inconsistent tasks (4 or more paths) achieve 41-65%, a gap that survives controls for task difficulty. Divergence concentrates at step 2 (50.5% of Llama tasks), and consistency metrics detect failures with AUROC 0.62-0.78. Exploiting this signal, selective prediction (answering only when k=3 runs agree) achieves 87-88% accuracy at 54-62% coverage, a 6-14pp gain over single-run baselines, and matches a split-conformal baseline without a held-out calibration set. A cross-benchmark validation on SWE-bench (50 tasks, 1,000 runs) preserves the consistency hierarchy while revealing an ~8x spread in mean trajectory length across models, and bootstrap analysis shows single-run evaluations misrank models 29.3% of the time.

Aman Mehta

Snowflake AI Research

AI Agent Reliability Selective Prediction LLM Agent Evaluation Reasoning Consistency in Language Models Language Model Calibration LLM Reliability+3

Ask PDF

Abstract

Uncertainty Quantification

Explore similar work

Beyond Single-Turn Confidence: Trajectory-Adapted Uncertainty Quantification for LLM Agents

Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate

When Agents Disagree With Themselves: Behavioral Consistency as an Uncertainty Signal for LLM Agents