CUSP: Decomposable Collective Uncertainty for Multi-Agent Multimodal Reasoning

Chung-En Johnny Yu; David Garcia; Brian Jalaian; Nathaniel D. Bastian

cs.AISep 4, 2026

CUSP: Decomposable Collective Uncertainty for Multi-Agent Multimodal Reasoning

Authors: Chung-En Johnny Yu, David Garcia, Brian Jalaian, Nathaniel D. Bastian

Organizations: University of West Florida Pensacola, Florida, USA · United States Military Academy West Point, New York, USA

Abstract

Aggregating heterogeneous vision-language models (VLMs) can improve multimodal reasoning, but neither an individual model's confidence nor that of the aggregated answer measures reliability at the system level. We present CUSP (Collective Uncertainty through Semantic Opinion Pooling), a training-free uncertainty quantification framework that maps multiple VLM responses to a shared semantic response space, pools them into a pooled semantic opinion, and reports two complementary system-level signals: collective uncertainty, the dispersion of the pooled opinion, and Jensen-Shannon divergence (JSD), the conflict among the model-level opinions. Within this pooled semantic opinion, the unnormalized collective entropy decomposes exactly into the mean of the models' individual semantic entropies and the JSD, separating total dispersion from model conflict. Requiring neither token logits nor calibration labels, CUSP applies to open-weight and commercial VLMs alike. In static multi-VLM ensembles, collective uncertainty is the strongest signal in the small-model regime (0.764 AUROC for prediction-error detection, 0.889 AUARC for abstention), outperforming uncertainty baselines majority voting and naive selection by 4.7 to 15.8 points and widening its margin as the ensemble grows; JSD is strongest in the evaluated commercial regime (0.819 AUROC, 0.910 AUARC) and ranks hard-answer model conflict with AUROC up to 0.982. The pooled prediction also improves accuracy over the average single model by 5.6 to 13.0 points. Over the full trajectory of a multi-step, multi-agent system, subagent collective uncertainty ranks system failures above chance (0.619 AUROC) and gives the best abstention ordering among the evaluated signals (0.699 AUARC).

Figures & tables

Figure 1: Overview of CUSP. Sampled VLM responses are mapped into a shared semantic response space, pooled into a semantic opinion, and summarized by collective uncertainty and JSD.

Panel of Figure 2 — Figure 2: Static prediction-error detection (AUROC) and abstention (AUARC) across model regimes. Bars show combination-level means over 56 small, 20 large, and 20 commercial three-VLM systems; dashed segments show the mean single-VLM reference. Higher values are better.

Regime	Avg. single VLM	CUSP	Majority voting	Naive selection
Small	66.28%	79.29%	71.86%	78.54%
Large	88.42%	93.98%	91.51%	89.14%
Commercial	71.00%	79.70%	73.98%	71.60%

Table 1: Benchmark accuracy over all systems in each model regime; bold marks the best value per row.

Model regime	Collective uncertainty	JSD
Small	0.872	0.916
Large	0.962	0.982
Commercial	0.879	0.874

Table 2: Hard-answer model-conflict discrimination across model regimes. Values report Conflict-AUROC; higher is better; bold marks the best value per row.

Signal	Mean		Max
Signal	AUROC	AUARC	AUROC	AUARC
Collective uncertainty	0.608	0.694	0.619	0.699
JSD	0.603	0.690	0.599	0.689
MeanSE	0.601	0.688	0.590	0.684
MaxSE	0.530	0.668	0.489	0.652
VoteEntropy	0.610	0.668	0.606	0.666

Table 3: Full-trajectory prediction-error detection (AUROC) and abstention ranking (AUARC) in the multi-agent system. Scores summarize the three evidence-acquisition rounds using temporal mean or maximum aggregation; bold and underline mark the best and second-best values per column.

$K$ -VLM	$L_{\mathrm{sec}}\downarrow$	Additional aggregation overhead $\Delta L_{\mu\mathrm{s}}\downarrow$
$K$ -VLM	$L_{\mathrm{sec}}\downarrow$	CUSP	Majority Voting	Naive Selection
2	2.04	2.56	1.94	1.84
3	3.07	3.06	2.22	2.48
4	4.11	3.56	2.50	3.11
5	5.15	4.03	2.74	3.72
6	6.19	4.49	2.97	4.34

Table S.1: Mean p50 latency for small-model systems.

L_{\mathrm{sec}}

is VLM UQ sampling latency in seconds per sample, and

\Delta L_{\mu\mathrm{s}}

is additional aggregation overhead in microseconds; lower is faster; bold marks the best value per row.

Signal	R0	R1	R2	Avg.
Collective uncertainty	0.634	0.598	0.551	0.594
JSD	0.626	0.576	0.536	0.580
MeanSE	0.598	0.601	0.552	0.584
MaxSE	0.647	0.592	0.480	0.573
VoteEntropy	0.610	0.529	0.518	0.552
VoteConflict	0.608	0.537	0.524	0.557

Table S.2: Round-wise prediction-error detection (Round-AUROC) for the non-interventional probe answer (

n=100

). The average column is computed from full-precision R0–R2 values; bold and underline mark the best and second-best values per column.

Round	CUSP-aggregated	Non-aggregated	Gain
R0	64.0%	49.0%	$+15.0$
R1	61.0%	60.0%	$+1.0$
R2	63.0%	60.0%	$+3.0$

Table S.3: Round-wise probe-answer accuracy with CUSP-aggregated and non-aggregated evidence. Accuracy is computed over all 100 examples per round, and gains are percentage points.

Panel of Figure 9 — Figure S.1: Small-model scaling for prediction-error detection (AUROC) and abstention ranking (AUARC). Each point macro-averages all systems of size $K$ .

Regime	Metric	Weighted	Uniform
Small	AUROC	0.755	0.764
Small	AUARC	0.882	0.889
Large	AUROC	0.928	0.906
Large	AUARC	0.989	0.987
Commercial	AUROC	0.800	0.793
Commercial	AUARC	0.906	0.904

Table S.4: Static pooling-weight ablation for three-model systems. Values compare uncertainty-weighted collective uncertainty with CUSP’s uniform collective uncertainty for prediction-error detection and abstention ranking. Bold marks the best values per row.

$K$ -VLM	AUROC		AUARC
$K$ -VLM	Weighted	Uniform	Weighted	Uniform
2	0.783	0.757	0.844	0.839
3	0.755	0.764	0.882	0.889
4	0.740	0.778	0.898	0.910
5	0.723	0.789	0.907	0.928
6	0.715	0.807	0.914	0.940

Table S.5: Pooling-weight ablation across small-model system sizes. Values report AUROC and AUARC for uncertainty-weighted versus uniform collective uncertainty. Bold marks the best values per row in the same metric.

Regime	Weighted	Uniform
Small	78.01%	79.29%
Large	94.20%	93.98%
Commercial	79.68%	79.70%

Table S.6: Pooling-weight ablation for combination-level answer accuracy. Values are percentages averaged over all non-singleton systems in each regime.

$K$ -VLM	Weighted	Uniform
2	4.18	2.56
3	5.29	3.06
4	6.43	3.56
5	7.59	4.03
6	8.69	4.49
7	9.68	4.95

Table S.7: Aggregation-overhead ablation for uncertainty-weighted and uniform pooling. Values report additional overhead

\Delta L_{\mu\mathrm{s}}

above the same VLM UQ sampling latency; lower is faster. Bold marks the best values per row.

Temporal	Metric	Weighted	Uniform
Mean	Traj-AUROC	0.599	0.608
Mean	Traj-AUARC	0.688	0.694
Max	Traj-AUROC	0.600	0.619
Max	Traj-AUARC	0.688	0.699

Table S.8: Multi-agent trajectory ablation for uncertainty-weighted and uniform pooling. Values report prediction-error detection (Traj-AUROC) and abstention ranking (Traj-AUARC) under temporal mean and maximum aggregation. Bold marks the best values per row.

Regime	Models
Small open-source models	ministral-3:3b , qwen3.5:2b , granite3.2-vision:2b , qwen3-vl:2b , llava-phi3:3.8b , moondream:1.8b , gemma3:4b , qwen2.5vl:3b
Large open-source models	gemma4:31b , nemotron3:33b , qwen3.5:35b , qwen3.6:35b , gemma3:27b , llava:34b
Commercial models	anthropic/claude-sonnet-5 , google/gemini-3.1-pro-preview , minimax/minimax-m3 , moonshot/kimi-k2.6 , openai/gpt-5.6-terra , xai/grok-4.5

Table S.9: Static multi-VLM model identifiers by regime. Identifiers match the final analysis configurations used for the reported results.

Panel of Figure 16 — Figure S.2: Centralized multi-agent evidence-acquisition architecture for Section 4.2 of the main paper. CUSP is applied to the VLM subagents’ UQ samples at each evidence-acquisition round; the final-answer orchestrator is not sampled for uncertainty. Full prompts appear later in this supplement.

Language Model Ensembles

Language model ensembles combine predictions or reasoning from multiple models to improve accuracy, reliability, coverage, or efficiency.

See all 134 papers

Mar 25, 2026

SCoOP: Semantic Consistent Opinion Pooling for Uncertainty Quantification in Multiple Vision-Language Model Systems

Combining multiple Vision-Language Models (VLMs) can enhance multimodal reasoning and robustness, but aggregating heterogeneous models' outputs amplifies uncertainty and increases the risk of hallucinations. We propose SCoOP (Semantic-Consistent Opinion Pooling), a training-free uncertainty quantification (UQ) framework for multi-VLM systems through uncertainty-weighted linear opinion pooling. The core idea is to treat each VLM as a probabilistic "expert," sample multiple outputs, map them to a unified space, aggregate their opinions, and produce a system-level uncertainty score. Unlike prior UQ methods designed for single models, SCoOP explicitly measures collective, system-level uncertainty across multiple VLMs, enabling effective hallucination detection and abstention for highly uncertain samples. On ScienceQA, SCoOP achieves an AUROC of 0.866 for hallucination detection, outperforming baselines (0.732-0.757) by approximately 10-13%. For abstention, it attains an AURAC of 0.907, exceeding baselines (0.818-0.840) by 7-9%. Despite these gains, SCoOP introduces only microsecond-level aggregation overhead relative to the baselines, which is trivial compared to typical VLM inference time (on the order of seconds). These results demonstrate that SCoOP provides an efficient and principled mechanism for uncertainty-aware aggregation, advancing the reliability of multimodal AI systems. Our code is publicly available at https://github.com/chungenyu6/SCoOP.

Chung-En Johnny Yu, Brian Jalaian, Nathaniel D. Bastian

University of West Florida Pensacola, Florida, USA · United States Military Academy West Point, New York, USA

Hallucination Detection in VLMs

Ask PDF

Mar 21, 2026

DiscoUQ: Structured Disagreement Analysis for Uncertainty Quantification in LLM Agent Ensembles

Multi-agent LLM systems, where multiple prompted instances of a language model independently answer questions, are increasingly used for complex reasoning tasks. However, existing methods for quantifying the uncertainty of their collective outputs rely on shallow voting statistics that discard the rich semantic information in agents' reasoning. We introduce DiscoUQ, a framework that extracts and leverages the structure of inter-agent disagreement -- both linguistic properties (evidence overlap, argument strength, divergence depth) and embedding geometry (cluster distances, dispersion, cohesion) -- to produce well-calibrated confidence estimates. We propose three methods of increasing complexity: DiscoUQ-LLM (logistic regression on LLM-extracted structure features), DiscoUQ-Embed (logistic regression on embedding geometry), and DiscoUQ-Learn (a neural network combining all features). Evaluated on four diverse benchmarks (StrategyQA, MMLU, TruthfulQA, ARC-Challenge) with a 5-agent system using Qwen3.5-27B, DiscoUQ-LLM achieves an average AUROC of 0.802, outperforming the best baseline (LLM Aggregator, 0.791) while being substantially better calibrated (ECE 0.036 vs. 0.098). The learned features generalize across benchmarks with near-zero performance degradation and provide the largest improvements where they are most needed: in the ambiguous "weak disagreement" tier where simple vote counting fails.

Bo Jiang

Temple University

Uncertainty Quantification LLM Uncertainty Estimation Language Model Ensembles Language Model Calibration

Ask PDF

Mar 30, 2026

CoE: Collaborative Entropy for Uncertainty Quantification in Agentic Multi-LLM Systems

Uncertainty estimation in multi-LLM systems remains largely single-model-centric: existing methods quantify uncertainty within each model but do not adequately capture semantic disagreement across models. To address this gap, we propose Collaborative Entropy (CoE), a unified information-theoretic metric for semantic uncertainty in multi-LLM collaboration. CoE is defined on a shared semantic cluster space and combines two components: intra-model semantic entropy and inter-model divergence to the ensemble mean. CoE is not a weighted ensemble predictor; it is a system-level uncertainty measure that characterizes collaborative confidence and disagreement. We analyze several core properties of CoE, including non-negativity, zero-value certainty under perfect semantic consensus, and the behavior of CoE when individual models collapse to delta distributions. These results clarify when reducing per-model uncertainty is sufficient and when residual inter-model disagreement remains. We also present a simple CoE-guided, training-free post-hoc coordination heuristic as a practical application of the metric. Experiments on TriviaQA and SQuAD with LLaMA-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, and Mistral-7B-Instruct show that CoE provides stronger uncertainty estimation than standard entropy- and divergence-based baselines, with gains becoming larger as additional heterogeneous models are introduced. Overall, CoE offers a useful uncertainty-aware perspective on multi-LLM collaboration.

Kangkang Sun, Jun Wu, Jianhua Li +3

Shanghai Key Laboratory of Integrated Administration Technologies for Information Security School of Computer Science Shanghai Jiao Tong University Shanghai, China · School of Science and Engineering The Chinese University of Hong Kong, Shenzhen Shenzhen, China

LLM Uncertainty Estimation Multi-LLM Collaboration Semantic Entropy

Ask PDF

Abstract

Language Model Ensembles

Explore similar work

SCoOP: Semantic Consistent Opinion Pooling for Uncertainty Quantification in Multiple Vision-Language Model Systems

DiscoUQ: Structured Disagreement Analysis for Uncertainty Quantification in LLM Agent Ensembles

CoE: Collaborative Entropy for Uncertainty Quantification in Agentic Multi-LLM Systems