CoE: Collaborative Entropy for Uncertainty Quantification in Agentic Multi-LLM Systems
Authors: Kangkang Sun, Jun Wu, Jianhua Li, Minyi Guo, Xiuzhen Che, Jianwei Huang
Organizations: Shanghai Key Laboratory of Integrated Administration Technologies for Information Security School of Computer Science Shanghai Jiao Tong University Shanghai, China · School of Science and Engineering The Chinese University of Hong Kong, Shenzhen Shenzhen, China
Uncertainty estimation in multi-LLM systems remains largely single-model-centric: existing methods quantify uncertainty within each model but do not adequately capture semantic disagreement across models. To address this gap, we propose Collaborative Entropy (CoE), a unified information-theoretic metric for semantic uncertainty in multi-LLM collaboration. CoE is defined on a shared semantic cluster space and combines two components: intra-model semantic entropy and inter-model divergence to the ensemble mean. CoE is not a weighted ensemble predictor; it is a system-level uncertainty measure that characterizes collaborative confidence and disagreement. We analyze several core properties of CoE, including non-negativity, zero-value certainty under perfect semantic consensus, and the behavior of CoE when individual models collapse to delta distributions. These results clarify when reducing per-model uncertainty is sufficient and when residual inter-model disagreement remains. We also present a simple CoE-guided, training-free post-hoc coordination heuristic as a practical application of the metric. Experiments on TriviaQA and SQuAD with LLaMA-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, and Mistral-7B-Instruct show that CoE provides stronger uncertainty estimation than standard entropy- and divergence-based baselines, with gains becoming larger as additional heterogeneous models are introduced. Overall, CoE offers a useful uncertainty-aware perspective on multi-LLM collaboration.
Figures & tables
Figure 1: Overview of the proposed Collaborative Entropy (CoE) framework for agentic multi-LLM uncertainty quantification. Each LLM generates candidate responses that are clustered semantically to form probability distributions, which are then aggregated to minimize system-level collaborative entropy.
Figure 2: Visualization of the four CoE quadrants using Dirichlet distributions over three semantic clusters. Each simplex (colour intensity = probability density) and accompanying bar chart illustrates a characteristic combination of UA (horizontal axis) and UE (vertical axis).
Uncertainty Metric
Two LLMs (Llama + Qwen)
Three LLMs (Llama + Qwen + Mistral)
Rejection Accuracy
AURAC
AUROC
Rejection Accuracy
AURAC
AUROC
80%
90%
95%
100%
80%
90%
95%
100%
Semantic Entropy
0.625
0.583
0.579
0.575
0.176
0.670
0.394
0.416
0.468
0.375
0.123
0.687
Regular Entropy
0.562
0.583
0.605
0.575
0.175
0.475
0.342
0.333
0.343
0.375
0.103
0.416
Pfalse
0.575
0.575
0.575
0.575
0.172
0.586
0.375
0.405
0.405
0.375
0.117
0.540
UE
0.625
0.583
0.589
0.575
0.177
0.661
0.368
0.388
0.437
0.375
0.116
0.716
Table 1: Selective prediction performance of uncertainty metrics in multi-LLM collaborative settings on TriviaQA (200 samples). For Semantic Entropy, Regular Entropy, and Pfalse , scores are reported as the arithmetic mean over constituent models. UA and UE are ablation baselines isolating each component of CoE. Bold denotes the best AUROC in each configuration.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
LLMs
TriviaQA Dataset
SQuAD Dataset
Train Accuracy
Validation Accuracy
Train Accuracy
Validation Accuracy
Llama-3.1-8b-instruct
0.65
0.675
0.225
0.1
Qwen-2.5-7b-instruct
0.6
0.55
0.2
0.15
Mistral-7b-instruct
0.375
0.375
0.05
0.075
Appendix
Table 2: The performance of different single LLMs on different dataset (200 samples)
Figure 3: Performance comparison of CoE with different divergence metrics on TriviaQA dataset (200 samples, KL divergence).
Figure 4: Performance comparison of CoE with different divergence metrics on SQuAD dataset (200 samples, KL divergence).
Figure 5: Robustness of the CoE-reduction algorithm to the convergence threshold ϵ (Llama + Qwen ensemble on TriviaQA , 200 shots).
Figure 6: Analysis of CoE-Reduction optimization algorithm under different sampling temperatures and generation counts (Llama + Qwen on TriviaQA dataset with 200 in-context examples).
Figure 7: Accuracy gain and residual CoE after applying the CoE-guided coordination heuristic under four divergence measures (Llama + Qwen on TriviaQA, 200 samples). Initial CoE (light bars) and optimised CoE (dark bars) are shown on the left axis; ensemble accuracy gain is annotated on each bar.
LLMs' overconfidence, particularly when hallucinating, poses a significant challenge for the deployment of the models in safety-critical settings and makes a reliable estimation of uncertainty necessary. Existing approaches for uncertainty quantification typically prioritize lexical or probabilistic measures; however, these techniques often ignore the semantic variance of different responses with similar meaning. In this paper, we propose Adaptive Conformal Semantic Entropy (ACSE), a method for estimating prompt-level uncertainty by adaptively measuring semantic dispersion in LLMs outputs. Our uncertainty scoring function is based on clustering semantic entropy of multiple diverse responses to the same prompt. The function adaptively adjusts the uncertainty score based on semantic features of each cluster. To ensure statistical reliability of our score, we use conformal calibration to apply a decision rule to accept/abstain the prompts, providing a finite-sample, distribution-free guarantee such that the error rate among the accepted responses remains bounded by a user-specified tolerance. Our extensive experimental evaluations using different LLMs and datasets, demonstrate that our approach consistently outperforms state-of-the-art uncertainty quantification baselines using discriminative performance, conformal guarantees, and probabilistic calibration indicators. As a highlight, for TriviaQA dataset, AUROC of our approach is 0.88 compared to 0.65 produced by the token entropy approach.
Hamed Karimi, Vaishali Meyappan, Reza Samavi
Toronto Metropolitan University, Toronto, Ontario, Canada · Vector Institute, Toronto, Ontario, Canada
As large language model (LLM)-based multi-agent systems become increasingly capable, coordinating agents under uncertainty becomes a fundamental challenge. Existing orchestration strategies typically rely on fixed interaction patterns and often lack mechanisms for assessing the reliability of intermediate reasoning steps, allowing errors and hallucinations to propagate through the system. This paper introduces a semantic-uncertainty-guided orchestration approach, HASSUM as a general framework for uncertainty-aware coordination in multi-agent systems. The method estimates uncertainty using semantic entropy and semantic density, which measure trust at the level of answer semantics rather than output probabilities. These signals enable adaptive orchestration decisions, including output verification, selective reprompting, additional deliberation, and confidence-aware response selection. Because the approach operates independently of any particular agent architecture, it can be integrated into a broad range of hierarchical and collaborative multi-agent systems. The evaluations demonstrate an implementation within a hierarchical agent framework and evaluate it on StrategyQA, JailbreakBench, and TruthfulQA benchmarks. Across tasks that require complex reasoning and are prone to ambiguity or hallucinations, uncertainty-guided orchestration yields more reliable outcomes than uncertainty-unaware coordination. Semantic entropy and semantic density in tandem outperformed either metric alone. Ablations testing different thresholds and model sizes demonstrated that both influence the effectiveness of semantic metrics. The results suggest that semantic uncertainty is a practical and general-purpose signal for improving robustness and trustworthiness in agentic AI systems.
John Knowlton, Aritra Guha, Risto Miikkulainen
The University of Texas at Austin, USA · AT&T Chief Data Office, USA · Cognizant AI Lab, USA
Uncertainty estimation has been a long-standing challenge in AI models; it amounts to "knowing what you don't know," and metacognition is notoriously difficult even for humans (cf. the Dunning-Kruger effect). Although it is still far from solved even in simpler classification systems, tackling it in multimodal large language models (MLLMs) is becoming increasingly important. Within MLLMs, uncertainty can stem from any of the diverse sources as well as from their relationships, and further can stem from the unbounded answers in the open-ended setting. To tackle the issues, we propose CoMet, an MLLM uncertainty estimation method by decomposing uncertainty into a context-specific term and a multiplicity-specific term. The former captures ambiguity induced by the given context (e.g., task or prompt), while the latter captures how many plausible answers determined by the context remain compatible with the given input. We train a lightweight post-hoc uncertainty module to estimate these quantities, which enables efficient uncertainty estimation without autoregressive answer generation or repeated sampling. Experiments on various open-ended multimodal benchmarks, hallucination detection, and multiple-choice visual question answering benchmarks show that CoMet consistently improves uncertainty estimation over existing baselines while remaining efficient in practice. Code is available at https://github.com/princetonvisualai/comet_uncertainty