cs.LGNov 18, 2026

Tell Me How Sure You Are: Communicating Uncertainty in Auxiliary Evidence to Foundation Models

Authors: Soumyajit Chatterjee, Baturalp Buyukates, Abhirup Ghosh

Organizations: Brave & University of Birmingham, United Kingdom · University of Birmingham, United Kingdom

Abstract

In many queries, the context information provided to generative foundation models, such as LLMs and VLMs, is inherently uncertain, including its representations of the physical world. While conventional prompting discards the associated uncertainty, this paper, for the first time in the literature, systematically demonstrates that communicating such uncertainty in context provides useful additional cues for downstream decisions and yields improvements across a wide range of settings. We formulate the decision-making problem as a zero-shot foundation model-assisted image classification task supported by auxiliary text-based characteristics. We choose this due to the availability of well-defined auxiliary properties, associated uncertainty, and unambiguous task correctness. Using five downstream foundation models across three datasets, we extensively evaluate uncertainty arising from both humans and learned models when observing or predicting auxiliary evidence. Across all settings, we find that communicating uncertainty substantially improves downstream LLM classification performance: macro-F1 improves up to 26.1826.18 percentage points compared to when the uncertainty information is not communicated. We further observe how the foundation models process such uncertainty information and find that auxiliary uncertainty contains predictive information about the correctness of downstream decisions. Together, these results establish uncertainty as useful contextual information and motivate more systematic consideration of uncertainty communication in future foundation-model inference pipelines.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty

    Apr 19, 2026Jingyi Ren, Ante Wang, Yunghwei Lai +5LLM EvaluationSelective Prediction

  2. The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models

    Mar 17, 2026Robert Welch, Emir Konuk, Kevin SmithConfidence Estimation for VLMsVLM Reasoning