In many queries, the context information provided to generative foundation models, such as LLMs and VLMs, is inherently uncertain, including its representations of the physical world. While conventional prompting discards the associated uncertainty, this paper, for the first time in the literature, systematically demonstrates that communicating such uncertainty in context provides useful additional cues for downstream decisions and yields improvements across a wide range of settings. We formulate the decision-making problem as a zero-shot foundation model-assisted image classification task supported by auxiliary text-based characteristics. We choose this due to the availability of well-defined auxiliary properties, associated uncertainty, and unambiguous task correctness. Using five downstream foundation models across three datasets, we extensively evaluate uncertainty arising from both humans and learned models when observing or predicting auxiliary evidence. Across all settings, we find that communicating uncertainty substantially improves downstream LLM classification performance: macro-F1 improves up to 26.18 percentage points compared to when the uncertainty information is not communicated. We further observe how the foundation models process such uncertainty information and find that auxiliary uncertainty contains predictive information about the correctness of downstream decisions. Together, these results establish uncertainty as useful contextual information and motivate more systematic consideration of uncertainty communication in future foundation-model inference pipelines.
Figures & tables
Figure 1: Overview of our evaluation framework. Upstream sources provide auxiliary evidence and associated uncertainty. We compare paired zero-shot contexts while holding the auxiliary evidence fixed and varying whether its uncertainty is communicated, and study the resulting effects on downstream prediction, evidence use, and decision correctness.
Uncertainty Communication
Downstream Analysis
Scope
Method
Communicates Uncertainty
Holds Evidence Fixed
Measures Effect of Uncertainty Communication
Analyzes Downstream Correctness
Analyzes Downstream Uncertainty
Studies Distinct Uncertainty Source Types
SRICE ( 2025a )
✓
✗
✓
✗
✗
✗
VideoAgent2 ( 2025b )
✓
✗
✓
✗
✗
✗
Tools in the Loop ( 2025b ; 2025a )
✗
✗
✗
✓
✗
✗
ProbCBM ( 2023 )
✗
✗
✗
✓
✓
✗
Our study
✓
✓
✓
✓
✓
✓
Table 1: Comparison with relevant prior works. A check denotes that the corresponding criterion is explicitly evaluated. “Evidence held fixed” means that auxiliary evidence is unchanged and only the communicated uncertainty varies. “Distinct uncertainty source types” refers to qualitatively different signals, such as human-reported certainty, model matching scores, and classifier confidence.
Dataset
Model
Aux
Aux + Uncertainty
Δ (pp)
CUB (7-class)
Qwen3-32B
0.4682±0.0165
0.5424±0.0131
+7.42±0.76
Llama-3.1-8B
0.5770±0.0095
0.6368±0.0083
+5.98±0.94
Phi-4-14B
0.6924±0.0174
0.7385±0.0194
+4.61±0.84
CUB (8-class)
Qwen3-32B
0.3602±0.0039
0.4387±0.0156
+7.84±1.72
Llama-3.1-8B
0.5260±0.0041
0.5753±0.0109
+4.93±1.10
Phi-4-14B
0.5827±0.0154
0.6083±0.0051
+2.56±1.06
Table 2: Macro-F1 across CUB, SUN, and Derm7pt. Δ denotes the Aux+Unc gain over Aux in percentage points. All experiments use three seeds (42, 96, 1001); CUB reports mean ± standard deviation across seed-specific sampled evaluation sets, while SUN and Derm7pt use fixed evaluation sets and omit ±0.0 when deterministic inference yields identical results.
Model
Classes
Image-only
Image+Aux
Image+Aux+Unc
Δ
InternVL3-8B
7
0.7873±0.0025
0.7963±0.0027
0.7993±0.0036
+0.30
Qwen2-VL-7B-Instruct
7
0.8022±0.0036
0.8189±0.0041
0.8183±0.0042
−0.06
InternVL3-8B
8
0.7154±0.0019
0.7288±0.0017
0.7313±0.0055
+0.25
Qwen2-VL-7B-Instruct
8
0.7782±0.0046
0.7735±0.0056
0.7942±0.0067
+2.07
InternVL3-8B
9
0.7370±0.0069
0.7632±0.0056
0.7682±0.0062
+0.50
Qwen2-VL-7B-Instruct
9
0.6862±0.0011
0.7048±0.0060
0.7124±0.0093
+0.76
Table 3: CUB VLM macro-F1 across 7-, 8-, and 9-class settings. Δ denotes the Image+Aux+Unc gain over Image+Aux in percentage points.
Figure 2: Effect of uncertainty communication across Derm7pt evidence compositions for LLMs Boxes show the distribution of Δ macro-F1 (pp) across subsets of size k ; black lines and diamonds denote the median and mean, respectively.
K:5→8
K:8→10
K:3→10
Model
Aux
Aux+Unc
Aux
Aux+Unc
Aux
Aux+Unc
Qwen3-32B
6
2
7
4
7
2
Llama-3.1-8B
5
1
5
0
5
2
Phi-4-14B
2
0
4
2
6
3
Total
13
3
16
6
18
7
Table 4: Number of predictions that are correct at the smaller top- K setting but become incorrect after lower-ranked auxiliary attributes are added on SUN. Communicating the corresponding BLIP scores substantially reduces these harmful transitions.
Model
No-Uncertainty
True-Uncertainty
Shuffled-Uncertainty
ΔTrue-No (pp)
ΔTrue-Shuffled (pp)
Qwen3-32B
0.3902±0.0638
0.6562±0.0414
0.4609±0.0105
+26.60±3.12
+19.53±3.12
Llama-3.1-8B-Instruct
0.5630±0.0270
0.6800±0.0399
0.5398±0.0386
+11.71±5.13
+14.02±6.08
Phi-4-14B
0.3846±0.0304
0.6439±0.0760
0.4694±0.0274
+25.93±8.36
+17.44±5.03
Table 5: SUN uncertainty-alignment control at K=10 . The same ten BLIP-selected attributes are presented in a score-independent random order under all conditions. No-U omits BLIP scores; True-U attaches each attribute’s correct score; and Shuffled-U preserves the same score multiset but reassigns scores to different attributes. Values are macro-F1 (mean ± standard deviation across three random-order seeds); Δ is reported in percentage points.
Figure 3: Supervised feature relevance versus the best-performing Aux+Unc evidence subsets on Derm7pt. Bars show Random-Forest permutation importance ( Breiman, 2001 ) ; filled cells indicate findings selected in each LLM’s best subset.
Dataset
Model
AUROC
CUB
Phi-4-14B
0.558±0.056
Qwen2-VL-7B-Instruct
0.472±0.084
SUN
Qwen3-32B
0.8002
Phi-4-14B
0.8150
Table 6: Downstream correctness prediction across CUB (from human-reported attribute certainty) and SUN (from four BLIP-derived reliability features, K=5 , using a stratified five-fold logistic-regression probe). Values report pooled out-of-fold AUROC.
Model
Upstream Features
AUROC
Llama-3.1-8B-Instruct
Confidence
0.6259
Entropy
0.6257
Full
0.6480
Qwen2-VL-7B-Instruct
Confidence
0.6388
Entropy
0.6377
Full
0.6358
Table 7: Predicting Derm7pt downstream decision correctness from upstream classifier uncertainty at k=7 . For all models, AUROC is obtained using a standardized logistic-regression probe with the indicated uncertainty features under stratified five-fold cross-validation. We report pooled out-of-fold AUROC.
Modality
Model
Correctness transition
HWithoutUnc
HWithUnc
ΔH
ΔH<0 (%)
LLM
Llama-3.1-8B-Instruct
Incorrect → Correct
0.6736
0.6861
+0.0125
48.3
Correct → Incorrect
0.6836
0.6850
+0.0015
52.4
Correct → Correct
0.4498
0.4737
+0.0239
34.6
Incorrect → Incorrect
0.5883
0.6065
+0.0182
39.3
VLM
Qwen2-VL-7B-Instruct
Incorrect → Correct
0.6315
0.6773
+0.0457
58.3
Correct → Incorrect
0.6885
0.6796
-0.0089
40.0
Table 8: Changes in downstream predictive entropy on Derm7pt at k=7 after communicating upstream classifier confidence. ΔH=HWithUnc−HWithoutUnc ; negative values indicate reduced predictive uncertainty. The final column reports the percentage of samples with ΔH<0 .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Candidate species added
3-class
Chuck-will’s-widow (22), Green Jay (75), White Pelican (101)
5-class
+ Whip-poor-will (105), Hooded Warbler (167)
7-class
+ Ivory Gull (63), Painted Bunting (16)
8-class
+ Nighthawk (92)
9-class
+ Herring Gull (62)
Appendix
Table 9: Nested candidate-species configurations used for CUB. Each larger configuration retains all species from the preceding setting.
Finding
Not suspicious
Suspicious
Pigment network
absent; typical
atypical
Blue-whitish veil
absent
present
Vascular structures
absent; arborizing; comma; hairpin; within regression; wreath
dotted; linear irregular
Pigmentation
absent; diffuse regular; localized regular
diffuse irregular; localized irregular
Streaks
absent; regular
irregular
Dots and globules
absent; regular
irregular
Appendix
Table 10: Binarisation of the original Derm7pt checklist annotations used to train the seven auxiliary finding predictors.
Auxiliary Finding
Macro-F1
F1 Not-Susp.
F1 Suspicious
Accuracy
Blue-whitish veil
0.7078
0.8528
0.5628
0.7797
Regression structures
0.6720
0.8148
0.5291
0.7342
Dots and globules
0.6063
0.5674
0.6452
0.6101
Pigmentation
0.5949
0.7231
0.4667
0.6354
Vascular structures
0.5852
0.9370
0.2333
0.8835
Pigment network
0.5786
0.8218
0.3353
0.7190
Appendix
Table 11: Held-out test performance of the seven Derm7pt auxiliary ResNet classifiers. Each model predicts whether the corresponding dermoscopic finding is not_susp or suspicious . All results are computed on the same 395 clinic images from the fixed test split. Macro-F1 is the primary metric; class-wise F1 and accuracy are also reported.
Feature / representation
Definition / interpretation
CUB: Human-reported certainty
frac_guessing
Fraction of retained attributes annotated as guessing .
frac_probably
Fraction of retained attributes annotated as probably .
frac_definitely
Fraction of retained attributes annotated as definitely .
mean_certainty
Mean ordinal certainty using the original certainty IDs 2 , 3 , and 4 .
certainty_entropy
Normalized entropy of the three certainty-level fractions: −∑c=24fclog(fc)/log3 .
Appendix
Table 12: Upstream features used by the downstream correctness probes. Features are constructed solely from the auxiliary evidence and its associated certainty or confidence signals, independently of the downstream model’s predictions.
Model
Top- K
Aux
Aux + Uncertainty
Δ (pp)
Qwen3-32B
3
0.6320
0.7028
+7.09
5
0.7181
0.7398
+2.17
8
0.6728
0.8690
+19.62
10
0.6066
0.8684
+26.18
Llama-3.1-8B
3
0.8881
0.9244
+3.63
5
0.8892
0.8874
-0.18
Appendix
Table 13: SUN macro-F1 across top- K auxiliary-attribute settings. Δ denotes the Aux+Unc gain over Aux in percentage points.
Model
k
# subsets
Aux
Aux+Unc
Gold Binary
Gold Original
Δ (pp)
Qwen3-32B
1
7
0.567 ± 0.040
0.567 ± 0.049
0.655 ± 0.063
0.636 ± 0.106
+0.00
Qwen3-32B
3
35
0.573 ± 0.043
0.595 ± 0.042
0.681 ± 0.062
0.715 ± 0.029
+2.20
Qwen3-32B
5
21
0.581 ± 0.029
0.612 ± 0.012
0.731 ± 0.023
0.758 ± 0.028
+3.14
Qwen3-32B
7
1
0.505
0.596
0.669
0.750
+9.10
Llama-3.1-8B
1
7
0.496 ± 0.086
0.498 ± 0.096
0.550 ± 0.138
0.542 ± 0.141
+0.12
Llama-3.1-8B
3
35
0.557 ± 0.047
0.577 ± 0.050
0.641 ± 0.066
0.659 ± 0.094
+2.05
Appendix
Table 14: Derm7pt classification performance across evidence subset sizes. Values are the mean ± standard deviation of macro-F1 across distinct subsets; for k=7 , the single-subset result is reported without deviation. Δ denotes Aux+Unc minus Aux in percentage points. ‘Gold Original’ refers to the original ground-truth auxiliary findings. ‘Gold Binary’ refers to the binarized version of the gold auxiliary findings.
Model
k
Image-only
Image+Aux
Image+Aux+Unc
Gold Binary
Gold Original
Δ (pp)
Qwen2-VL-7B-Instruct
1
0.2649
0.5904
0.5673
0.6655
0.6535
−2.31
3
0.5505
0.5496
0.6710
0.7017
−0.09
5
0.5422
0.5483
0.6662
0.7273
+0.61
7
0.5264
0.5442
0.6271
0.7306
+1.78
InternVL3-8B
1
0.4604
0.4808
0.5301
0.5213
0.5946
+4.93
3
0.5210
0.5674
0.6315
0.6994
+4.64
Appendix
Table 15: Derm7pt VLM performance across auxiliary-evidence subset sizes. All conditions include the original clinical image. Image+Aux provides predicted checklist findings, while Image+Aux+Unc provides the same findings together with their corresponding upstream classifier confidence. Gold Binary and Gold Original replace the predicted findings with their ground-truth binarized and original categorical annotations, respectively. Results are macro-F1 on the fixed 395-image test set. Δ denotes the paired change from Image+Aux to Image+Aux+Unc in percentage points.
Figure 4: BLIP match score as a function of attribute rank across the 80 selected SUN images. For each image, the 20 candidate attributes are ranked by BLIP match score; points show the mean score at each rank and error bars denote one standard deviation. Scores decrease with rank, confirming that increasing top- K progressively introduces attributes that BLIP itself scores as less strongly matched to the image.
Figure 5: Derm7pt LLM Prompt. This is an illustrative example with only one auxiliary evidence and its associated classifier confidence. The prompt contains additional background information obtained from ( Argenziano et al., 1998 ) .
Figure 21
Figure 6: CUB LLM Prompt. This is an illustrative example with 7-classes bird species classification task.
Figure 7: SUN scene classification VLM prompt. This is an illustrative example with three auxiliary evidences and their associated uncalibrated BLIP image–text matching scores.
Figure 8: Derm7pt VLM Prompt. This is an illustrative example with all the seven auxiliary evidence with their associated classifier confidence.
Model
Prompt
Aux
Full
Conf.
Ent.
Qwen3-32B
P1
0.5053
0.6093
0.6132
0.5807
P2
0.5442
0.5775
0.5775
0.5213
P3
0.5053
0.6067
0.5963
0.5496
Llama-3.1-8B
P1
0.6051
0.5915
0.5772
0.5859
P2
0.5083
0.5106
0.5083
0.5111
P3
0.6011
0.6194
0.6301
0.5982
Appendix
Table 16: Sensitivity of Derm7pt classification to the output instruction formulation. All experiments use the fixed 395-case test set and all seven ResNet-predicted findings ( k=7 ). Results are reported in macro-F1. Aux provides predicted finding labels only; Full additionally provides both softmax probabilities and entropy; Conf. provides maximum softmax confidence; and Ent. provides predictive entropy.
Qwen2-VL-7B
InternVL3-8B
Condition
Original
P3
Original
P3
Image-only
0.2649
0.2270
0.4604
0.3671
Aux
0.5264
0.4251
0.4739
0.5104
Aux+Unc
0.5442
0.5183
0.4913
0.4916
Gold Binary
0.6271
0.5595
0.6377
0.6479
Gold Original
0.7306
0.6982
0.7220
0.7612
Appendix
Table 17: Prompt sensitivity on the 395 held-out Derm7pt clinic images. Macro-F1 is reported for the original VLM prompt and the P3-style prompt adapted from Figure 5 . All evidence based conditions use all seven findings ( k=7 ); Image-only uses no auxiliary findings ( k=0 ). The final row reports the macro-F1 change from Aux to Aux+Unc in percentage points.
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China