In the Eye of the Beholder: Human Label Variation in Textual and Visual Entailment
Authors: Monica Riedler, Jasmin Orth, Raffaella Bernardi, Barbara Plank
Organizations: MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Free University of Bozen-Bolzano, Italy
Human Label Variation (HLV) is increasingly recognised as an informative signal that reflects meaningful differences in how people interpret the same input. While HLV has been studied extensively in Natural Language Inference (NLI), much less is known about HLV in Visual Entailment (VE; Xie et al., 2019), where the textual premise is replaced by an image, keeping the hypothesis and label space fixed. We introduce VariNLI-VE, a dataset for studying how HLV changes between paired NLI and VE instances. We re-annotate a subset of originally neutral SNLI (Bowman et al., 2015) instances and their visual counterparts, with the same annotators evaluating both the textual and visual versions. Each annotation includes a label, confidence score, and explanation, yielding 2,250 annotations. Labels and explanations are collected jointly, providing ecologically valid explanations. Because the image may reveal information not stated in the text, one might expect disagreement to decrease in VE. Instead, we find no clear overall reduction: disagreement decreases for some instances while it increases for others. Explanations reveal further variation, as annotators can retain the same label while changing their rationale. When evaluating vision-language models (VLMs) on the paired data, we find lower accuracy, weaker alignment with human label distributions, and generally higher uncertainty in VE than in NLI.
Figures & tables
Figure 1: Setup and analyses. For each VariNLI-VE instance, we collect five annotations, including a label (E, N, C), confidence score (1-6), and explanation from the same annotators across NLI and VE tasks, and compare with model predictions and uncertainty estimates.
Figure 2: Label transitions from the original SNLI label to our newly derived NLI and VE majority labels in VariNLI-VE. Ties indicate cases with no majority label among five annotators.
Figure 3: Agreement-pattern transitions NLI ↔ VE.
Metric
Mean ± Std.
95% CI
Entropy (NLI)
0.47±0.47
[0.41, 0.53]
Entropy (VE)
0.54±0.49
[0.48, 0.60]
Δ entropy (VE–NLI)
0.07±0.64
[-0.01, 0.15]
TVD (VE–NLI)
0.48±0.34
[0.44, 0.53]
JSD (VE–NLI)
0.52±0.32
[0.48, 0.57]
Table 1: Human disagreement and NLI–VE distributional shifts with 95% paired bootstrap confidence intervals. Values are mean ± standard deviation.
What changes between NLI and VE?
Operational criterion
n
Disagreement restructuring?
D↓ : Disagreement decreases in VE
HVE<HNLI
63
Yes
D↑ : Disagreement increases in VE
HVE>HNLI
85
Yes
D↔ : Disagreement composition changes
HNLI=HVE>0 and 0<TVD<1
22
Yes
Unanimous label changes
HNLI=HVE=0 and TVD=1
27
No
Label distribution unchanged
HNLI=HVE and TVD=0
28
No
Table 2: Operationalization of disagreement restructuring across paired NLI–VE instances ( n=225 ). We consider disagreement restructured when its amount changes, or when the amount remains the same but the label distributions differ. H denotes human label entropy, while n indicates item count. TVD is total variation distance.
Figure 4: Explanation analysis for instances exhibiting disagreement restructuring. Numbers indicate explanation-pair counts. D↓ : Disagreement decreases in VE; D↑ : Disagreement increases in VE; D↔ : Disagreement composition changes.
Figure 5: Model performance and uncertainty across NLI and VE. The left half shows accuracy with respect to human majority labels across all instances and on the neutral-only paired instances. The right half shows model uncertainty, measured by normalized predictive entropy and LAC conformal prediction set size, with larger LAC sets indicating higher uncertainty. Accuracy decreases from NLI to VE, while uncertainty generally increases.
Pattern
NLI
VE
n (NLI/VE)
5-0-0
5.39±0.49
5.41±0.54
105/92
4-1-0
4.94±0.60
5.00±0.52
68/68
3-2-0
4.67±0.58
4.76±0.44
39/46
3-1-1
4.82±0.41
4.69±0.40
9/11
2-2-1
4.75±0.77
4.62±0.68
4/8
Table 3: Mean confidence by annotation pattern in NLI and VE. Values are mean ± standard deviation on a six-point confidence scale, while n indicates item count.
Model
NLI
VE
VE (by label)
Overall
Ent
Neu
Con
Gemma-3-4B
71.5
45.2
87.7
55.6
3.8
Gemma-3-12B
79.6
54.8
87.7
51.9
34.2
Gemma-3-27B
82.8
55.3
87.7
53.1
34.2
InternVL3.5-2B
81.0
59.4
93.0
53.1
41.8
InternVL3.5-4B
91.4
54.8
89.5
69.1
15.2
Table 4: Model accuracies across modalities and by VE majority label. Cells are shaded according to decrease in accuracy wrt NLI; darker cells indicate larger drops.
Figure 6: Per-model accuracy by annotator agreement.
Model
All instances
Neutral-only instances
TVD NLI ↓
TVD VE ↓
JSD NLI ↓
JSD VE ↓
TVD NLI ↓
TVD VE ↓
JSD NLI ↓
JSD VE ↓
Gemma-3-4B
0.325
0.561
0.367
0.593
0.278
0.497
0.319
0.541
Gemma-3-12B
0.268
0.460
0.309
0.500
0.245
0.481
0.286
0.528
Gemma-3-27B
0.245
0.477
0.289
0.508
0.204
0.492
0.248
0.531
InternVL3.5-2B
0.253
0.392
0.322
0.434
0.255
0.454
0.331
0.500
InternVL3.5-4B
0.185
0.407
0.244
0.443
0.148
0.329
0.211
0.382
Table 5: Alignment between model predictive distributions and human judgment distributions. Lower TVD and JSD indicate closer alignment. VE columns are shaded according to the increase over the corresponding NLI value within each block, with darker cells indicating a larger gap.
Figure 7: Spearman’s ρ between model predictive entropy and human uncertainty measures (human label entropy and inverse mean confidence). Asterisks indicate statistically significant correlations (p < 0.05).
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Distribution of the first languages of the annotators who participated in this study.
Metric
Mean
Std.
Range
AoA
7.71
2.65
3 - 14
Reading Proficiency
5.71
0.46
5 - 6
Writing Proficiency
5.24
0.62
4 - 6
Appendix
Table 6: Annotator metadata: Age of Acquisition (AoA) of English, as well as self-assessed English reading and writing proficiency of the annotators used in this study. We report mean, standard deviation (std) and range. Proficiency was measured on a 6-point scale with 1 indicating minimal and 6 high proficiency.
Figure 9: Annotation guidelines that were presented to annotators before starting their task and available to them throughout the session.
Figure 10: Annotation interface presenting a textual (NLI) instance.
Figure 11: Annotation interface presenting a visual (VE) instance.
Figure 12: Prompt template used for natural language inference.
Figure 13: Prompt template used for visual entailment.
Figure 14: Label shifts from original single e-SNLI-VE labels to VE majority labels from our new annotations.
Metric
5 votes
4 votes
Sensitivity
Entropy (NLI)
0.468
0.430
±0.026
Entropy (VE)
0.538
0.495
±0.026
Δ entropy (VE–NLI)
0.070
0.065
±0.036
TVD (VE–NLI)
0.484
0.487
±0.013
JSD (VE–NLI)
0.524
0.526
±0.013
Appendix
Table 7: Paired leave-one-vote-out stability analysis. The 4-vote values average over all five possible paired removals per instance. Sensitivity reflects variation across these removals.
Metric
All annotators
w/o authors
Entropy (NLI)
0.468 [0.41, 0.53]
0.467 [0.40, 0.53]
Entropy (VE)
0.538 [0.48, 0.60]
0.469 [0.40, 0.54]
Δ entropy (VE–NLI)
0.070 [-0.01, 0.15]
0.002 [-0.08, 0.09]
TVD (VE–NLI)
0.484 [0.44, 0.53]
0.488 [0.44, 0.53]
JSD (VE–NLI)
0.524 [0.48, 0.57]
0.526 [0.48, 0.57]
Appendix
Table 8: Human disagreement metrics with all annotators and after excluding author-annotators. Values are means with 95% percentile bootstrap confidence intervals based on 10,000 resamples of the 225 instances.
Figure 15: Illustrative examples of the four label-rationale patterns used in the explanation analysis. Each panel shows the textual premise, hypothesis, and paired NLI and VE judgments from the same annotator.
Model
Acc. ↑
TVD ↓
JSD ↓
Ent.
LAC
Qwen3-1.7B
42.1
0.556
0.583
0.167
2.77
Qwen3-4B
81.4
0.259
0.304
0.021
1.74
Qwen3-8B
83.7
0.249
0.297
0.056
2.04
Appendix
Table 9: Text-only Qwen3 results on NLI. Ent. denotes normalized predictive entropy, and LAC denotes conformal prediction set size using LAC score.
Model
NLI acc.
VE acc.
Gemma-3-4B
76.2
55.6
Gemma-3-12B
80.0
51.9
Gemma-3-27B
87.5
53.1
InternVL3.5-2B
82.5
53.1
InternVL3.5-4B
92.5
69.1
InternVL3.5-8B
88.8
65.4
Appendix
Table 10: Accuracy on paired instances whose VE majority label is neutral ( n=81 ). VE cells are shaded according to the decrease in accuracy compared to the corresponding NLI instances, with darker cells indicating larger drops.
Figure 16: Predicted vs human label distribution for VE across our 225 instances.
Model
Entropy
LAC
NLI
VE
NLI
VE
Gemma-3-4B
0.045
0.041
1.476
2.627
Gemma-3-12B
0.025
0.105
1.716
2.200
Gemma-3-27B
0.010
0.027
1.409
2.293
InternVL3.5-2B
0.387
0.346
2.080
2.129
InternVL3.5-4B
0.114
0.344
1.098
1.969
Appendix
Table 11: Detailed model uncertainty scores measured with normalized predictive entropy and LAC set size. VE cells are shaded according to their increase over the corresponding NLI value.
Figure 17: Spearman’s ρ between LAC set size and human uncertainty measures (human label entropy and inverse mean confidence). Asterisks indicate p < 0.05. n/a indicates cases where Spearman’s ρ is undefined because LAC set size does not vary across instances.
Model
Accuracy ( Δ=NLI−VE )
Uncertainty ( Δ=VE−NLI )
Overall
Neutral-only
Norm. entropy
LAC size
Gemma-3-4B
26.3 [17.6, 35.0]
20.7 [9.9, 31.7]
−0.004 [ −0.027 , 0.017] n.s.
1.151 [1.036, 1.267]
Gemma-3-12B
24.8 [16.9, 32.5]
28.1 [17.3, 39.5]
0.080 [0.051, 0.109]
0.484 [0.356, 0.613]
Gemma-3-27B
27.5 [19.5, 35.5]
34.4 [23.5, 45.6]
0.016 [0.001, 0.033]
0.884 [0.769, 1.000]
InternVL3.5-2B
21.5 [13.4, 29.5]
29.4 [18.4, 40.7]
−0.041 [ −0.091 , 0.010] n.s.
0.049 [ −0.098 , 0.196] n.s.
InternVL3.5-4B
36.6 [29.2, 43.9]
23.4 [13.5, 33.3]
0.231 [0.184, 0.278]
0.871 [0.756, 0.987]
Appendix
Table 12: Paired NLI–VE differences in model accuracy and uncertainty with 95% bootstrap confidence intervals based on 10,000 paired resamples. Cells report Δ [95% CI]. For accuracy, Δ=NLI−VE and is reported in percentage points. For normalized predictive entropy and LAC set size, Δ=VE−NLI . Non-significant differences (95% CI includes zero) are marked with n.s. and shaded in gray.
Model
All instances
Neutral-only ( n=81 )
TVD
JSD
TVD
JSD
Gemma-3-4B
0.236 [0.170, 0.303]
0.226 [0.163, 0.290]
0.218 [0.136, 0.301]
0.222 [0.139, 0.304]
Gemma-3-12B
0.192 [0.134, 0.249]
0.191 [0.135, 0.246]
0.235 [0.155, 0.318]
0.242 [0.162, 0.323]
Gemma-3-27B
0.231 [0.173, 0.290]
0.219 [0.160, 0.277]
0.288 [0.208, 0.369]
0.283 [0.201, 0.363]
InternVL3.5-2B
0.139 [0.088, 0.190]
0.112 [0.067, 0.158]
0.198 [0.132, 0.266]
0.169 [0.112, 0.227]
InternVL3.5-4B
0.222 [0.171, 0.272]
0.200 [0.151, 0.248]
0.180 [0.113, 0.246]
0.171 [0.104, 0.236]
Appendix
Table 13: Paired NLI–VE differences in human–model distributional alignment with 95% bootstrap confidence intervals based on 10,000 paired resamples. Cells report Δ [95% CI], where Δ=VE−NLI . The neutral-only comparison is restricted to the 81 instances whose VE majority label is neutral.
Figure 18: Three selected samples in both NLI and VE, complete with all five annotations. Each bounding box represents one sample. The left side shows the textual variant and the right side shows the visual one. Below each instance we show the five annotations collected for each sample. Annotations on the same line signify annotations given by the same annotator. Each annotation includes the label, confidence score, and explanation.
Model
TVD ↓
JSD ↓
Ent
Neu
Con
Ent
Neu
Con
Gemma-3-4B
0.228
0.497
0.860
0.290
0.541
0.859
Gemma-3-12B
0.198
0.481
0.613
0.260
0.528
0.632
Gemma-3-27B
0.219
0.492
0.630
0.265
0.531
0.642
InternVL3.5-2B
0.139
0.454
0.506
0.195
0.500
0.536
InternVL3.5-4B
0.161
0.329
0.672
0.222
0.382
0.672
Appendix
Table 14: VE label-wise alignment between model predictive distributions and human judgment distributions. Results are grouped by VE majority label. Lower TVD and JSD indicate closer alignment. Cells are shaded according to the distance value, with darker cells indicating weaker alignment.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Visual entailment (VE) asks whether an image supports, contradicts, or leaves undecided a textual hypothesis. Strong results come from fine-tuning large vision-language models on labelled data, while zero-shot and hybrid approaches remain far behind. A VE hypothesis often bundles several visual claims, yet existing zero-shot methods reason over it as a single unit. We propose Atomic Visual Entailment (AVE), which decomposes the hypothesis into atomic facts, produces candidate predictions from both the full hypothesis and its facts using frozen vision-language models, and predicts the final label with a lightweight classifier trained only on how those candidates behave. We find that decomposition helps only when the hypothesis context is preserved: judging facts in isolation is worse than not decomposing at all. Full-hypothesis and atomic prediction make complementary errors, and learning which to trust recovers far more of that complementarity than majority voting, reaching 0.803 test accuracy on SNLI-VE without fine-tuning any vision-language model. AVE also localises the visual evidence behind its prediction without region-level supervision. These results suggest that learning which candidate prediction to trust can close much of the gap to fine-tuned systems, offering a practical alternative where fine-tuning a vision-language model directly would need more labelled data or compute than is available.
Vision-language models (VLMs) are increasingly used in place of human annotators, making it important that substitutability tests reflect the model rather than incidental evaluation conditions. We introduce MIST, the Misleading-Image Stress Test: 200 English sentences, each built around a phrase readable either figuratively or literally and shown with an aligned image depicting its reading, a misleading image depicting the opposite, or no image at all. The guidelines require the label to be decided from the sentence alone, so no image should change any answer. We expected each image to pull a judge's labels toward the sense it depicts, and neither kind did. Across thirteen VLM judges, an aligned image changed 20.5% of labels and a misleading one 19.4%, close for every judge and both above the 11.6% produced by deleting the ignore-the-image instruction with the image left in place. Yet only 37% of the labels that differ between the two images moved toward the sense shown, and agreement with our human annotators is unchanged whether the image is absent, aligned or misleading. The effect is smaller in the seven judges that pass the alt-test than in the six that never do, but present in all of them: what moves a judge is that an image is there, not which of the two it is, so a substitutability verdict describes a configuration as much as a model.
Vision-language models must reconcile conflicting text and images. Prior work has characterized the resulting behavior in terms of modality biases--e.g., a "blind faith in text"--but why preferences vary across models, domains, and input conditions remains unclear. We develop a predictive account from a Bayesian perspective, positing that VLMs combine conditionally independent evidence from each modality with prior beliefs, subject to modality-specific biases. The resulting two-parameter cue combination model accurately predicts VLM preferences under unseen combinations of input degradation and transfers to new forms of unreliability without refitting. Separating evidence strength from integration bias explains how VLMs can follow text even while weighting visual evidence more heavily, reconciling reports of text and vision dominance under a single combination rule.