In the Eye of the Beholder: Human Label Variation in Textual and Visual Entailment
Authors: Monica Riedler, Jasmin Orth, Raffaella Bernardi, Barbara Plank
Organizations: MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Free University of Bozen-Bolzano, Italy
Human Label Variation (HLV) is increasingly recognised as an informative signal that reflects meaningful differences in how people interpret the same input. While HLV has been studied extensively in Natural Language Inference (NLI), much less is known about HLV in Visual Entailment (VE; Xie et al., 2019), where the textual premise is replaced by an image, keeping the hypothesis and label space fixed. We introduce VariNLI-VE, a dataset for studying how HLV changes between paired NLI and VE instances. We re-annotate a subset of originally neutral SNLI (Bowman et al., 2015) instances and their visual counterparts, with the same annotators evaluating both the textual and visual versions. Each annotation includes a label, confidence score, and explanation, yielding 2,250 annotations. Labels and explanations are collected jointly, providing ecologically valid explanations. Because the image may reveal information not stated in the text, one might expect disagreement to decrease in VE. Instead, we find no clear overall reduction: disagreement decreases for some instances while it increases for others. Explanations reveal further variation, as annotators can retain the same label while changing their rationale. When evaluating vision-language models (VLMs) on the paired data, we find lower accuracy, weaker alignment with human label distributions, and generally higher uncertainty in VE than in NLI.
Figures & tables
Figure 1: Setup and analyses. For each VariNLI-VE instance, we collect five annotations, including a label (E, N, C), confidence score (1-6), and explanation from the same annotators across NLI and VE tasks, and compare with model predictions and uncertainty estimates.
Figure 2: Label transitions from the original SNLI label to our newly derived NLI and VE majority labels in VariNLI-VE. Ties indicate cases with no majority label among five annotators.
Figure 3: Agreement-pattern transitions NLI ↔ VE.
Metric
Mean ± Std.
95% CI
Entropy (NLI)
0.47±0.47
[0.41, 0.53]
Entropy (VE)
0.54±0.49
[0.48, 0.60]
Δ entropy (VE–NLI)
0.07±0.64
[-0.01, 0.15]
TVD (VE–NLI)
0.48±0.34
[0.44, 0.53]
JSD (VE–NLI)
0.52±0.32
[0.48, 0.57]
Table 1: Human disagreement and NLI–VE distributional shifts with 95% paired bootstrap confidence intervals. Values are mean ± standard deviation.
What changes between NLI and VE?
Operational criterion
n
Disagreement restructuring?
D↓ : Disagreement decreases in VE
HVE<HNLI
63
Yes
D↑ : Disagreement increases in VE
HVE>HNLI
85
Yes
D↔ : Disagreement composition changes
HNLI=HVE>0 and 0<TVD<1
22
Yes
Unanimous label changes
HNLI=HVE=0 and TVD=1
27
No
Label distribution unchanged
HNLI=HVE and TVD=0
28
No
Table 2: Operationalization of disagreement restructuring across paired NLI–VE instances ( n=225 ). We consider disagreement restructured when its amount changes, or when the amount remains the same but the label distributions differ. H denotes human label entropy, while n indicates item count. TVD is total variation distance.
Figure 4: Explanation analysis for instances exhibiting disagreement restructuring. Numbers indicate explanation-pair counts. D↓ : Disagreement decreases in VE; D↑ : Disagreement increases in VE; D↔ : Disagreement composition changes.
Figure 5: Model performance and uncertainty across NLI and VE. The left half shows accuracy with respect to human majority labels across all instances and on the neutral-only paired instances. The right half shows model uncertainty, measured by normalized predictive entropy and LAC conformal prediction set size, with larger LAC sets indicating higher uncertainty. Accuracy decreases from NLI to VE, while uncertainty generally increases.
Pattern
NLI
VE
n (NLI/VE)
5-0-0
5.39±0.49
5.41±0.54
105/92
4-1-0
4.94±0.60
5.00±0.52
68/68
3-2-0
4.67±0.58
4.76±0.44
39/46
3-1-1
4.82±0.41
4.69±0.40
9/11
2-2-1
4.75±0.77
4.62±0.68
4/8
Table 3: Mean confidence by annotation pattern in NLI and VE. Values are mean ± standard deviation on a six-point confidence scale, while n indicates item count.
Model
NLI
VE
VE (by label)
Overall
Ent
Neu
Con
Gemma-3-4B
71.5
45.2
87.7
55.6
3.8
Gemma-3-12B
79.6
54.8
87.7
51.9
34.2
Gemma-3-27B
82.8
55.3
87.7
53.1
34.2
InternVL3.5-2B
81.0
59.4
93.0
53.1
41.8
InternVL3.5-4B
91.4
54.8
89.5
69.1
15.2
Table 4: Model accuracies across modalities and by VE majority label. Cells are shaded according to decrease in accuracy wrt NLI; darker cells indicate larger drops.
Figure 6: Per-model accuracy by annotator agreement.
Model
All instances
Neutral-only instances
TVD NLI ↓
TVD VE ↓
JSD NLI ↓
JSD VE ↓
TVD NLI ↓
TVD VE ↓
JSD NLI ↓
JSD VE ↓
Gemma-3-4B
0.325
0.561
0.367
0.593
0.278
0.497
0.319
0.541
Gemma-3-12B
0.268
0.460
0.309
0.500
0.245
0.481
0.286
0.528
Gemma-3-27B
0.245
0.477
0.289
0.508
0.204
0.492
0.248
0.531
InternVL3.5-2B
0.253
0.392
0.322
0.434
0.255
0.454
0.331
0.500
InternVL3.5-4B
0.185
0.407
0.244
0.443
0.148
0.329
0.211
0.382
Table 5: Alignment between model predictive distributions and human judgment distributions. Lower TVD and JSD indicate closer alignment. VE columns are shaded according to the increase over the corresponding NLI value within each block, with darker cells indicating a larger gap.
Figure 7: Spearman’s ρ between model predictive entropy and human uncertainty measures (human label entropy and inverse mean confidence). Asterisks indicate statistically significant correlations (p < 0.05).
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Distribution of the first languages of the annotators who participated in this study.
Metric
Mean
Std.
Range
AoA
7.71
2.65
3 - 14
Reading Proficiency
5.71
0.46
5 - 6
Writing Proficiency
5.24
0.62
4 - 6
Appendix
Table 6: Annotator metadata: Age of Acquisition (AoA) of English, as well as self-assessed English reading and writing proficiency of the annotators used in this study. We report mean, standard deviation (std) and range. Proficiency was measured on a 6-point scale with 1 indicating minimal and 6 high proficiency.
Figure 9: Annotation guidelines that were presented to annotators before starting their task and available to them throughout the session.
Figure 10: Annotation interface presenting a textual (NLI) instance.
Figure 11: Annotation interface presenting a visual (VE) instance.
Figure 12: Prompt template used for natural language inference.
Figure 13: Prompt template used for visual entailment.
Figure 14: Label shifts from original single e-SNLI-VE labels to VE majority labels from our new annotations.
Metric
5 votes
4 votes
Sensitivity
Entropy (NLI)
0.468
0.430
±0.026
Entropy (VE)
0.538
0.495
±0.026
Δ entropy (VE–NLI)
0.070
0.065
±0.036
TVD (VE–NLI)
0.484
0.487
±0.013
JSD (VE–NLI)
0.524
0.526
±0.013
Appendix
Table 7: Paired leave-one-vote-out stability analysis. The 4-vote values average over all five possible paired removals per instance. Sensitivity reflects variation across these removals.
Metric
All annotators
w/o authors
Entropy (NLI)
0.468 [0.41, 0.53]
0.467 [0.40, 0.53]
Entropy (VE)
0.538 [0.48, 0.60]
0.469 [0.40, 0.54]
Δ entropy (VE–NLI)
0.070 [-0.01, 0.15]
0.002 [-0.08, 0.09]
TVD (VE–NLI)
0.484 [0.44, 0.53]
0.488 [0.44, 0.53]
JSD (VE–NLI)
0.524 [0.48, 0.57]
0.526 [0.48, 0.57]
Appendix
Table 8: Human disagreement metrics with all annotators and after excluding author-annotators. Values are means with 95% percentile bootstrap confidence intervals based on 10,000 resamples of the 225 instances.
Figure 15: Illustrative examples of the four label-rationale patterns used in the explanation analysis. Each panel shows the textual premise, hypothesis, and paired NLI and VE judgments from the same annotator.
Model
Acc. ↑
TVD ↓
JSD ↓
Ent.
LAC
Qwen3-1.7B
42.1
0.556
0.583
0.167
2.77
Qwen3-4B
81.4
0.259
0.304
0.021
1.74
Qwen3-8B
83.7
0.249
0.297
0.056
2.04
Appendix
Table 9: Text-only Qwen3 results on NLI. Ent. denotes normalized predictive entropy, and LAC denotes conformal prediction set size using LAC score.
Model
NLI acc.
VE acc.
Gemma-3-4B
76.2
55.6
Gemma-3-12B
80.0
51.9
Gemma-3-27B
87.5
53.1
InternVL3.5-2B
82.5
53.1
InternVL3.5-4B
92.5
69.1
InternVL3.5-8B
88.8
65.4
Appendix
Table 10: Accuracy on paired instances whose VE majority label is neutral ( n=81 ). VE cells are shaded according to the decrease in accuracy compared to the corresponding NLI instances, with darker cells indicating larger drops.
Figure 16: Predicted vs human label distribution for VE across our 225 instances.
Model
Entropy
LAC
NLI
VE
NLI
VE
Gemma-3-4B
0.045
0.041
1.476
2.627
Gemma-3-12B
0.025
0.105
1.716
2.200
Gemma-3-27B
0.010
0.027
1.409
2.293
InternVL3.5-2B
0.387
0.346
2.080
2.129
InternVL3.5-4B
0.114
0.344
1.098
1.969
Appendix
Table 11: Detailed model uncertainty scores measured with normalized predictive entropy and LAC set size. VE cells are shaded according to their increase over the corresponding NLI value.
Figure 17: Spearman’s ρ between LAC set size and human uncertainty measures (human label entropy and inverse mean confidence). Asterisks indicate p < 0.05. n/a indicates cases where Spearman’s ρ is undefined because LAC set size does not vary across instances.
Model
Accuracy ( Δ=NLI−VE )
Uncertainty ( Δ=VE−NLI )
Overall
Neutral-only
Norm. entropy
LAC size
Gemma-3-4B
26.3 [17.6, 35.0]
20.7 [9.9, 31.7]
−0.004 [ −0.027 , 0.017] n.s.
1.151 [1.036, 1.267]
Gemma-3-12B
24.8 [16.9, 32.5]
28.1 [17.3, 39.5]
0.080 [0.051, 0.109]
0.484 [0.356, 0.613]
Gemma-3-27B
27.5 [19.5, 35.5]
34.4 [23.5, 45.6]
0.016 [0.001, 0.033]
0.884 [0.769, 1.000]
InternVL3.5-2B
21.5 [13.4, 29.5]
29.4 [18.4, 40.7]
−0.041 [ −0.091 , 0.010] n.s.
0.049 [ −0.098 , 0.196] n.s.
InternVL3.5-4B
36.6 [29.2, 43.9]
23.4 [13.5, 33.3]
0.231 [0.184, 0.278]
0.871 [0.756, 0.987]
Appendix
Table 12: Paired NLI–VE differences in model accuracy and uncertainty with 95% bootstrap confidence intervals based on 10,000 paired resamples. Cells report Δ [95% CI]. For accuracy, Δ=NLI−VE and is reported in percentage points. For normalized predictive entropy and LAC set size, Δ=VE−NLI . Non-significant differences (95% CI includes zero) are marked with n.s. and shaded in gray.
Model
All instances
Neutral-only ( n=81 )
TVD
JSD
TVD
JSD
Gemma-3-4B
0.236 [0.170, 0.303]
0.226 [0.163, 0.290]
0.218 [0.136, 0.301]
0.222 [0.139, 0.304]
Gemma-3-12B
0.192 [0.134, 0.249]
0.191 [0.135, 0.246]
0.235 [0.155, 0.318]
0.242 [0.162, 0.323]
Gemma-3-27B
0.231 [0.173, 0.290]
0.219 [0.160, 0.277]
0.288 [0.208, 0.369]
0.283 [0.201, 0.363]
InternVL3.5-2B
0.139 [0.088, 0.190]
0.112 [0.067, 0.158]
0.198 [0.132, 0.266]
0.169 [0.112, 0.227]
InternVL3.5-4B
0.222 [0.171, 0.272]
0.200 [0.151, 0.248]
0.180 [0.113, 0.246]
0.171 [0.104, 0.236]
Appendix
Table 13: Paired NLI–VE differences in human–model distributional alignment with 95% bootstrap confidence intervals based on 10,000 paired resamples. Cells report Δ [95% CI], where Δ=VE−NLI . The neutral-only comparison is restricted to the 81 instances whose VE majority label is neutral.
Figure 18: Three selected samples in both NLI and VE, complete with all five annotations. Each bounding box represents one sample. The left side shows the textual variant and the right side shows the visual one. Below each instance we show the five annotations collected for each sample. Annotations on the same line signify annotations given by the same annotator. Each annotation includes the label, confidence score, and explanation.
Model
TVD ↓
JSD ↓
Ent
Neu
Con
Ent
Neu
Con
Gemma-3-4B
0.228
0.497
0.860
0.290
0.541
0.859
Gemma-3-12B
0.198
0.481
0.613
0.260
0.528
0.632
Gemma-3-27B
0.219
0.492
0.630
0.265
0.531
0.642
InternVL3.5-2B
0.139
0.454
0.506
0.195
0.500
0.536
InternVL3.5-4B
0.161
0.329
0.672
0.222
0.382
0.672
Appendix
Table 14: VE label-wise alignment between model predictive distributions and human judgment distributions. Results are grouped by VE majority label. Lower TVD and JSD indicate closer alignment. Cells are shaded according to the distance value, with darker cells indicating weaker alignment.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.