cs.CLNov 13, 2026

In the Eye of the Beholder: Human Label Variation in Textual and Visual Entailment

Authors: Monica Riedler, Jasmin Orth, Raffaella Bernardi, Barbara Plank

Organizations: MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Free University of Bozen-Bolzano, Italy

Abstract

Human Label Variation (HLV) is increasingly recognised as an informative signal that reflects meaningful differences in how people interpret the same input. While HLV has been studied extensively in Natural Language Inference (NLI), much less is known about HLV in Visual Entailment (VE; Xie et al., 2019), where the textual premise is replaced by an image, keeping the hypothesis and label space fixed. We introduce VariNLI-VE, a dataset for studying how HLV changes between paired NLI and VE instances. We re-annotate a subset of originally neutral SNLI (Bowman et al., 2015) instances and their visual counterparts, with the same annotators evaluating both the textual and visual versions. Each annotation includes a label, confidence score, and explanation, yielding 2,250 annotations. Labels and explanations are collected jointly, providing ecologically valid explanations. Because the image may reveal information not stated in the text, one might expect disagreement to decrease in VE. Instead, we find no clear overall reduction: disagreement decreases for some instances while it increases for others. Explanations reveal further variation, as annotators can retain the same label while changing their rationale. When evaluating vision-language models (VLMs) on the paired data, we find lower accuracy, weaker alignment with human label distributions, and generally higher uncertainty in VE than in NLI.

Figures & tables

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Atomic Visual Entailment: Enhancing Zero-Shot Vision-Language Reasoning through Atomic Fact Decomposition and Learned Selection

    Oct 4, 2026Nallathambi Vethiappan, Derya Soydaner, Gijs WijnholdsZero-Shot LearningVLM Reasoning

  2. It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them

    Sep 29, 2026Nagham Omar, Mahmoud Jabarin, Kinan Ibraheem +1VLM EvaluationFigurative Language Understanding

  3. Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers

    Oct 14, 2026Josh Barua, Kyle Mahowald, William RudmanVision-Language ModelsModality Gap in VLMs