Natural language explanation generation serves as a key mechanism for exposing and evaluating vision-language reasoning. Prior work on explanation-driven vision-language models predominantly follows a post-hoc (answer-first) paradigm, implicitly suggesting that supervised rationales can reflect underlying reasoning processes. In contrast, modern large vision-language models increasingly exhibit a rationale-first generation tendency, which more closely aligns with structured, stepwise reasoning. In this work, we systematically evaluate whether explanations are causally tied to model predictions within a single generation step under a controlled experimental setup, explicitly eliminating unnecessary chain-of-thought or other intermediate reasoning processes across knowledge-intensive QA, visual entailment, and compositional grounding benchmarks. We find that larger models emerge as a prerequisite for reliably supporting rationale-first reasoning at scale. However, answer-first generation is less prone to format-related errors in structured output. Overall, explanation ordering, model scale and pre-training knowledge, task-specific fine-tuning, and task structure jointly influence both prediction accuracy and reasoning faithfulness.
Figures & tables
Figure 1: Conventional explanation‑driven vision–language reasoning through post‑hoc rationalization. The model generates an answer first and then produces post‑hoc explanations, primarily grounding the justification in visual evidence, often with limited use of world knowledge.
Figure 2: The instruction prompt and output format of each VLR problem remain identical across both settings, and the only difference lies in the ordering of answer and explanation generation.
Subtask
Input
Target
caption_0_vs_images
caption 0 + image 0, image 1
image 0
caption_1_vs_images
caption 1 + image 0, image 1
image 1
image_0_vs_captions
image 0 + caption 0, caption 1
caption 0
image_1_vs_captions
image 1 + caption 0, caption 1
caption 1
Table 1: Winoground subtasks in our experiments. Each original sample is decomposed into four evaluation instances covering caption-to-image and image-to-caption matching.
Dataset
Split
Instances
A-OKVQA
Train
17k
Test
1.1k
e-SNLI-VE
Train
100k
Test
14k
Winoground
Train
800
Test
800
Table 2: Summary of dataset statistics for A-OKVQA, e-SNLI-VE, and Winoground. The number of instances corresponds to the total number of image–question–answer tuples in each dataset split.
Qwen-VL
LLaVA
Difficulty
RF
AF
Consist.
RF
AF
Consist.
zero-shot
Easy (1075)
86.2
87.8
91.5
76.7
76.0
82.2
Hard (70)
80.0
75.7
82.2
65.7
75.7
74.3
fine-tuned
Easy (1075)
86.5
88.0
92.2
78.8
78.6
90.5
Table 3: Answer accuracy (%) under different response modes (RF vs. AF) and consistency score (%) by task difficulty for zero-shot and fine-tuned models on the A-OKVQA validation set. Support counts are shown by difficulty category.
Qwen-VL
LLaVA
Label
RF
AF
Consist.
RF
AF
Consist.
Zero-shot
Entailment (5217)
81.3
81.7
83.9
63.3
67.2
88.3
Neutral (3799)
55.1
54.1
70.3
14.2
11.4
69.0
Contradiction (5720)
79.2
52.1
58.1
62.4
75.6
63.5
fine-tuned
Table 4: Per-label F1 scores (%) and consistency (%) for e-SNLI-VE test set. Support counts are shown per label.
Qwen-VL
LLaVA
Sub-Task
RF
AF
Consist.
RF
AF
Consist.
zero-shot
Caption_0_vs_Images
71.5
42.5
66.5
94.7
95.0
94.6
Caption_1_vs_Images
55.5
82.5
71.5
0.8
0.8
98.7
Image_0_vs_Captions
84.0
76.5
87.5
44.5
90.2
50.1
Image_1_vs_Captions
74.0
87.0
81.5
8.5
0.7
63.2
Table 5: Winoground sub-task accuracy (%) RF and AF response modes, and consistency (%) between RF and AF predictions, for zero-shot and fine-tuned models.
Response Mode
Response
Qwen-VL Zero-shot
Rationale-first
{ "rationale": "The orange vehicles in the image appear to be small utility vehicles typically used by airport ground crews for transporting equipment or personnel. They are not police cars, shuttles for passengers, or part of air traffic control systems.", "answer": "B. shuttle" }
Answer-first
{ "answer": "B. shuttle", "rationale": "The orange vehicles in the image appear to be small shuttles or service vehicles typically used on airport runways for transporting luggage, supplies, or personnel between different areas of the airport." }
Qwen-VL Fine-tuned
Rationale-first
{ "rationale": "The orange vehicles are used to shuttle people around. The orange vehicles are shuttles. The orange vehicles can carry many people quickly.", "answer": "B. shuttle" }
Answer-first
{ "answer": "B. shuttle", "rationale": "The orange vehicles shuttle around the airport. The orange vehicles move quickly to transport people or luggage. The orange vehicles are shuttles." }
Table 6: Response outputs from Qwen-VL and LLaVA across settings and response modes.
Figure 3: Question: What are the orange vehicles for? Options: A. police B. shuttle C. passengers D. air traffic Gold Answer: D. air traffic Rationales: ”There are large planes.”, ”They are there for the pilots to tell them where to go or not go.”, ”They keep an eye on the planes.”