CCRV-Bench: Constraint-Based Evaluation of Causal Reasoning in Vision-Language Models

Linyuan Gao; Yuan Wu; Yi Chang

cs.CVSep 25, 2026

CCRV-Bench: Constraint-Based Evaluation of Causal Reasoning in Vision-Language Models

Authors: Linyuan Gao, Yuan Wu, Yi Chang

Organizations: School of Artificial Intelligence, Jilin University Changchun 130012, China · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University Changchun 130012, China · International Center of Future Science, Jilin University Changchun 130012, China

Abstract

Vision-language models (VLMs) have demonstrated excellent performance in visual tasks, but their visual causal reasoning capabilities still lack reliable evaluation. Existing evaluations struggle to distinguish whether a model is performing causal reasoning based on visual evidence or relying on statistical correlations for shortcut learning, thereby potentially overestimating their actual capabilities. This paper proposes CCRV-Bench, a constraint-driven visual causal reasoning benchmark for single-image physical scenarios. We construct an orthogonal framework that evaluates four causal task dimensions: causal relation discovery, state prediction, causal diagnosis, and intervention-outcome prediction. We further introduce entity symbolization, spatial grounding, the factual adversarial constraint, and minimalist output constraints to reduce shortcut cues while preserving the physical commonsense required by the task. Experiments across 14 multimodal models show that constraint sensitivity is task- and model-dependent: intervention-outcome prediction has the largest average effective degradation among the four causal tasks, spatial grounding is the most damaging constraint on average, and the factual adversarial constraint improves DCR for all evaluated models. These results show that unconstrained performance does not determine constrained robustness and that a single aggregate score can obscure distinct failures in causal identification, spatial grounding, and constraint-compliant expression. CCRV-Bench provides a standardized framework for diagnosing image-grounded causal reasoning under controlled constraints. The code is available at https://github.com/0815linyuan/CCRVBench

Figures & tables

Benchmark	Causal Capability				Constraint			Input	Process
Benchmark	Und.	Pred.	Attr.	Interv.	Ground.	Factual Adversarial Constraint	Shortcut	Modality	Proc. Eval
CELLO Chen et al. (2024)	✓	✓	✗	✓	✗	✗	✗	Image	✗
CausalVLBench Komanduri et al. (2025)	✗	✓	✓	✓	✗	✗	✗	Image	✗
CausalVQA Foss et al. (2025)	✓	✓	✓	✓	✗	✗	✗	video	✗
GQA Hudson and Manning (2019)	✗	✗	✓	✗	✗	✗	✗	Image	✗
CFBench Zhang et al. (2025)	✗	✗	✗	✗	✗	✗	✓	Text	✗

Table 1: Comparison of CCRV-Bench and related benchmarks across causal tasks, constraint mechanisms, input modalities, and evaluation granularity. “Und.”, “Pred.”, “Attr.”, and “Interv.” denote causal structure understanding, prediction, attribution, and intervention-outcome prediction, respectively. “Ground.” indicates explicit spatial grounding requirements; “Factual Adversarial Constraint” indicates verification of false textual premises against visual evidence; and “Shortcut” indicates controls targeting shortcut cues. “Modality” specifies the input modality, while “Proc. Eval” indicates evaluation of intermediate reasoning components beyond the final answer. Checkmarks indicate explicitly evaluated capabilities or implemented mechanisms.

Figure 1: Overview of CCRV-Bench. The top panel presents the data construction pipeline, yielding 800 baseline QA pairs and 3,200 constrained variants. The lower-left panel organizes four causal tasks and four constraint mechanisms into 16 evaluation settings. The right panels show model response generation, evaluation, and the reported metrics.

Evaluated Models	$\text{DCR}_{\text{baseline}}$				$\text{DCR}_{\text{constraint}}$				CSR	eDCR	CDI	eCDI
Evaluated Models	L1	L2	L3	DCR	L1	L2	L3	DCR	CSR	eDCR	CDI	eCDI
GPT-5.5	92.6	89.3	86.1	89.3	86.9	79.2	70.8	79.0	95.0	76.9	10.4	12.4
Claude-Opus-4.7	95.8	91.0	85.3	90.7	84.7	76.8	70.2	77.2	93.0	74.5	13.5	16.1
Doubao-Seed-2-Pro	95.4	91.9	82.3	89.8	70.2	61.6	55.8	62.6	91.9	60.0	27.3	29.8
Gemini-3.1-Pro-Preview	93.9	88.3	79.6	87.3	71.9	59.6	48.5	60.0	92.4	58.6	27.3	28.6
Gemini-3.5-Flash	94.9	90.5	84.0	89.8	69.2	59.8	51.3	60.1	86.3	56.2	29.7	33.6

Table 2: Main evaluation results of 14 VLMs on CCRV-Bench. L1–L3 report cascaded pass rates, and DCR is their mean. DCR, CSR, and eDCR are reported as percentages; CDI and eCDI are reported in percentage points. The best and second-best scores are highlighted in bold and underlined, respectively.

Evaluated Models	$\text{DCR}_{\text{baseline}}$				$\text{DCR}_{\text{constraint}}$				eCDI
	$X_{1}$	$X_{2}$	$X_{3}$	$X_{4}$	$X_{1}$	$X_{2}$	$X_{3}$	$X_{4}$	$X_{1}$	$X_{2}$	$X_{3}$	$X_{4}$
GPT-5.5	82.3	92.7	88.2	94.2	82.3	78.4	78.6	76.6	1.8	16.5	11.8	19.5
Claude-Opus-4.7	89.7	91.0	89.3	92.7	77.0	76.8	76.9	78.1	15.0	17.0	15.8	16.7
Doubao-Seed-2-Pro	91.5	90.0	87.3	90.5	67.1	61.3	63.8	58.0	27.3	30.8	27.4	33.7
Gemini-3.1-Pro-Preview	85.7	85.8	88.0	89.5	65.4	56.6	60.4	57.5	22.0	29.8	29.8	33.0
Gemini-3.5-Flash	86.0	88.5	91.7	93.0	64.3	58.3	60.9	56.9	25.6	34.5	34.4	40.1

Table 3: Detailed cognitive performance breakdown across four causal task dimensions (

X_{1}

: Discovery,

X_{2}

: Prediction,

X_{3}

: Diagnosis,

X_{4}

: Intervention-Outcome Prediction). DCR values are percentages and eCDI values are percentage-point drops (the internal differences are multiplied by 100). Best results are bolded , and second-best are underlined .

Evaluated Models	$\text{DCR}_{\text{baseline}}$	$\text{DCR}_{\text{constraint}}$				eCDI
		$Y_{1}$	$Y_{2}$	$Y_{3}$	$Y_{4}$	$Y_{1}$	$Y_{2}$	$Y_{3}$	$Y_{4}$
GPT-5.5	89.3	77.0	75.5	97.7	65.7	12.5	21.5	-8.4	23.9
Claude-Opus-4.7	90.7	75.5	74.8	95.2	63.3	17.2	23.8	-4.2	27.8
Doubao-Seed-2-Pro	89.8	72.4	32.4	92.3	53.1	22.7	62.2	-2.5	36.7
Gemini-3.1-Pro-Preview	87.3	48.3	32.3	94.7	64.8	39.0	60.1	-7.0	22.5
Gemini-3.5-Flash	89.8	48.6	37.6	95.3	58.8	47.0	61.7	-5.2	31.0

Table 4: Detailed performance breakdown across four specific adversarial constraints (

Y_{1}

: Entity Symbolization,

Y_{2}

: Entity-Region Spatial Grounding,

Y_{3}

: Factual Adversarial Constraint,

Y_{4}

: Minimalist Output). DCR values are percentages and eCDI values are percentage-point drops (the internal differences are multiplied by 100). Best results are bolded , and second-best are underlined .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Cascaded pass rates for entity grounding (L1), causal mechanism (L2), and task-specific causal conclusion (L3) across 14 models under baseline and constrained settings. Each stacked bar comprises the complete-chain pass rate (L3), the L2–L3 gap, and the L1–L2 gap; its total height equals the L1 pass rate.

Model	Vanilla DCR	CoT-A DCR	CoT-B DCR	$Y_{3}$ eDCR
Claude-Opus-4.7	90.7	89.6	88.5	94.9
Doubao-Seed-2-Pro	89.8	88.9	80.8	92.3
GPT-5.5	89.3	88.6	89.5	97.7
GLM-5V-Turbo	77.5	68.9	84.8	92.8

Appendix

Table 5: Comparison with general-purpose reasoning prompts. All values are percentages; the first three score columns report DCR, and the final column reports effective DCR under

Y_{3}

Model	DCR (Image/Text-only)	L1 raw (Image/Text-only)	L2 raw (Image/Text-only)	L3 raw (Image/Text-only)
Claude-Opus-4.7	90.67/42.17	95.75/48.13	92.75/49.75	89.00/69.75
Gemini-3.1-Pro-Preview	87.25/14.50	93.88/19.63	92.13/25.25	86.25/51.25
GPT-5.5	89.33/30.37	92.63/36.75	91.88/38.75	92.63/64.75
InternVL3-38B	68.13/18.00	79.38/25.63	73.13/23.13	72.38/46.88
Qwen3.6-35B-A3B	72.54/8.33	80.50/11.00	78.38/17.38	80.38/47.88
Average	81.58/22.67	88.42/28.23	85.65/30.85	84.12/56.10

Appendix

Table 6: Text-only diagnostic control. Entries report image-based/text-only scores in percent; L1–L3 are raw binary rates, whereas DCR uses the cascaded scoring rule. Averages are computed from unrounded scores.

Model	Baseline	Text-only+ $Y_{1}$	Oracle-informed+ $Y_{1}$	Gain over Text-only (points)
Claude-Opus-4.7	90.67	71.87	97.79	+25.92
Gemini-3.1-Pro-Preview	87.25	58.21	85.42	+27.21
GPT-5.5	89.33	62.29	86.08	+23.79
InternVL3-38B	68.13	45.24	74.92	+29.67
Qwen3.6-35B-A3B	72.54	36.50	74.83	+38.34
Average	81.58	54.82	83.81	+28.99

Appendix

Table 7: Image-based baseline and no-image

Y_{1}

controls (DCR in percent). The Oracle-informed condition supplies the symbolized adapted reference answer. It is an answer-informed diagnostic control, not a strict edge-only intervention. The gain is Oracle-informed minus Text-only. Averages and gains are computed from unrounded scores.

Model	Baseline	$Y_{1}$ : constrained $\rightarrow$ Oracle-informed	$Y_{2}$ : constrained $\rightarrow$ Oracle-informed	$Y_{4}$ : constrained $\rightarrow$ Oracle-informed
GPT-5.5	89.33	77.04 $\rightarrow$ 80.62 (+3.58)	75.46 $\rightarrow$ 80.79 (+5.33)	65.71 $\rightarrow$ 83.17 (+17.46)
Claude-Opus-4.7	90.67	75.46 $\rightarrow$ 93.75 (+18.29)	74.83 $\rightarrow$ 79.29 (+4.46)	63.29 $\rightarrow$ 81.38 (+18.08)
Gemini-3.1-Pro-Preview	87.25	48.25 $\rightarrow$ 76.50 (+28.25)	32.25 $\rightarrow$ 57.42 (+25.17)	64.79 $\rightarrow$ 76.83 (+12.04)
InternVL3-38B	68.13	48.08 $\rightarrow$ 74.75 (+26.67)	32.04 $\rightarrow$ 33.75 (+1.71)	51.04 $\rightarrow$ 80.12 (+29.09)
Qwen3.6-35B-A3B	72.54	43.58 $\rightarrow$ 60.58 (+17.00)	20.21 $\rightarrow$ 38.71 (+18.50)	52.96 $\rightarrow$ 76.62 (+23.67)
Average	81.58	58.48 $\rightarrow$ 77.24 (+18.76)	46.96 $\rightarrow$ 57.99 (+11.03)	59.56 $\rightarrow$ 79.63 (+20.07)

Appendix

Table 8: Image + Oracle-informed diagnostic control (DCR in percent; gains in percentage points). The supplied reference text is derived from answer fields:

Y_{1}

uses the symbolized adapted answer, while

Y_{2}

and

Y_{4}

use the baseline answer and causal rationale. The

Y_{2}

constrained entries use the fixed-800 DCR values from the final harmonized release; all entries remain DCR rather than CSR/eDCR. These are answer-informed diagnostics, not strict edge-only controls. Averages and gains are computed from unrounded scores.

Model	$X_{1}$ DCR	$X_{2}$ DCR	$X_{3}$ DCR	$X_{4}$ DCR	Avg DCR
GPT-5.5	82.3	92.7	88.2	94.2	89.3
Claude-Opus-4.7	89.7	91.0	89.3	92.7	90.7
Doubao-Seed-2-Pro	91.5	90.0	87.3	90.5	89.8
Gemini-3.1-Pro-Preview	85.7	85.8	88.0	89.5	87.3
Gemini-3.5-Flash	86.0	88.5	91.7	93.0	89.8
GLM-5V-Turbo	71.5	76.5	79.3	82.7	77.5

Appendix

Table 9: Performance on the unconstrained Baseline set. DCR values are percentages.

X_{1}

: Causal Discovery,

X_{2}

: Forward Prediction,

X_{3}

: Retrospective Diagnosis, and

X_{4}

: Intervention-Outcome Prediction.

Model	$Y_{1}$ DCR	$Y_{2}$ DCR	$Y_{3}$ DCR	$Y_{4}$ DCR	eDCR	eCDI (points)
GPT-5.5	77.0	75.5	97.7	65.7	76.9	12.4
Claude-Opus-4.7	75.5	74.8	95.2	63.3	74.5	16.1
Doubao-Seed-2-Pro	72.4	32.4	92.3	53.1	60.0	29.8
Gemini-3.1-Pro-Preview	48.3	32.3	94.7	64.8	58.6	28.6
Gemini-3.5-Flash	48.6	37.6	95.3	58.8	56.2	33.6
GLM-5V-Turbo	58.8	33.5	92.9	63.7	60.4	17.1

Appendix

Table 10: Performance on the constrained main set. The

Y_{1}

–

Y_{4}

columns report per-constraint DCR, whereas eDCR and eCDI aggregate all four constraints. DCR and eDCR are percentages; eCDI is the baseline-to-effective-score drop in percentage points, computed from unrounded fixed-denominator sample-level scores.

Y_{1}

: Entity Symbolization,

Y_{2}

: Spatial Grounding,

Y_{3}

: Factual Adversarial Constraint, and

Y_{4}

: Minimalist Output.

Figure 3: Example of benchmark construction (Image ID: 2317269)

Model	Parameters (B)	Access method	Model source
GPT-5.5	$\sim 9700$	https://platform.openai.com/docs/models	https://platform.openai.com/docs/models
Claude-Opus-4.7	$\sim 4000$	https://platform.claude.com/docs/en/api/models/list	https://platform.claude.com/docs/en/about-claude/models/overview
Gemini-3.1-Pro-Preview	$\sim$	https://ai.google.dev/gemini-api/docs/get-started	https://ai.google.dev/gemini-api/docs/models
Gemini-3.5-Flash	$\sim$	https://ai.google.dev/gemini-api/docs/get-started	https://deepmind.google/models/gemini/flash/
Doubao-Seed-2-Pro	$\sim$	https://docs.byteplus.com/en/docs/Byteplus_LAS/Multimodal-Deep-Thinking-Doubao-Seed-2-0#rest-api-call	https://docs.byteplus.com/en/docs/Byteplus_LAS/Multimodal-Deep-Thinking-Doubao-Seed-2-0
GLM-5V-Turbo	744	https://docs.bigmodel.cn/cn/api/introduction	https://huggingface.co/papers/2604.26752

Appendix

Table 11: Overview of the 14 evaluated MLLMs and the GPT-5.1 automated judge. Open/open-weight parameter sizes are taken from the corresponding official model cards. A numerical value prefixed by

\sim

denotes a literature-based estimate of effective knowledge capacity in open-model-equivalent parameters ( Li, 2026 ) , not an officially disclosed parameter count;

\sim

alone indicates that the parameter count is not publicly disclosed. Access methods and model sources are reported separately.

Metric	$n$	Unanimous	Fleiss’ $\kappa$	Human–GPT-5.1
L1 perception	280	249/280 (88.93%)	0.7143	227/280 (81.07%)
L2 interaction/mechanism	280	252/280 (90.00%)	0.7848	235/280 (83.93%)
L3 causal outcome	280	255/280 (91.07%)	0.8365	244/280 (87.14%)
Full-chain pass	280	234/280 (83.57%)	0.7415	217/280 (77.50%)

Appendix

Table 12: Independent human validation of the DCR rubric and GPT-5.1 judge. The audit contains 280 responses from 14 models. Unanimous denotes agreement among all three annotators. Human–GPT-5.1 denotes agreement between the human-majority label and the automated judge. Full-chain pass is the binary conjunction

L1\land L2\land L3

per annotator; the GPT-5.1 full-chain label is computed from its three binary layer scores.

Setting	$n$	Human DCR	GPT-5.1 DCR	Human–GPT-5.1 gap
Baseline	56	90.48	72.02	$+18.45$
$Y_{1}$	56	63.10	50.00	$+13.10$
$Y_{2}$	56	76.19	51.19	$+25.00$
$Y_{3}$	56	98.81	93.45	$+5.36$
$Y_{4}$	56	63.69	49.40	$+14.29$

Appendix

Table 13: Setting-level DCR on the 280-response human-audit subset (percent). Human DCR applies the standard cascade after layer-wise majority voting. The gap is human DCR minus GPT-5.1 DCR in percentage points. Values are computed from unrounded scores.

Panel of Figure 4 — Figure 4: Prompt used for generating causal reasoning questions from human-verified causal-edge specifications and scene-graph annotations. It enforces structured grounding, causal validity, and diversity across four dimensions ( $X_{1}$ – $X_{4}$ ).

Panel of Figure 5 — Figure 5: LLM judge prompts for evaluation. The DCR prompt decomposes causal reasoning into perception (L1), mechanism (L2), and outcome (L3), while the CSR prompt evaluates constraint adherence under contradiction settings ( $Y_{3}$ ). The prompts are scored separately; under $Y_{3}$ , their false-premise criteria partially overlap rather than forming a strictly orthogonal decomposition.

Explore similar work

Jun 1, 2025cs.CL

What's Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning

Despite the impressive performance of vision-language models (VLMs) on downstream tasks, their ability to understand and reason about causal relationships in visual inputs remains unclear. Robust causal reasoning is fundamental to solving complex high-level reasoning tasks, yet existing benchmarks often include a mixture of reasoning questions, and VLMs can frequently exploit object recognition and activity identification as shortcuts to arrive at the correct answers, making it challenging to truly assess their causal reasoning abilities. To bridge this gap, we introduce VQA-Causal and VCR-Causal, two new benchmarks specifically designed to isolate and rigorously evaluate VLMs' causal reasoning abilities. Our findings reveal that while VLMs excel in object and activity recognition, they perform poorly on causal reasoning tasks, often only marginally surpassing random guessing. Further analysis suggests that this limitation stems from a severe lack of causal expressions in widely used training datasets, where causal relationships are rarely explicitly conveyed. We additionally explore fine-tuning strategies with hard negative cases, showing that targeted fine-tuning can improve model's causal reasoning while maintaining generalization and downstream performance. Our study highlights a key gap in current VLMs and lays the groundwork for future work on causal understanding.

Zhaotian Weng, Haoxuan Li, Xin Eric Wang +2

University of Southern California · University of California, Santa Barbara · Texas A&M University

VLM Evaluation VLM Reasoning

Ask PDF

Jun 10, 2026cs.CV

From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning

Visual causal reasoning is essential for understanding and intervening in the physical world, requiring identification of causal variables from visual inputs and reasoning over intervention effects. Despite recent progress, large vision--language models (VLMs) remain brittle at such tasks, especially for interventional and counterfactual queries over multi-image inputs. Most existing explorations inject causal knowledge via textual prompts, leaving causal mechanisms external to model execution and limiting reliable control during inference. To address this problem, we propose BridgeVLM, which internalizes visual causal reasoning by inducing a causal graph from multi-image inputs and converting it into structured Causal Tokens executed by RAMP layers injected into the LLM decoder for causal message passing. We further introduce a unified training interface M3S for fine-grained causal supervision from different granularities (local/global level). BridgeVLM achieves 54.4% accuracy on intervention tasks on CausalVLBench (vs. 33.2% with prompt-level supervision), improves results on Causal3D from 43.6% to 49.0%, and substantially improves causal structure learning on CausalVLBench (

F_1

: 33.4%

\rightarrow

75.1%).

Haoping Yu, Yuanxi Li, Jing Ma

Department of Computer & Data Sciences, Case Western University, Cleveland, OH, US.

Visual Reasoning Causal Structure Learning Causal Reasoning in Language Models VLM Reasoning

Ask PDF

Jun 4, 2026cs.DB

Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs

Understanding and reasoning about the physical world is the foundation of intelligent behavior, yet state-of-the-art vision-language models (VLMs) still fail at causal physical reasoning, often producing plausible but incorrect answers. To address this gap, we introduce CausalPhys, a benchmark of over 3,000 carefully curated video- and image-based questions spanning four domains: Perception, Anticipation, Intervention, and Goal Orientation. Each question is paired with an expert-annotated causal graph capturing object-attribute-event dependencies, enabling interpretable and fine-grained evaluation of causal understanding. Building on this, we formulate a causal-graph-grounded metric that quantitatively measures how well a model's chain-of-thought reasoning aligns with the correct causal relations, moving beyond answer-only accuracy and enabling systematic diagnosis of VLMs' causal reasoning failures. Using this metric, we conduct a comprehensive analysis of leading VLMs, revealing systematic gaps in capturing causal dependencies and underscoring the need for causality-aware learning. To address these limitations, we further propose Causal Rationale-informed Fine-Tuning (CRFT), which explicitly aligns VLM reasoning with causal structures. Extensive experiments demonstrate that CRFT substantially enhances both reasoning accuracy and interpretability across multiple model backbones. By unifying dataset curation, causal evaluation, and causality-informed learning, CausalPhys establishes a strong foundation for advancing modern VLMs toward causally grounded physical reasoning.

Tianyi Tang, Zhuoyi Lin, Zeyu Feng +4

CFAR, IHPC, Agency for Science, Technology and Research (A*STAR) Singapore · I2R, Agency for Science, Technology and Research (A*STAR) Singapore · Nanyang Technological University Singapore

VLM Evaluation VLM Reasoning Physical Reasoning in Video

Ask PDF