CST-Bench: Benchmarking Counterfactual Spatial Transition Reasoning with Causal Transition On-Policy Self-Distillation

Ying Li; Yongle Cheng; Jing Li; Xingquan Cai; Yawei Luo

cs.AINov 5, 2026

CST-Bench: Benchmarking Counterfactual Spatial Transition Reasoning with Causal Transition On-Policy Self-Distillation

Authors: Ying Li, Yongle Cheng, Jing Li, Xingquan Cai, Yawei Luo

Organizations: School of Artificial Intelligence and Computer Science, North China University of Technology, Beijing, China · School of Artificial Intelligence, Jilin University, Changchun, China · ReLER Lab, Zhejiang University, Hangzhou, China

Abstract

Spatial reasoning is a fundamental capability of multimodal large language models (MLLMs), yet existing benchmarks mainly focus on static scene understanding. When objects are moved or swapped, models must infer not only spatial relations but also how interventions change the scene state. To address this problem, we introduce CST-Bench, a benchmark for counterfactual spatial reasoning. CST-Bench contains 7,082 examples covering 6 intervention types and 12 transformation operators, with dense causal annotations including intervention sources, affected relations, expected effects, invariant relations, and metric changes. We benchmark general-purpose MLLMs on CST-Bench and further compare post-training strategies on CST-Bench and standard spatial benchmarks. Results show that methods effective for ordinary spatial reasoning can still fail under counterfactual interventions. We further formulate counterfactual spatial reasoning as causal state-transition modeling and propose CT-OPSD, which uses CST-Bench annotations to guide a teacher model in learning counterfactual state transitions and causal reasoning chains. These chains are then distilled into a student model that reasons only from real observations. Overall, experiment results establish CST-Bench as a challenging benchmark and reusable supervision resource for counterfactual spatial reasoning. CT-OPSD further provides an initial demonstration of how its annotations can support model improvement.

Figures & tables

Figure 1: Counterfactual spatial transition reasoning. Moving the lamp backward changes its relation to the TV from in front of to behind , while unaffected scene properties remain invariant.

Figure 2: Overview of the CST-Bench transition taxonomy. CST-Bench contains 7,082 counterfactual spatial QA samples across six intervention families and twelve transition operators. The operators cover relational state updates, metric distance changes, and qualitative relation-threshold crossings, with dense causal transition annotations.

Figure 3: Overview of CST-Bench construction. Constraint-aware generation jointly derives counterfactual states and annotations, answer-blind VLM consensus filters ambiguous candidates, and uncertainty-guided human verification resolves uncertain cases and audits all transformation operators.

Figure 4: Overview of CT-OPSD with CST-Bench supervision. Privileged transition annotations from CST-Bench train a causal teacher but are never provided as student inputs. The frozen teacher aligns with the observation-only student on student-generated prefixes, and only the updated student is retained for inference.

	Transition Families						Aggregate Results
Model	Mirror	Swap	Metric	Translation	Depth	Viewpoint	Macro Avg.	Micro Acc.
General-purpose Open-source MLLMs
Qwen2.5-VL-3B	15.5	15.1	17.8	20.7	11.6	17.0	16.3	17.0
Qwen3-VL-4B	42.5	25.0	35.6	96.3	37.5	30.0	44.5	49.4
Qwen2.5-VL-7B	27.6	19.2	22.3	36.3	12.5	20.6	23.1	25.1
LLaVA-OneVision-7B	26.3	35.0	39.4	90.0	32.5	27.5	41.8	46.6

Table 1: Performance of MLLMs on CST-Bench (%) under the answer-only protocol on the balanced 480-example evaluation split. Best and second-best results are bolded and underlined, respectively.

	CST-Bench	3DSRBench	CV-Bench			BLINK val
Model	Micro Acc.	Acc.	2D	3D	Avg.	Spatial Relation	Relative Depth	Avg.
Post-training Strategies with Qwen2.5-VL-3B
Qwen2.5-VL-3B	17.0	46.0	64.0	66.0	65.0	83.0	64.0	73.5
Qwen2.5-VL-3B + SFT	50.3	38.0	63.0	68.0	65.5	70.0	67.0	68.5
Qwen2.5-VL-3B + GRPO	46.7	52.9	71.0	76.3	73.7	81.8	66.9	74.4
Qwen2.5-VL-3B + OPSD	53.3	51.8	73.8	78.3	76.1	82.5	67.2	74.9

Table 2: Post-training results on CST-Bench and standard spatial benchmarks (%). All variants use Qwen2.5-VL-3B as the base model. Best and second-best results are bolded and underlined, respectively.

Variant	CST W-Avg.	Ans. Acc.	Src.	Eff.	Inv.	Cons.
OPSD	24.0	53.3	34.1	0.7	0.3	22.5
CT w/o Aux.	24.7	52.7	40.1	0.9	0.4	30.0
CT w/o Bridge	24.5	52.1	40.0	1.0	0.7	30.6
CT w/o Eff.	24.0	50.4	40.0	1.2	1.0	28.5
CT w/o Inv.	24.6	51.7	41.2	1.3	0.5	31.2
CT w/o Metric	24.1	50.4	41.3	1.2	0.5	28.5

Table 3: Component ablation of CT-OPSD. CST W-Avg. is the weighted diagnostic aggregate. Best and second-best results are bolded and underlined, respectively.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Representative source images from the six CST-Bench intervention families. Each example is paired with a single intervention-conditioned query and a symbolic target transition.

Family	Samples	Share
Global Mirror Transformation	1,600	22.6%
Object Position Swap	1,600	22.6%
Metric Distance Transition	1,600	22.6%
Translation-Based Crossing	1,300	18.4%
Depth Transition	300	4.2%
Viewpoint Reversal	682	9.6%

Appendix

Table 4: Full-dataset distribution by intervention family.

Transition operator	Samples	Transition operator	Samples
Pairwise Left–Right Flip	1,300	Lateral Relation Crossing	386
Multi-object Horizontal Order Flip	300	Above–Below Crossing	614
Pairwise Relation Swap	300	Vertical Near–Far Crossing	300
Third-object Relation Update	1,300	Depth Order Change	300
Metric Distance Change (Relation Preserved)	1,300	Front-to-Back Viewpoint Shift	382
Distance-to-Reference Near–Far Crossing	300	Side-to-Side Viewpoint Shift	300

Appendix

Table 5: Sample counts by transition operator. The official evaluation split contains 40 examples per operator.

Family	Operator	Precondition	Intervention	Affected / expected	Principal invariants
Global mirror	Pairwise Left–Right Flip	Clear horizontal pair	Mirror image coordinates	left $\leftrightarrow$ right	identity, count, vertical relation
Global mirror	Multi-object Horizontal Order Flip	Three ordered, distinct objects	Mirror all selected objects	reverse all horizontal pair orders	identity, count, vertical relations
Object swap	Pairwise Relation Swap	Two distinct objects	Exchange their positions	queried-pair relation reverses	identity, count, attributes
Object swap	Third-object Relation Update	Swapped pair plus unchanged reference	Exchange two positions	update relation to third object	reference object, identity, count
Metric distance	Metric Distance Change (Relation Preserved)	Clear near/far pair	Change subject–reference distance	metric distance changes; label preserved	direction, identity, image-plane order
Metric distance	Distance-to-Reference Near–Far Crossing	Clear pair with valid offset	Change distance without crossing reference object	near $\leftrightarrow$ far	side of reference, identity, orthogonal relation

Appendix

Table 6: Complete CST-Bench operator specification.

Field	Meaning
sample_uid	Stable sample identifier.
image_id	Source-image identifier used for grouped splitting.
operator	Intervention family and fine-grained transition operator.
question	Intervention-conditioned natural-language query.
options , answer	Answer choices and target choice.
source_state	Relevant source objects and spatial relations.

Appendix

Table 7: Principal CST-Bench fields.

Component	Role
Observable input $x$	Image, question, intervention description, and answer options.
Causal teacher	Predicts source state, intervention consequences, invariants, target state, and answer.
Student rollout	Samples a trajectory from the observation-only student policy.
On-policy alignment	Matches the teacher and student distributions on the shared student prefix.
Answer supervision	Anchors the final response to the gold answer $a^{*}$ .
Inference student	Uses only the image and question after teacher removal.

Appendix

Table 8: CT-OPSD components and their roles.

Spatial Reasoning Benchmarks

Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.

See all 168 papers

CVT-Bench: Probing Spatial-State Integrity through Counterfactual Viewpoint Transformations
Mar 22, 2026Shanmukha Vellamcheti, Uday Kiran Kothapalli, Disharee Bhowmick +1Spatial Reasoning Benchmarks Multi-View Consistency
Department of Computer Science and Software Engineering, Auburn University Auburn, AL 36849, USA
SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models
Oct 8, 2026Hongxing Li, Jinyue Su, Dingming Li +5Spatial Reasoning Spatial Reasoning Benchmarks
Zhejiang University
Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs
Aug 5, 2026Yang Yang, Jiawei Chen, Tairan Chen +1Multimodal Grounding Visual Spatial Reasoning
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy · Stevens Institute of Technology

Mar 22, 2026J/K move · Enter open · S save

CVT-Bench: Probing Spatial-State Integrity through Counterfactual Viewpoint Transformations

Shanmukha Vellamcheti, Uday Kiran Kothapalli, Disharee Bhowmick, Sathyanarayanan N. Aakur

Department of Computer Science and Software Engineering, Auburn University Auburn, AL 36849, USA

Spatial Reasoning Benchmarks Multi-View Consistency VLM Robustness Counterfactual Evaluation of VLMs

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) perform strongly on isolated spatial tasks, but whether their predictions remain persistent and mutually coherent across viewpoints and competing scenes is unclear. We formalize this behavioral property as spatial-state integrity and introduce CVT-Bench, a factorial diagnostic suite spanning two domains (CVT-Synthetic, CVT-Real), four context regimes (CVT-Isolated, irrelevant filler, attribute filler, CVT-Competing), three representations (Image, Text/BBox, Scene Graph), and ten viewpoint conditions (nine azimuths from

0^\circ

360^\circ

45^\circ

increments, plus Top) with target views withheld. Across 200 scenes and 11,833 relational queries, we measure counterfactual accuracy, cycle consistency, the normalized survival coefficient, spatial realizability, and context-specific interference. Five state-of-the-art MLLMs exhibit rapid persistence loss and frequently produce jointly unrealizable states despite high local accuracy, with failures amplified by competing contexts and natural-scene complexity. Matched irrelevant and attribute filler controls show that prompt length, position, or loss of scene access alone are insufficient explanations. On average, Text/BBox improves accuracy, persistence, and realizability, while Scene Graph provides complementary gains; neither eliminates the instability. Thus, isolated spatial accuracy substantially overestimates robustness, establishing spatial-state integrity as a distinct evaluation target. The complete benchmark and codebase will be publicly released.

Abstract

Spatial Reasoning Benchmarks

Explore similar work

CVT-Bench: Probing Spatial-State Integrity through Counterfactual Viewpoint Transformations

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs