VR-JEPA: Learning Contrastive-State Latent Guidance for Generation-based Video Reasoning

Zehua Ma; Kun Xiang; Yunshuang Nie; Quanlin Chen; Haoyuan Li; Xiuwei Chen; Jiang Ji; Haijun Wu

cs.CVSep 30, 2026

VR-JEPA: Learning Contrastive-State Latent Guidance for Generation-based Video Reasoning

Authors: Zehua Ma, Kun Xiang, Yunshuang Nie, Quanlin Chen, Haoyuan Li, Xiuwei Chen, Jiang Ji, Haijun Wu, +4 more

Organizations: Shenzhen Campus of Sun Yat-sen University · Tsinghua Shenzhen International Graduate School · Shenzhen Loop Area Institute · Tencent · Mohamed bin Zayed University of Artificial Intelligence · UiT The Arctic University of Norway

Abstract

Reasoning through video generation offers a promising path toward visual intelligence by modeling latent visual states and their dynamics. However, current video generation models often lack explicit guidance on how these states should evolve, leaving generated trajectories prone to physical and structural inconsistencies that undermine reasoning reliability. While the Video Joint-Embedding Predictive Architecture (V-JEPA) provides rich spatiotemporal priors learned through latent prediction, these general priors do not naturally adapt to the logical reasoning capabilities required for complex visual tasks. To bridge this gap, we propose VR-JEPA, a framework that aligns the V-JEPA predictor with task-specific reasoning logic through localized contrastive-state learning and uses its predicted latent trajectories to guide video generation for visual reasoning. Specifically, (i) we pair successful trajectories with generated alternatives under the same input conditions and use discrepancies in their V-JEPA representations to identify informative states and tokens for localized contrastive supervision. (ii) We further equip the V-JEPA predictor with skill-specific experts trained on anchor-task data, allowing the model to adaptively specialize its shared spatiotemporal priors across diverse cognitive domains. Together with skill-specific experts, this contrastive supervision enables VR-JEPA to predict latent trajectories that provide task-specific logical guidance for video generation. Comprehensive experiments on the large-scale VBVR-Pro-Bench dataset demonstrate that VR-JEPA achieves an $11.33\%$ relative improvement over the cutting-edge generation-based reasoning baseline, significantly mitigating physical artifacts and enhancing logical consistency.

Figures & tables

Figure 1: Generation-based visual reasoning with VR-JEPA. Top: Comparison of different video reasoning paradigms: (1) direct video generation; (2) video generation guided by static DINOv3 features of the initial state; and (3) VR-JEPA, which adapts V-JEPA’s spatiotemporal priors to diverse reasoning skills and guides video generation through predicted latent trajectories. Bottom: Video reasoning results of VR-JEPA and VBVR-Wan2.2, showing that VR-JEPA generates trajectories that satisfy task requirements. Red dashed boxes highlight reasoning errors.

Figure 2: Overview of VR-JEPA. VR-JEPA comprises a Latent Reasoner and a Video Generator. The Latent Reasoner predicts a latent rollout from the initial scene and task instruction, with SR-MoE adaptively selecting expert combinations for different reasoning tasks. Its seven experts cover VIS (visual discrimination), NUM (numerical reasoning), IDN (object identity and persistence), SPA (spatial reasoning and planning), DYN (physical dynamics), CMP (compositional reasoning), and ABS (abstract rule inference). During predictor training, localized contrastive-state learning uses dense V-JEPA discrepancies between successful and generated videos to focus contrastive supervision on tokens corresponding to informative states and regions. The Video Generator conditions Wan2.2 DiT on the rollout to synthesize a reasoning video.

Figure 3: State selection for localized contrastive-state learning. Successful reference trajectories and task-matched generated candidates are compared in dense V-JEPA feature space. After state-specific quantile subtraction, large residual discrepancies select states for contrastive supervision (green checks), while small discrepancies exclude them (red crosses).

		In-Domain (ID) (%)						Out-of-Domain (OOD) (%)
Models	Overall (%)	Avg.	Abst.	Know.	Perc.	Spat.	Trans.	Avg.	Abst.	Know.	Perc.	Spat.	Trans.
Open-source Video Models
Wan2.2-I2V-A14B	18.2	15.7	12.3	15.2	15.0	17.0	23.3	20.7	29.2	17.9	13.8	19.5	31.8
LTX-2.3-I2AV	11.2	10.6	9.3	12.3	9.7	13.6	5.1	11.9	21.7	19.8	7.0	9.1	5.9
Wan2.1-I2V-14B-720P	10.0	10.5	8.7	14.6	12.4	9.9	4.0	9.5	13.9	8.7	7.6	12.3	5.2
Wan2.2-TI2V-5B	9.4	6.6	5.0	7.4	7.7	9.2	2.5	12.2	22.2	9.2	9.1	6.3	11.6

Table 1: Main results on VBVR-Pro-Bench ( Xu et al., 2026 ) . We report overall, in-domain (ID), and out-of-domain (OOD) scores, followed by results for abstraction (Abst.), knowledge (Know.), perception (Perc.), spatial reasoning (Spat.), and transformation (Trans.). Avg. denotes the mean of task scores weighted by the number of samples per task. All scores are reported as percentages. Higher is better. Bold and underlined values indicate the best and second-best results.

Figure 4: Qualitative comparison on VBVR-Pro-Bench. Given the initial state and instruction, we compare the reasoning results of VR-JEPA with those of a video reasoning model (VBVR-Wan2.2), an open-source video model (LTX-2.3-I2AV), and two proprietary video models (Kling VIDEO 3.0 and Seedance 2.0). Green and red dashed boxes highlight correct results and reasoning errors, respectively.

Figure 5: Qualitative ablation of localized contrastive-state learning (LCL) and skill-routed mixture of experts (SR-MoE). Top and middle: adding LCL and SR-MoE to the dense baseline, respectively. Bottom: adding SR-MoE to the LCL-equipped predictor to obtain the full model. Green and red dashed boxes highlight correct results and reasoning errors, respectively.

Variant	Components		Overall (%)	In-Domain (ID) (%)						Out-of-Domain (OOD) (%)
Variant	LCL	SR-MoE	Overall (%)	Avg.	Abst.	Know.	Perc.	Spat.	Trans.	Avg.	Abst.	Know.	Perc.	Spat.	Trans.
Dense baseline	✗	✗	53.9	63.4	53.8	58.9	58.9	68.8	91.4	44.4	43.3	44.5	31.9	59.3	75.3
+ LCL	✓	✗	55.1	64.9	52.6	61.2	61.3	71.6	93.0	45.3	44.1	38.5	33.1	65.4	75.8
+ SR-MoE	✗	✓	55.5	65.2	54.0	61.4	60.7	69.8	95.6	45.9	42.8	42.3	34.3	64.2	77.3
Full model	✓	✓	56.0	65.9	56.3	63.8	61.8	69.1	91.4	46.1	41.5	42.1	35.2	65.2	77.5

Table 2: Ablation of localized contrastive-state learning (LCL) and skill-routed mixture of experts (SR-MoE). Scores are reported as percentages; higher is better.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Training strategy	Trainable modules		Task success (%)
Training strategy	TCI	LoRA	ID $\uparrow$	OOD $\uparrow$
VBVR-Wan2.2	✗	✗	55.4	45.1
LoRA-only tuning	✗	✓	57.4	45.5
TCI-only tuning	✓	✗	63.1	43.7
Joint tuning	✓	✓	65.9	46.1

Appendix

Table 4: Ablation of the training strategy. We compare tuning the trajectory-conditioning interface (TCI), LoRA parameters, or both, with VBVR-Wan2.2 as the baseline. ID and OOD task-success scores are reported on VBVR-Pro-Bench as percentages; higher is better.

Routing strategy	Task success(%)
Routing strategy	ID $\uparrow$	OOD $\uparrow$
Shared FFN only	64.2	44.7
Uniform routing	63.9	44.7
Random routing	64.8	44.8
Learned routing	65.9	46.1

Appendix

Table 5: Ablation of the SR-MoE routing strategy. We compare learned routing with uniform routing, random routing, and a shared-FFN-only variant to assess the contribution of adaptive expert composition. Shared FFN only disables the expert branches of the trained full model at inference, retaining the shared FFNs. ID and OOD task-success scores are reported on VBVR-Pro-Bench as percentages; higher is better.

Figure 6: Qualitative comparison of SR-MoE routing strategies. We compare video reasoning results with learned routing, shared FFN only, uniform routing, and random routing. The shared-FFN-only variant disables expert branches at inference. Disrupting learned expert coordination degrades reasoning performance in the illustrated examples. Green and red dashed boxes highlight correct and incorrect results, respectively.

Rollout condition	Intervention	Task success (%)
Rollout condition		ID $\uparrow$	OOD $\uparrow$
Predicted rollout	Use the model-predicted trajectory	65.9	46.1
No conditioning	Remove all trajectory conditioning	51.5	44.9
Repeated initial state	Repeat the initial state across all rollout steps	62.5	43.3
Cross-sample rollout	Use the predicted trajectory from another sample	58.5	45.9

Appendix

Table 6: Ablation of latent trajectory guidance. We test whether task success depends on the sample-specific temporal content of the rollout, rather than merely the presence of an additional conditioning signal.

Figure 7: Additional qualitative comparisons with open-source models on VBVR-Pro-Bench. Given the initial state and instruction, we compare the video reasoning results of VR-JEPA with VBVR-Wan2.2 and LTX-2.3-I2AV. Green and red dashed boxes highlight correct results and reasoning errors, respectively.

Figure 8: Additional qualitative comparisons with proprietary video models on VBVR-Pro-Bench. Given the initial state and instruction, we compare the video reasoning results of VR-JEPA with Kling VIDEO 3.0 and Seedance 2.0. Green and red dashed boxes highlight correct results and reasoning errors, respectively.

Figure 9: Qualitative comparison of visual representations on VBVR-Pro-Bench. Given the initial state and instruction, we compare video reasoning results guided by, from left to right: (1) predicted latent rollouts in V-JEPA 2.1 space (VR-JEPA), (2) initial-state DINOv3 features, and (3) predicted latent rollouts in DINOv3 space. Green and red dashed boxes highlight correct results and reasoning errors, respectively.

Figure 10: Qualitative ablation of latent trajectory guidance. We compare video reasoning results using the predicted rollout with three interventions: removing trajectory conditioning, repeating the initial state across rollout steps, and substituting a rollout from another sample. Red dashed boxes highlight reasoning errors.

Figure 11: Task-dependent expert routing. Mean routing weights for ten tasks, averaged across the two SR-MoE predictor layers (layer4 and layer5). Rows denote tasks and columns denote the seven skill-specific experts; darker cells indicate larger weights. The distributions show both concentrated expert selection and allocations spread across multiple experts.

Figure 12: Probing expert specialization through skill removal. We remove the highest-weight expert for each of seven examples and examine whether the resulting errors reflect the loss of its associated reasoning skill. From top to bottom, the removed experts correspond to visual discrimination, numerical reasoning, object identity and persistence, spatial reasoning and planning, physical dynamics, compositional reasoning, and abstract rule inference. Green and red dashed boxes highlight correct results and reasoning errors, respectively.

Explore similar work

Sep 12, 2026cs.CV

From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models

Video generation has advanced to produce visually compelling and temporally coherent results. Yet, whether these models can genuinely think with video--executing symbolic rules, respecting physical laws, and pursuing intentional goals--remains an open question. Existing benchmarks only partially address this, often conflating visual quality with cognitive correctness. We introduce VWG-Bench (Video World Generalist Benchmark), a comprehensive benchmark spanning 9 reasoning dimensions and 38 fine-grained tasks. To enable precise diagnosis, we design a three-level VLM-as-Judge protocol that independently assesses video-level fluency, task-level rule adherence, and sample-level goal realization. Evaluations of leading models reveal a striking gap: while models achieve strong rendering scores, they consistently fail on logic-heavy and rule-constrained tasks. To address this, we propose Vid-PRE (Video Prompt Reasoner and Enhancer), a model-agnostic prompt rewriter that offloads the cognitive burden of reasoning to a dedicated VLM. Trained via reinforcement learning with purely text-based rewards, Vid-PRE produces concise, constraint-aware prompts without the instability of video-level reward signals. Experiments show that Vid-PRE yields substantial reasoning improvements across multiple generators without architectural modifications. Together, VWG-Bench and Vid-PRE offer a rigorous diagnostic lens and a scalable path toward true think-with-video capabilities. All data and code are publicly available at https://huggingface.co/datasets/KlingTeam/VWG-Bench.

Meng Luo, Yicheng Liu, Jiahao Wang +5

School of Computing, National University of Singapore, Singapore, Singapore · Kling Team, Kuaishou Technology, Beijing, China · Department of Computer Science and Technology, Nanjing University, Nanjing, China +1

Generative Video Models Video Understanding Reasoning Skills

Ask PDF

Jul 9, 2026cs.CV

OpenCoF: Learning to Reason Through Video Generation

Reasoning has become a core capability for large models, especially when reliable decisions require understanding logical consequences. Recent video generation models offer a reasoning path distinct from previous Chain-of-Thought (CoT): reasoning can unfold through temporally connected frames, known as Chain-of-Frame (CoF) reasoning. However, existing video generators are primarily trained on general video corpora, still lacking diverse supervision and dedicated designs for CoF reasoning. To address this gap, we introduce OpenCoF, a framework comprising the OpenCoF-17K dataset, a reasoning video dataset spanning 11 task families, and Wan-CoF, a fine-tuned video model for studying whether diverse temporal supervision improves CoF behavior. Across four video reasoning benchmarks, Wan-CoF achieves considerable gains over the Wan2.2-I2V-A14B baseline. Building on this, we empirically explore more advanced designs for CoF capabilities, i.e., equipping the model with visual and textual reasoning tokens. This mechanism respectively captures low-level visual cues and high-level semantic priors for spatial and temporal reasoning. Through performance comparisons and attention analysis, we examine how these tokens contribute across model depth, denoising steps, space, and time. Our results suggest that stronger video reasoning requires both broad temporal supervision and explicit mechanisms for organizing intermediate reasoning state. We open-source the dataset, model, and code to facilitate future research on reasoning-oriented video generation.

Xinyan Chen, Ziyu Guo, Renrui Zhang +2

ByteDance Seed · CUHK IMIXR · CUHK MMLab

Video Generation Video Understanding Video Dataset Reasoning Skills Chain-of-Thought Reasoning

Ask PDF

May 9, 2026cs.CV

CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models

Recent "Thinking with Video" approaches use Video Generation Models (VGMs) for visual reasoning by producing temporally coherent Chain-of-Frames as reasoning artifacts. Even strong VGMs, however, exhibit two recurring failure modes on goal-directed tasks: long-horizon drift on multi-step tasks and mid-clip simulation errors that compound. Both stem from the absence of explicit reasoning built upon the VGM's short-horizon visual prior, a role naturally filled by Vision-Language Models (VLMs), but where to place the VLM is non-trivial: upfront plans commit before any frame is generated and post-hoc critiques over whole videos intervene too late. We propose VLM-VGM Collaborative Video Reasoning (CollabVR), a closed-loop framework that couples the VLM with the VGM at step-level granularity: the VLM plans the immediate next action, inspects the clip the VGM generates, and folds the verifier's diagnosis directly into the next action prompt to repair detected failures. On Gen-ViRe and VBVR-Bench, CollabVR improves both open-source and closed-source VGMs over single-inference, Pass@

k

, and prior test-time scaling baselines at matched compute, with the largest gains on the hardest tasks. It also yields further improvements on top of a reasoning-fine-tuned VGM, indicating that step-level VLM supervision is orthogonal to and stackable with reasoning-oriented fine-tuning. We provide video samples and additional qualitative results at our project page: https://joow0n-kim.github.io/collabvr-project-page.

Joowon Kim, Seungho Shin, Joonhyung Park +1

KAIST · Kyung Hee University · AITRICS

Video Understanding Visual Reasoning Video Generation

Ask PDF