ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

Haichao Zhang; Yijiang Li; Shwai He; Tushar Nagarajan; Mingfei Chen; Jianglin Lu; Ang Li; Yun Fu

cs.CVMar 23, 2026

ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

Authors: Haichao Zhang, Yijiang Li, Shwai He, Tushar Nagarajan, Mingfei Chen, Jianglin Lu, Ang Li, Yun Fu

Organizations: Northeastern University · University of California San Diego · University of Maryland · The University of Texas at Austin · University of Washington

Abstract

Recent progress in latent world models (e.g., V-JEPA2) has shown promising capability in forecasting future world states from video observations. Nevertheless, dense prediction from a short observation window limits temporal context and can bias predictors toward local, low-level extrapolation, making it difficult to capture long-horizon semantics and reducing downstream utility. Vision--language models (VLMs), in contrast, provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames, but they are not ideal as standalone dense predictors due to compute-driven sparse sampling, a language-output bottleneck that compresses fine-grained interaction states into text-oriented representations, and a data-regime mismatch when adapting to small action-conditioned datasets. We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance via a dual-temporal pathway: a dense JEPA branch for fine-grained motion and interaction cues, and a uniformly sampled VLM thinker branch with a larger temporal stride for knowledge-rich guidance. To transfer the VLM's progressive reasoning signals effectively, we introduce a hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction. Experiments on hand-manipulation trajectory prediction show that our method outperforms both a strong VLM-only baseline and a JEPA-predictor baseline, and yields more robust long-horizon rollout behavior.

Figures & tables

Figure 1: Overall Architecture of ThinkJEPA. ThinkJEPA couples a dense JEPA branch for fine-grained latent dynamics modeling with a uniformly sampled VLM- thinker branch that provides long-horizon semantic guidance. The VLM guidance—including visual tokens from the ViT visual tokenizer and intermediate hidden states from the language model—is distilled by a pyramidal representation extraction module and injected into the V-JEPA predictor via layer-wise modulation. Concretely, guidance derived from language-model layers $\{L_{0},\dots,L_{N}\}$ is mapped to modulation parameters for predictor layers $\{T_{0},\dots,T_{K}\}$ . The predicted future latents are concatenated with past teacher latents to form the full latent sequence, which is then fed into a task head to produce downstream trajectory predictions.

Dataset	Model	ADE $\downarrow$	FDE $\downarrow$	Acc $\uparrow$	FD $\downarrow$	SL1 $\downarrow$	CD $\downarrow$
EgoDex	Qwen3-VL Thinking	0.142	0.144	0.084	99.538	1.656	0.615
	V-JEPA Predictor	0.071	0.066	0.471	74.223	1.252	0.317
	ThinkJEPA	0.061	0.056	0.596	74.032	1.248	0.315
EgoExo4D	Qwen3-VL Thinking	0.661	0.690	0.038	104.548	1.756	0.690
	V-JEPA Predictor	0.659	0.636	0.074	89.244	1.520	0.469
	ThinkJEPA	0.622	0.597	0.171	79.654	1.364	0.359

Table 1: Quantitative comparison across datasets. We report trajectory metrics (ADE/FDE/Acc) and latent forecasting metrics (FD/SL1/CD). FD/SL1/CD denote V-JEPA feature distance, latent SmoothL1, and latent cosine distance, respectively. All values are reported with three decimal places.

Group	Model	ADE $\downarrow$	FDE $\downarrow$
Trajectory Baselines	Decoder-only + Behavior Cloning	0.0767	0.0818
	Decoder-only + DDPM	0.1148	0.1238
	Decoder-only + Flow Matching	0.1527	0.1574
	Encoder-decoder + Behavior Cloning	0.0774	0.0924
	Encoder-decoder + DDPM	0.1272	0.1245
	Encoder-decoder + Flow Matching	0.1736	0.1557

Table 2: Comparison with EgoDex trajectory prediction baselines on EgoDex. We compare ThinkJEPA against the trajectory prediction baselines reported in EgoDex, including decoder-only and encoder-decoder architectures with Behavior Cloning, DDPM, and Flow Matching. ThinkJEPA achieves the best ADE/FDE among all compared methods.

Table 4

Abl.	ADE $\downarrow$	FDE $\downarrow$	Acc $\uparrow$	FD $\downarrow$	SL1 $\downarrow$	CD $\downarrow$
Encoder+V-JEPA predictor	0.128	0.129	0.100	$78.869$	$1.340$	$0.360$
Encoder-only	$0.143$	$0.145$	$0.086$	$102.910$	$1.700$	$0.615$
AR+V-JEPA predictor	$0.128$	$0.130$	$0.098$	78.514	1.333	0.356
AR-only	$0.142$	$0.144$	$0.086$	$102.910$	$1.700$	$0.615$
No-dual-temporal sampling	$0.128$	$0.130$	$0.099$	$78.862$	$1.340$	$0.360$
No-Th	$0.071$	$0.066$	$0.471$	$74.223$	$1.252$	$0.317$

Table 5: Ablation studies. We vary the VLM token sources and the thinker module. Encoder denotes encoding tokens, AR denotes autoregressive tokens; No-dual disables the dual-temporal VLM pathway, and No-Th removes the thinker module. We abbreviate latent metrics as FD (feature distance), SL1 (SmoothL1), and CD (cosine distance).

Variant	ADE $\downarrow$	FDE $\downarrow$	Acc $\uparrow$	FD $\downarrow$	SL1 $\downarrow$	CD $\downarrow$
Last-layer	0.128	0.130	0.099	$78.858$	$1.340$	$0.360$
Mid-layer	$0.128$	$0.131$	$0.098$	78.517	1.333	0.356
All layers (ThinkJEPA)	0.061	0.056	0.596	74.747	1.263	0.324

Table 6: VLM layer selection on EgoDex. We compare guidance derived from different VLM layer selections. FD denotes V-JEPA feature distance, SL1 denotes latent SmoothL1, and CD denotes latent cosine distance.

Model	A@4	A@8	A@16	A@32	F@4	F@8	F@16	F@32
Qwen3-VL Thinking	0.140	0.819	1.375	1.026	0.143	2.850	0.286	1.092
V-JEPA Predictor	0.121	0.126	0.134	0.142	0.124	0.136	0.149	0.153
ThinkJEPA	0.071	0.078	0.092	0.111	0.073	0.090	0.118	0.136

Table 7: Recursive rollout on EgoDex: trajectory error vs. horizon. We perform autoregressive rollout for horizons

H\in\{4,8,16,32\}

. A@H and F@H denote ADE@H and FDE@H, respectively; the lower, the better.

Figure 2: Qualitative results. Predicted future hand-manipulation trajectories visualized as heat maps overlaid on the reference frame. Colors indicate temporal progression from blue (earlier) to red (later). Ideally, trajectories transition smoothly from blue to red, indicating coherent motion over time. ThinkJEPA produces smoother trajectories with better temporal consistency and joint alignment.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	ADE $\downarrow$	FDE $\downarrow$	Acc $\uparrow$	FD $\downarrow$	SL1 $\downarrow$	CD $\downarrow$
Prompt + video $\rightarrow$ VLM condition	0.069	0.062	0.495	74.007	1.248	0.315
ThinkJEPA	0.061	0.056	0.596	74.747	1.263	0.324

Appendix

Table 8: Prompt + video to VLM-conditioned features. The predictor uses cached visual features as the main trajectory backbone and VLM-derived features as external conditioning.

Stride	ADE $\downarrow$	FDE $\downarrow$	Acc $\uparrow$	FD $\downarrow$	SL1 $\downarrow$	CD $\downarrow$
Temporal stride 1	0.071	0.064	0.471	73.920	1.246	0.314
Temporal stride 2	0.073	0.071	0.458	74.266	1.247	0.317

Appendix

Table 9: Temporal stride ablation. Denser temporal sampling improves trajectory prediction and latent forecasting quality.

Conditioning	ADE $\downarrow$	FDE $\downarrow$	Acc $\uparrow$	FD $\downarrow$	SL1 $\downarrow$	CD $\downarrow$
FiLM	0.0706	0.064	0.471	73.878	1.245	0.314
Cross-attn	0.0707	0.066	0.475	73.965	1.247	0.315
AdaLN	0.0708	0.065	0.474	74.280	1.253	0.317

Appendix

Table 10: Conditioning mechanism ablation. We compare FiLM, cross-attention, and AdaLN under the same training setup.

Variant	ADE $\downarrow$	FDE $\downarrow$	Acc $\uparrow$
Direct visual conditioning	0.071	0.066	0.475
Drop deepstack tokens	0.072	0.066	0.464
ThinkJEPA	0.061	0.056	0.596

Appendix

Table 11: Direct visual conditioning vs. deepstack-token removal. Both ablations remain competitive, while ThinkJEPA achieves the strongest downstream trajectory performance.

Baseline	ADE $\downarrow$	FDE $\downarrow$	Acc $\uparrow$
Qwen3-VL prompt-only	10.855	10.927	0.000
ThinkJEPA	0.061	0.056	0.596

Appendix

Table 12: Pure prompt-only VLM baseline. We directly prompt Qwen3-VL (Thinking) to predict future trajectories from video and text, without any task-specific fine-tuning or trained prediction head. The large performance gap to ThinkJEPA indicates that zero-shot prompting is not sufficient for fine-grained metric-space trajectory forecasting. This study is included as a weak reference point only; the VLM-only baseline in the main paper is a substantially fairer comparison because it is trained with the same task-specific supervision and downstream head.

Hyperparameter	Value
Input frames ( $T$ )	64
Past/Future split ( $T_{p}/T_{f}$ )	32/32
Input resolution	$256{\times}256$
Backbone	V-JEPA-L ( vit_large_rope )
Backbone depth / dim	24 / 1024
Patch embedding	Conv3d kernel/stride $(2,16,16)$

Appendix

Table 13: Key architectural hyperparameters and tensor dimensions.

Explore similar work

V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
Sep 29, 2026Yang Zhang, Jiangyuan Zhao, Chenyou Fan +4Visual Representation Learning Visuomotor Policy Learning
Tsinghua University · Shanghai Jiao Tong University · Fudan University +4
JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling
Aug 10, 2026Yihan Lin, Jiawei He, Shifeng Bao +6Joint-Embedding Predictive Architecture Robotic Manipulation
School of Information, Renmin University of China, Beijing, China · XYZ Embodied AI, Beijing, China · Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China +3
Chain of World: World Model Thinking in Latent Motion
Mar 3, 2026Fuxiang Yang, Donglin Di, Lulu Tang +6Vision-Language-Action Models Reasoning for VLA Models
Harbin Institute of Technology · Li Auto · Beijing Academy of Artificial Intelligence (BAAI) +3

Sep 29, 2026J/K move · Enter open · S save

V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents

Yang Zhang, Jiangyuan Zhao, Chenyou Fan, Jiayu Hu+3

Tsinghua University · Shanghai Jiao Tong University · Fudan University+4

Visual Representation Learning Visuomotor Policy Learning Robotic Manipulation World Action Models

Open paper AskPDF arXiv

World-action models (WAMs) couple future visual-state prediction with action generation. By adapting video generators or image-editing models pretrained at scale, a prominent line of recent WAMs inherits both predictive knowledge and the models in which it was learned. We ask whether a predictive visual latent space induced by large-scale predictive pretraining can instead provide a sufficient foundation for effective WAM learning without inheriting a complete pretrained visual generative model. To answer this question, we introduce V-JEPA Policy, a simple framework that builds a WAM on the latent space of a frozen V-JEPA 2.1 encoder. An instruction-conditioned future-latent predictor and a flow-matching action expert are jointly learned from scratch in a single downstream stage, with the predictor's future-informed context key--value states conditioning action generation. With 0.9B total parameters, of which 0.6B are trainable, V-JEPA Policy achieves competitive performance with representative WAM and vision-language-action baselines across LIBERO, LIBERO-Plus, and RoboCasa-GR1. Comparing visual foundations under the same downstream framework and training budget identifies V-JEPA latents as more effective than the discriminative, reconstructive, and video-understanding-oriented alternatives, particularly under distribution shifts. Beyond task-specific learning, pretraining the predictor on DROID video--instruction pairs without action labels and adapting it into a WAM yields substantial gains in downstream control and out-of-distribution generalization. Together, these findings establish predictive visual latents as a foundation for effective WAM learning from task-specific demonstrations and for transferring future-modeling knowledge acquired from broader in-the-wild videos. Project page: https://breez3young.github.io/vjepa-policy/

Abstract

Explore similar work

V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents

JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling

Chain of World: World Model Thinking in Latent Motion