ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model
Authors: Haichao Zhang, Yijiang Li, Shwai He, Tushar Nagarajan, Mingfei Chen, Jianglin Lu, Ang Li, Yun Fu
Organizations: Northeastern University · University of California San Diego · University of Maryland · The University of Texas at Austin · University of Washington
Recent progress in latent world models (e.g., V-JEPA2) has shown promising capability in forecasting future world states from video observations. Nevertheless, dense prediction from a short observation window limits temporal context and can bias predictors toward local, low-level extrapolation, making it difficult to capture long-horizon semantics and reducing downstream utility. Vision--language models (VLMs), in contrast, provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames, but they are not ideal as standalone dense predictors due to compute-driven sparse sampling, a language-output bottleneck that compresses fine-grained interaction states into text-oriented representations, and a data-regime mismatch when adapting to small action-conditioned datasets. We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance via a dual-temporal pathway: a dense JEPA branch for fine-grained motion and interaction cues, and a uniformly sampled VLM thinker branch with a larger temporal stride for knowledge-rich guidance. To transfer the VLM's progressive reasoning signals effectively, we introduce a hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction. Experiments on hand-manipulation trajectory prediction show that our method outperforms both a strong VLM-only baseline and a JEPA-predictor baseline, and yields more robust long-horizon rollout behavior.
Figures & tables
Figure 1: Overall Architecture of ThinkJEPA. ThinkJEPA couples a dense JEPA branch for fine-grained latent dynamics modeling with a uniformly sampled VLM- thinker branch that provides long-horizon semantic guidance. The VLM guidance—including visual tokens from the ViT visual tokenizer and intermediate hidden states from the language model—is distilled by a pyramidal representation extraction module and injected into the V-JEPA predictor via layer-wise modulation. Concretely, guidance derived from language-model layers {L0,…,LN} is mapped to modulation parameters for predictor layers {T0,…,TK} . The predicted future latents are concatenated with past teacher latents to form the full latent sequence, which is then fed into a task head to produce downstream trajectory predictions.
Dataset
Model
ADE ↓
FDE ↓
Acc ↑
FD ↓
SL1 ↓
CD ↓
EgoDex
Qwen3-VL Thinking
0.142
0.144
0.084
99.538
1.656
0.615
V-JEPA Predictor
0.071
0.066
0.471
74.223
1.252
0.317
ThinkJEPA
0.061
0.056
0.596
74.032
1.248
0.315
EgoExo4D
Qwen3-VL Thinking
0.661
0.690
0.038
104.548
1.756
0.690
V-JEPA Predictor
0.659
0.636
0.074
89.244
1.520
0.469
ThinkJEPA
0.622
0.597
0.171
79.654
1.364
0.359
Table 1: Quantitative comparison across datasets. We report trajectory metrics (ADE/FDE/Acc) and latent forecasting metrics (FD/SL1/CD). FD/SL1/CD denote V-JEPA feature distance, latent SmoothL1, and latent cosine distance, respectively. All values are reported with three decimal places.
Group
Model
ADE ↓
FDE ↓
Trajectory Baselines
Decoder-only + Behavior Cloning
0.0767
0.0818
Decoder-only + DDPM
0.1148
0.1238
Decoder-only + Flow Matching
0.1527
0.1574
Encoder-decoder + Behavior Cloning
0.0774
0.0924
Encoder-decoder + DDPM
0.1272
0.1245
Encoder-decoder + Flow Matching
0.1736
0.1557
Table 2: Comparison with EgoDex trajectory prediction baselines on EgoDex. We compare ThinkJEPA against the trajectory prediction baselines reported in EgoDex, including decoder-only and encoder-decoder architectures with Behavior Cloning, DDPM, and Flow Matching. ThinkJEPA achieves the best ADE/FDE among all compared methods.
Table 4
Abl.
ADE ↓
FDE ↓
Acc ↑
FD ↓
SL1 ↓
CD ↓
Encoder+V-JEPA predictor
0.128
0.129
0.100
78.869
1.340
0.360
Encoder-only
0.143
0.145
0.086
102.910
1.700
0.615
AR+V-JEPA predictor
0.128
0.130
0.098
78.514
1.333
0.356
AR-only
0.142
0.144
0.086
102.910
1.700
0.615
No-dual-temporal sampling
0.128
0.130
0.099
78.862
1.340
0.360
No-Th
0.071
0.066
0.471
74.223
1.252
0.317
Table 5: Ablation studies. We vary the VLM token sources and the thinker module. Encoder denotes encoding tokens, AR denotes autoregressive tokens; No-dual disables the dual-temporal VLM pathway, and No-Th removes the thinker module. We abbreviate latent metrics as FD (feature distance), SL1 (SmoothL1), and CD (cosine distance).
Variant
ADE ↓
FDE ↓
Acc ↑
FD ↓
SL1 ↓
CD ↓
Last-layer
0.128
0.130
0.099
78.858
1.340
0.360
Mid-layer
0.128
0.131
0.098
78.517
1.333
0.356
All layers (ThinkJEPA)
0.061
0.056
0.596
74.747
1.263
0.324
Table 6: VLM layer selection on EgoDex. We compare guidance derived from different VLM layer selections. FD denotes V-JEPA feature distance, SL1 denotes latent SmoothL1, and CD denotes latent cosine distance.
Model
A@4
A@8
A@16
A@32
F@4
F@8
F@16
F@32
Qwen3-VL Thinking
0.140
0.819
1.375
1.026
0.143
2.850
0.286
1.092
V-JEPA Predictor
0.121
0.126
0.134
0.142
0.124
0.136
0.149
0.153
ThinkJEPA
0.071
0.078
0.092
0.111
0.073
0.090
0.118
0.136
Table 7: Recursive rollout on EgoDex: trajectory error vs. horizon. We perform autoregressive rollout for horizons H∈{4,8,16,32} . A@H and F@H denote ADE@H and FDE@H, respectively; the lower, the better.
Figure 2: Qualitative results. Predicted future hand-manipulation trajectories visualized as heat maps overlaid on the reference frame. Colors indicate temporal progression from blue (earlier) to red (later). Ideally, trajectories transition smoothly from blue to red, indicating coherent motion over time. ThinkJEPA produces smoother trajectories with better temporal consistency and joint alignment.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
ADE ↓
FDE ↓
Acc ↑
FD ↓
SL1 ↓
CD ↓
Prompt + video → VLM condition
0.069
0.062
0.495
74.007
1.248
0.315
ThinkJEPA
0.061
0.056
0.596
74.747
1.263
0.324
Appendix
Table 8: Prompt + video to VLM-conditioned features. The predictor uses cached visual features as the main trajectory backbone and VLM-derived features as external conditioning.
Table 10: Conditioning mechanism ablation. We compare FiLM, cross-attention, and AdaLN under the same training setup.
Variant
ADE ↓
FDE ↓
Acc ↑
Direct visual conditioning
0.071
0.066
0.475
Drop deepstack tokens
0.072
0.066
0.464
ThinkJEPA
0.061
0.056
0.596
Appendix
Table 11: Direct visual conditioning vs. deepstack-token removal. Both ablations remain competitive, while ThinkJEPA achieves the strongest downstream trajectory performance.
Baseline
ADE ↓
FDE ↓
Acc ↑
Qwen3-VL prompt-only
10.855
10.927
0.000
ThinkJEPA
0.061
0.056
0.596
Appendix
Table 12: Pure prompt-only VLM baseline. We directly prompt Qwen3-VL (Thinking) to predict future trajectories from video and text, without any task-specific fine-tuning or trained prediction head. The large performance gap to ThinkJEPA indicates that zero-shot prompting is not sufficient for fine-grained metric-space trajectory forecasting. This study is included as a weak reference point only; the VLM-only baseline in the main paper is a substantially fairer comparison because it is trained with the same task-specific supervision and downstream head.
Hyperparameter
Value
Input frames ( T )
64
Past/Future split ( Tp/Tf )
32/32
Input resolution
256×256
Backbone
V-JEPA-L ( vit_large_rope )
Backbone depth / dim
24 / 1024
Patch embedding
Conv3d kernel/stride (2,16,16)
Appendix
Table 13: Key architectural hyperparameters and tensor dimensions.
World-action models (WAMs) couple future visual-state prediction with action generation. By adapting video generators or image-editing models pretrained at scale, a prominent line of recent WAMs inherits both predictive knowledge and the models in which it was learned. We ask whether a predictive visual latent space induced by large-scale predictive pretraining can instead provide a sufficient foundation for effective WAM learning without inheriting a complete pretrained visual generative model. To answer this question, we introduce V-JEPA Policy, a simple framework that builds a WAM on the latent space of a frozen V-JEPA 2.1 encoder. An instruction-conditioned future-latent predictor and a flow-matching action expert are jointly learned from scratch in a single downstream stage, with the predictor's future-informed context key--value states conditioning action generation. With 0.9B total parameters, of which 0.6B are trainable, V-JEPA Policy achieves competitive performance with representative WAM and vision-language-action baselines across LIBERO, LIBERO-Plus, and RoboCasa-GR1. Comparing visual foundations under the same downstream framework and training budget identifies V-JEPA latents as more effective than the discriminative, reconstructive, and video-understanding-oriented alternatives, particularly under distribution shifts. Beyond task-specific learning, pretraining the predictor on DROID video--instruction pairs without action labels and adapting it into a WAM yields substantial gains in downstream control and out-of-distribution generalization. Together, these findings establish predictive visual latents as a foundation for effective WAM learning from task-specific demonstrations and for transferring future-modeling knowledge acquired from broader in-the-wild videos. Project page: https://breez3young.github.io/vjepa-policy/
Yang Zhang, Jiangyuan Zhao, Chenyou Fan +4
Tsinghua University · Shanghai Jiao Tong University · Fudan University +4
Robust robot control benefits from explicitly modeling state transitions, but video-generation world action models (WAMs) introduce substantial deployment cost. Existing latent WAMs avoid explicit future generation, but often compress predictive representations or separate predictive modeling from the representations used for action generation. We introduce JEPA-WAM, a latent WAM built in a pretrained V-JEPA space, which couples latent transition prediction with continuous action generation through a shared predictor. JEPA-WAM predicts a spatially structured joint current-future target that captures task-shared visual temporal structure between current and future observations, while preserving dense patch-level correspondence. Through the shared predictor, transition supervision directly shapes the backbone, from which dedicated representations are extracted for action prediction. The same design can also be instantiated in pretrained VLA policies while preserving their original perception and action pathways. On LIBERO-Plus, JEPA-WAM achieves 79.2%, the best result without large-scale robot-policy pretraining, while its pretrained π0.5 instantiation reaches 86.3%, achieving the best overall performance. Experiments on RoboTwin 2.0 and real-world bimanual manipulation further demonstrate strong generalization under visual and spatial shifts.
Yihan Lin, Jiawei He, Shifeng Bao +6
School of Information, Renmin University of China, Beijing, China · XYZ Embodied AI, Beijing, China · Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China +3
Vision-Language-Action (VLA) models are a promising path toward embodied intelligence, yet they often overlook the predictive and temporal-causal structure underlying visual dynamics. World-model VLAs address this by predicting future frames, but waste capacity reconstructing redundant backgrounds. Latent-action VLAs encode frame-to-frame transitions compactly, but lack temporally continuous dynamic modeling and world knowledge. To overcome these limitations, we introduce CoWVLA (Chain-of-World VLA), a new "Chain of World" paradigm that unifies world-model temporal reasoning with a disentangled latent motion representation. First, a pretrained video VAE serves as a latent motion extractor, explicitly factorizing video segments into structure and motion latents. Then, during pre-training, the VLA learns from an instruction and an initial frame to infer a continuous latent motion chain and predict the segment's terminal frame. Finally, during co-fine-tuning, this latent dynamic is aligned with discrete action prediction by jointly modeling sparse keyframes and action sequences in a unified autoregressive decoder. This design preserves the world-model benefits of temporal reasoning and world knowledge while retaining the compactness and interpretability of latent actions, enabling efficient visuomotor learning. Extensive experiments on robotic simulation benchmarks show that CoWVLA outperforms existing world-model and latent-action approaches and achieves moderate computational efficiency, highlighting its potential as a more effective VLA pretraining paradigm. The project website can be found at https://fx-hit.github.io/cowvla-io.
Fuxiang Yang, Donglin Di, Lulu Tang +6
Harbin Institute of Technology · Li Auto · Beijing Academy of Artificial Intelligence (BAAI) +3