Video provides a natural source of supervision: what happens next serves as a learning signal for visual world modeling. Yet masked video pretraining focuses on recovering missing content from surrounding context rather than predicting the future from the past. Autoregressive methods instead predict forward one token at a time, but have yet to match joint-embedding methods in representation quality. This gap motivates us to revisit the granularity and space of future prediction. In this work, we introduce VANE, a self-supervised visual pretraining framework built on next-frame prediction in embedding space. A causal encoder summarizes past observations for a state predictor that infers the representation of the entire next frame, making temporal evolution the primary learning signal. This prediction is carried out in a fixed embedding space provided by a frozen semantic tokenizer, emphasizing scene content and structure. To retain fine-grained spatial detail alongside semantic content, a lightweight auxiliary generative decoder reconstructs future-frame patches conditioned on the predicted representations. To unify visual modeling across images and videos, we construct a local-to-global crop sequence for still images, allowing the same encoder to predict newly revealed regions from earlier crops under the same objective. Experiments show that VANE improves with model and data scaling, achieving SOTA on most anticipation, recognition, and dense perception benchmarks among unified encoders. Especially on EPIC-KITCHENS-100 action anticipation, it surpasses V-JEPA 2.1 by 7.1 points in mean-class recall@5. These results demonstrate that next-frame prediction in embedding space is effective and scalable for visual world modeling.
Figures & tables
Figure 1: From masked completion to next-frame prediction in embedding space. (a) V-JEPA masks part of a clip, encodes the visible patches with bidirectional attention, and predicts the embeddings of the masked regions, supervised by an EMA encoder of the full video. (b) VANE encodes previous frames with frame-causal attention and predicts the embedding of the entire next frame from them, with targets from a frozen semantic tokenizer. An auxiliary generative decoder, used only during pretraining, regularizes the predicted embedding to retain fine-grained spatial detail.
Figure 2: Overview of VANE. VANE predicts the embedding of the next frame from the frames observed so far. Top: the causal encoder maps the observed frames x1,…,xt to context states h1,…,ht , which are concatenated with prediction tokens pt+1 ; the state predictor uses the context states only as keys and values and outputs the predicted state st+1 . Middle: a projector maps st+1 to the embedding space of the frozen semantic tokenizer, where it is supervised by the embedding of the next frame xt+1 with an L1 loss; the generative decoder, conditioned on st+1 , predicts the flow velocity v^t+1 from the noisy next frame x~t+1 . Bottom: images are presented as local-to-global crops and videos as frame sequences, so both share the same objective.
Figure 3: State predictor attention. Context states supply only keys and values; prediction tokens attend within their own target frame.
Method
Param.
Verb
Noun
Action
InAViT ( Roy et al., 2024 )
160M
51.9
52.0
25.8
Video-LLaMA ( Zhang et al., 2023 )
7B
52.9
52.0
26.0
PlausiVL ( Mittal et al., 2024 )
8B
55.6
54.2
27.6
V-JEPA 2 ViT-g ( Assran et al., 2025 )
1B
63.6
57.1
39.7
V-JEPA 2.1 ViT-g ( Mur-Labadia et al., 2026 )
1B
63.6
56.2
38.4
VANE (ours)
1B
65.7
67.6
45.5
Table 1: Action anticipation on EPIC-KITCHENS-100. Mean-class recall@5 (%) for verb, noun, and action on the validation set, with a one-second anticipation gap. Bold denotes the best result.
Model
AP b
AP b+N
AP b+V
AP b+δ
AP b+N+V
AP b+δ+N
AP b+δ+V
AP All
mAP N
mAP N+V
mAP N+δ
mAP All
Results reported in the literature
StillFast ( Ragusa et al., 2023 )
—
—
—
—
—
—
—
—
20.2
10.3
7.17
3.96
STAformer ( Mur-Labadia et al., 2024 )
38.3
28.3
16.6
12.3
12.8
8.89
5.47
4.06
29.4
15.3
9.94
5.67
Evaluated using the same protocol
DINOv2 ViT-g ( Oquab et al., 2024 )
47.8
38.1
23.6
14.1
19.1
10.9
7.29
5.73
30.6
16.8
9.37
5.44
DINOv3 ViT-H+ ( Siméoni et al., 2025 )
48.9
40.2
23.4
13.6
19.4
10.8
6.71
5.45
32.4
17.2
9.53
5.20
Table 2: Short-term object-interaction anticipation on Ego4D STA v2. The model localizes the next-active object and predicts its noun, the interaction verb, and the time to contact. Each column reports Top-5 AP or mAP counting a prediction as correct only if all listed components are correct: box ( b ), noun ( N ), verb ( V ), and time to contact ( δ ). Bold denotes the best results.
Method
Param.
Appearance Understanding
Motion Understanding
Avg.
IN1K
K400
SSv2
Diving-48
Results reported in the literature
Toto ( Rajasegaran et al., 2025 )
1B
75.3
74.4
–
–
–
VideoMAEv2 ( Wang et al., 2023a )
1B
71.4
82.8
56.1
–
–
InternVideo2-1B ( Wang et al., 2024 )
1B
–
87.9
67.3
–
–
VideoPrism ( Zhao et al., 2024 )
1B
–
87.6
68.5
71.3
–
Table 3: Global classification. Top-1 accuracy (%) of attentive probes on frozen features, covering appearance understanding (IN1K, K400) and motion understanding (SSv2, Diving-48). Avg. is reported for models evaluated on all four tasks. Bold denotes the best result in image–video encoders.
Figure 4: Qualitative comparison on ADE20K. V-JEPA 2.1 and VANE with the same dense linear probe on frozen features.
Method
Param./patch
Segmentation
Depth
ADE20K
Citysc.
VOC
NYUv2 ↓
KITTI ↓
Image encoders evaluated under the same protocol
PEcore ( Bolya et al., 2025 )
2B/14
38.9
61.1
69.2
0.590
4.119
PEspatial ( Bolya et al., 2025 )
2B/14
49.3
73.2
82.7
0.362
3.082
AM-RADIOv2.5 ( Ranzinger et al., 2024 )
1B/14
53.0
78.4
85.4
0.340
2.918
SigLIP 2 ( Tschannen et al., 2025 )
1B/16
42.7
64.8
72.7
0.494
3.273
Table 4: Dense perception. Semantic segmentation (mIoU, %) and monocular depth estimation (RMSE, lower is better) with a linear probe on frozen final-layer patch features. Param./patch denotes encoder size/patch size. Bold denotes the best result among image–video encoders.
Target
Prediction
IN1K
SSv2
ADE20K
Reference
Online EMA
Next frame
81.0
67.5
35.3
Pixels
Next frame
75.8
56.1
24.1
Semantic tokenizers
VA-VAE
Next frame
81.8
68.0
37.1
SigLIP 2
Next frame
77.5
65.5
29.6
Table 5: Ablation studies. Top-1 accuracy on IN1K/SSv2 and mIoU on ADE20K with a ViT-L encoder trained for 50K iterations. (a) Prediction space and granularity; next-frame and next-token variants share the frozen DINOv2 ViT-L target space. (b) Image objectives, with the video objective, tokenizer, and generative decoder fixed. (c) Auxiliary generative decoder.
Figure 5: Cumulative training recipe.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
ViT-L
ViT-g
Causal Encoder
Block
ViT Block
Depth
24
40
Width
1024
1408
Num. Heads
16
22
MLP Ratio
4.0
4.36
Appendix
Table 6: Model architectures. The main model uses a ViT-g causal encoder, a 12-layer context-isolated state predictor, a frozen DINOv2 ViT-L semantic tokenizer, and a six-layer patch-isolated DiffMLP decoder.
Figure 6: Attention masks. Left: frame-causal attention in the causal encoder, bidirectional within each frame and unidirectional across frames. Right: context-isolated causal attention in the state predictor. Prediction tokens for frame t+1 read the context states of frames up to t , which supply only keys and values, and attend to each other only within the same target frame.
Regime
Modality
Training data
Homogeneous ( 10% )
Image
IN1K
Video
SSv2
Heterogeneous ( 90% )
Image
VISTA-I
Video
VISTA-V
Appendix
Table 7: Pretraining data mixture. VISTA-I and VISTA-V denote the image and video subsets of our unified VISTA corpus. Each combines public seed datasets, distribution-matched retrieval, and class-balanced data as described in Sec. B.1 .
Figure 7: Visualization of local-to-global next-crop prediction. Each row shows the spatial layout of three sampled crops and the corresponding prediction sequence Crop1→Crop2→Crop3 . The first two observations are local crops and the last provides broader scene context. At each step, supervision is restricted to spatial regions newly revealed by the next crop. Crops are resized to a common display size for visualization only; their original spatial coordinates are used during training.
Figure 8: Additional examples of local-to-global next-crop prediction. We show additional sampled crop sequences following the same local-to-global sampling strategy as in Fig. 7 . Supervision is applied only to regions newly revealed by each subsequent crop. Crops are resized to a common display size for visualization only; their original spatial coordinates are used during training.
Warm-up
Base Pretrain
Long-Video
High-Res.
Post-train
Cooldown
Input Configuration
Video Frames
16
16
64
64
Video Sampling Rate
4 FPS
Video Tubelet Size
2 frames
Resolution
2562
2562
2562
3842
Appendix
Table 8: Multi-stage training strategy. The base model is first trained with 16-frame clips at 256×256 , followed by long-video post-training with 64-frame clips and a final high-resolution stage.
Figure 9: Qualitative short-term anticipation results on Ego4D STA v2. Each example pairs the top-scoring predictions of VANE (left) with the ground truth (right). Each box marks a next-active object and is labeled with the interaction verb, the object noun, and the time to contact (TTC). Predictions are produced by a probe trained on frozen VANE features.
Figure 10: Qualitative semantic segmentation results on ADE20K. Each example shows the input image, ground truth, and predictions from dense probes trained on frozen features. All compared encoders use the same downstream protocol.
Figure 11: Qualitative semantic segmentation results on Cityscapes. Each example shows the input image, ground truth, and the prediction from a dense linear probe trained on frozen VANE features.
Figure 12: Qualitative monocular depth estimation on NYUv2. The error column shows Δe=d^VANE−dgt−d^V-JEPA 2.1−dgt ; blue favors VANE, while red favors V-JEPA 2.1.
Figure 13: Qualitative monocular depth estimation results on KITTI. Predicted depth maps are obtained from dense readouts trained on frozen visual features using the same protocol as the quantitative evaluation.
Method
Objective
Target
IN1K
SSv2
ADE20K
V-JEPA 2.1
Masked
EMA
81.8
72.0
38.4
V-JEPA 2.1
Masked
Frozen DINOv2
82.7
70.6
41.5
VANE
Next frame
Frozen DINOv2
84.5
72.3
45.1
Appendix
Table 9: Masked versus next-frame prediction under matched pretraining. All models use ViT-L encoders trained on VISTA for 50K iterations. The frozen-target V-JEPA 2.1 variant uses the same DINOv2 ViT-L semantic tokenizer as VANE. We report top-1 accuracy (%) on IN1K and SSv2 and mIoU (%) on ADE20K.
Number of Crops K
IN1K
SSv2
ADE20K
2
84.3
72.1
44.4
3 (default)
84.5
72.3
45.1
4
84.4
71.9
44.7
Appendix
Table 10: Ablation of the number of image crops. We vary the number of local-to-global views K while keeping the remaining training configuration fixed.
Predictor Attention
IN1K
SSv2
ADE20K
Causal self-attention
81.2
66.8
35.5
Context-isolated cross-attention
84.5
72.3
45.1
Appendix
Table 11: Ablation of state-predictor attention. Context-isolated prediction treats historical encoder states as read-only context rather than jointly updating them with prediction tokens inside the predictor.
We present V-JEPA 2.1, a family of self-supervised models that learn dense, high-quality visual representations for both images and videos while retaining strong global scene understanding. The approach combines four key components. First, a dense predictive loss uses a masking-based objective in which both visible and masked tokens contribute to the training signal, encouraging explicit spatial and temporal grounding. Second, deep self-supervision applies the self-supervised objective hierarchically across multiple intermediate encoder layers to improve representation quality. Third, multi-modal tokenizers enable unified training across images and videos. Finally, the model benefits from effective scaling in both model capacity and training data. Together, these design choices produce representations that are spatially structured, semantically coherent, and temporally consistent. Empirically, V-JEPA 2.1 achieves state-of-the-art performance on several challenging benchmarks, including 7.71 mAP on Ego4D for short-term object-interaction anticipation and 40.8 Recall@5 on EPIC-KITCHENS for high-level action anticipation, as well as a 20-point improvement in real-robot grasping success rate over V-JEPA-2 AC. The model also demonstrates strong performance in robotic navigation (5.687 ATE on TartanDrive), depth estimation (0.307 RMSE on NYUv2 with a linear probe), and global recognition (77.7 on Something-Something-V2). These results show that V-JEPA 2.1 significantly advances the state of the art in dense visual understanding and world modeling.
Video world models are increasingly used to provide predictive visual representations, yet it remains unclear which pretraining signals induce action-relevant structure in their latent spaces. We study this question through a unified probe-based evaluation across diverse encoder families, including image-only self-supervision, video pretraining with and without latent prediction, reconstruction-based autoencoders, diffusion models, and shortcut-forcing dynamics models. Using a common inverse-dynamics probing objective, we find that action-relevant structure is driven primarily by temporal video pretraining rather than pixel reconstruction fidelity: models with strong pixel decoding quality can exhibit near-zero action recoverability, while video-pretrained self-supervised encoders consistently achieve the best Pareto trade-off between visual fidelity and action prediction. Comparing V-JEPA and VideoMAE further shows that most gains arise from natural-video temporal context, with feature-level latent prediction providing a smaller additional benefit. These trends transfer across robotic benchmarks, though CALVIN reveals that static-environment tasks can partially mask the importance of temporal structure by allowing strong image priors to suffice. Finally, inverse-dynamics supervision substantially improves robustness to visual corruption, suggesting that action-aware objectives regularize latent geometry beyond clean-setting performance. Our results identify temporal predictive structure -- not reconstruction fidelity -- as the primary ingredient underlying action-relevant video representations.
Jewon Yeom, Hanseul Kim, Jeongjae Park +3
Graduate School of Data Science, Seoul National University
Video event prediction (VEP) requires models to infer unobserved future states from partial video evidence. Existing video MLLMs usually verbalize intermediate future reasoning in text space: once visual evidence is verbalized, fine-grained motion, geometry, and interaction cues can be lost, leading to plausible but visually ungrounded hallucinations. We introduce Future-L1, an interleaved latent visual reasoning framework that lets an MLLM alternate between language tokens and continuous latent visual spans during autoregressive decoding. To train this capability, we construct Future-L1-50K by selecting examples where future visual hints help prediction and align latent states to future-frame embeddings, then further optimize sampled latent trajectories with LA-DAPO, a latent-aware RL objective with outcome-contrastive and temporal-diversity rewards. Future-L1 achieves new state-of-the-art results on both benchmarks: on FutureBench, it improves Qwen3-VL-8B from 61.0 to 85.4 and exceeds the previous best Video-CoE by 10.4 points; on TwiFF-Bench, it improves the average score from 2.44 to 3.04. These results suggest that future-oriented video reasoning benefits from preserving intermediate visual semantics in latent space rather than translating every reasoning step into text.
Tianxiang Jiang, Linquan Wu, Sheng Xia +5
University of Science and Technology of China · Shanghai AI Laboratory · City University of Hong Kong +4