Learning from Next-Frame Prediction: Towards Unified and Scalable Visual World Modeling

Yifeng Yao; Jinghan Li; Zhihao Zhang; Yajiao Xiong; Yang Jin; Kun Xu; Huishuai Zhang; Dongyan Zhao

cs.CVOct 22, 2026

Learning from Next-Frame Prediction: Towards Unified and Scalable Visual World Modeling

Authors: Yifeng Yao, Jinghan Li, Zhihao Zhang, Yajiao Xiong, Yang Jin, Kun Xu, Huishuai Zhang, Dongyan Zhao

Organizations: Peking University · ByteDance · UNC-Chapel Hill

Abstract

Video provides a natural source of supervision: what happens next serves as a learning signal for visual world modeling. Yet masked video pretraining focuses on recovering missing content from surrounding context rather than predicting the future from the past. Autoregressive methods instead predict forward one token at a time, but have yet to match joint-embedding methods in representation quality. This gap motivates us to revisit the granularity and space of future prediction. In this work, we introduce VANE, a self-supervised visual pretraining framework built on next-frame prediction in embedding space. A causal encoder summarizes past observations for a state predictor that infers the representation of the entire next frame, making temporal evolution the primary learning signal. This prediction is carried out in a fixed embedding space provided by a frozen semantic tokenizer, emphasizing scene content and structure. To retain fine-grained spatial detail alongside semantic content, a lightweight auxiliary generative decoder reconstructs future-frame patches conditioned on the predicted representations. To unify visual modeling across images and videos, we construct a local-to-global crop sequence for still images, allowing the same encoder to predict newly revealed regions from earlier crops under the same objective. Experiments show that VANE improves with model and data scaling, achieving SOTA on most anticipation, recognition, and dense perception benchmarks among unified encoders. Especially on EPIC-KITCHENS-100 action anticipation, it surpasses V-JEPA 2.1 by 7.1 points in mean-class recall@5. These results demonstrate that next-frame prediction in embedding space is effective and scalable for visual world modeling.

Figures & tables

Figure 1: From masked completion to next-frame prediction in embedding space. (a) V-JEPA masks part of a clip, encodes the visible patches with bidirectional attention, and predicts the embeddings of the masked regions, supervised by an EMA encoder of the full video. (b) VANE encodes previous frames with frame-causal attention and predicts the embedding of the entire next frame from them, with targets from a frozen semantic tokenizer. An auxiliary generative decoder, used only during pretraining, regularizes the predicted embedding to retain fine-grained spatial detail.

Figure 2: Overview of VANE. VANE predicts the embedding of the next frame from the frames observed so far. Top: the causal encoder maps the observed frames $x_{1},\ldots,x_{t}$ to context states $h_{1},\ldots,h_{t}$ , which are concatenated with prediction tokens $p_{t+1}$ ; the state predictor uses the context states only as keys and values and outputs the predicted state $s_{t+1}$ . Middle: a projector maps $s_{t+1}$ to the embedding space of the frozen semantic tokenizer, where it is supervised by the embedding of the next frame $x_{t+1}$ with an $L_{1}$ loss; the generative decoder, conditioned on $s_{t+1}$ , predicts the flow velocity $\hat{v}_{t+1}$ from the noisy next frame $\tilde{x}_{t+1}$ . Bottom: images are presented as local-to-global crops and videos as frame sequences, so both share the same objective.

Figure 3: State predictor attention. Context states supply only keys and values; prediction tokens attend within their own target frame.

Method	Param.	Verb	Noun	Action
InAViT ( Roy et al., 2024 )	160M	51.9	52.0	25.8
Video-LLaMA ( Zhang et al., 2023 )	7B	52.9	52.0	26.0
PlausiVL ( Mittal et al., 2024 )	8B	55.6	54.2	27.6
V-JEPA 2 ViT-g ( Assran et al., 2025 )	1B	63.6	57.1	39.7
V-JEPA 2.1 ViT-g ( Mur-Labadia et al., 2026 )	1B	63.6	56.2	38.4
VANE (ours)	1B	65.7	67.6	45.5

Table 1: Action anticipation on EPIC-KITCHENS-100. Mean-class recall@5 (%) for verb, noun, and action on the validation set, with a one-second anticipation gap. Bold denotes the best result.

Model	AP $b$	AP $b{+}N$	AP $b{+}V$	AP $b{+}\delta$	AP $b{+}N{+}V$	AP $b{+}\delta{+}N$	AP $b{+}\delta{+}V$	AP All	mAP $N$	mAP $N{+}V$	mAP $N{+}\delta$	mAP All
Results reported in the literature
StillFast ( Ragusa et al., 2023 )	—	—	—	—	—	—	—	—	20.2	10.3	7.17	3.96
STAformer ( Mur-Labadia et al., 2024 )	38.3	28.3	16.6	12.3	12.8	8.89	5.47	4.06	29.4	15.3	9.94	5.67
Evaluated using the same protocol
DINOv2 ViT-g ( Oquab et al., 2024 )	47.8	38.1	23.6	14.1	19.1	10.9	7.29	5.73	30.6	16.8	9.37	5.44
DINOv3 ViT-H+ ( Siméoni et al., 2025 )	48.9	40.2	23.4	13.6	19.4	10.8	6.71	5.45	32.4	17.2	9.53	5.20

Table 2: Short-term object-interaction anticipation on Ego4D STA v2. The model localizes the next-active object and predicts its noun, the interaction verb, and the time to contact. Each column reports Top-5 AP or mAP counting a prediction as correct only if all listed components are correct: box (

b

), noun (

N

), verb (

V

), and time to contact (

\delta

). Bold denotes the best results.

Method	Param.	Appearance Understanding		Motion Understanding		Avg.
		IN1K	K400	SSv2	Diving-48
Results reported in the literature
Toto ( Rajasegaran et al., 2025 )	1B	75.3	74.4	–	–	–
VideoMAEv2 ( Wang et al., 2023a )	1B	71.4	82.8	56.1	–	–
InternVideo2-1B ( Wang et al., 2024 )	1B	–	87.9	67.3	–	–
VideoPrism ( Zhao et al., 2024 )	1B	–	87.6	68.5	71.3	–

Table 3: Global classification. Top-1 accuracy (%) of attentive probes on frozen features, covering appearance understanding (IN1K, K400) and motion understanding (SSv2, Diving-48). Avg. is reported for models evaluated on all four tasks. Bold denotes the best result in image–video encoders.

Figure 4: Qualitative comparison on ADE20K. V-JEPA 2.1 and VANE with the same dense linear probe on frozen features.

Method	Param./patch	Segmentation			Depth
		ADE20K	Citysc.	VOC	NYUv2 $\downarrow$	KITTI $\downarrow$
Image encoders evaluated under the same protocol
$\mathrm{PE}_{\mathrm{core}}$ ( Bolya et al., 2025 )	2B/14	38.9	61.1	69.2	0.590	4.119
$\mathrm{PE}_{\mathrm{spatial}}$ ( Bolya et al., 2025 )	2B/14	49.3	73.2	82.7	0.362	3.082
AM-RADIOv2.5 ( Ranzinger et al., 2024 )	1B/14	53.0	78.4	85.4	0.340	2.918
SigLIP 2 ( Tschannen et al., 2025 )	1B/16	42.7	64.8	72.7	0.494	3.273

Table 4: Dense perception. Semantic segmentation (mIoU, %) and monocular depth estimation (RMSE, lower is better) with a linear probe on frozen final-layer patch features. Param./patch denotes encoder size/patch size. Bold denotes the best result among image–video encoders.

Target	Prediction	IN1K	SSv2	ADE20K
Reference
Online EMA	Next frame	81.0	67.5	35.3
Pixels	Next frame	75.8	56.1	24.1
Semantic tokenizers
VA-VAE	Next frame	81.8	68.0	37.1
SigLIP 2	Next frame	77.5	65.5	29.6

Table 5: Ablation studies. Top-1 accuracy on IN1K/SSv2 and mIoU on ADE20K with a ViT-L encoder trained for 50K iterations. (a) Prediction space and granularity; next-frame and next-token variants share the frozen DINOv2 ViT-L target space. (b) Image objectives, with the video objective, tokenizer, and generative decoder fixed. (c) Auxiliary generative decoder.

Figure 5: Cumulative training recipe.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	ViT-L	ViT-g
Causal Encoder
Block	ViT Block
Depth	24	40
Width	1024	1408
Num. Heads	16	22
MLP Ratio	4.0	4.36

Appendix

Table 6: Model architectures. The main model uses a ViT-g causal encoder, a 12-layer context-isolated state predictor, a frozen DINOv2 ViT-L semantic tokenizer, and a six-layer patch-isolated DiffMLP decoder.

Figure 6: Attention masks. Left: frame-causal attention in the causal encoder, bidirectional within each frame and unidirectional across frames. Right: context-isolated causal attention in the state predictor. Prediction tokens for frame $t+1$ read the context states of frames up to $t$ , which supply only keys and values, and attend to each other only within the same target frame.

Regime	Modality	Training data
Homogeneous ( $10\%$ )	Image	IN1K
Homogeneous ( $10\%$ )	Video	SSv2
Heterogeneous ( $90\%$ )	Image	VISTA-I
Heterogeneous ( $90\%$ )	Video	VISTA-V

Appendix

Table 7: Pretraining data mixture. VISTA-I and VISTA-V denote the image and video subsets of our unified VISTA corpus. Each combines public seed datasets, distribution-matched retrieval, and class-balanced data as described in Sec. B.1 .

Figure 7: Visualization of local-to-global next-crop prediction. Each row shows the spatial layout of three sampled crops and the corresponding prediction sequence $\mathrm{Crop}_{1}\rightarrow\mathrm{Crop}_{2}\rightarrow\mathrm{Crop}_{3}$ . The first two observations are local crops and the last provides broader scene context. At each step, supervision is restricted to spatial regions newly revealed by the next crop. Crops are resized to a common display size for visualization only; their original spatial coordinates are used during training.

Figure 8: Additional examples of local-to-global next-crop prediction. We show additional sampled crop sequences following the same local-to-global sampling strategy as in Fig. 7 . Supervision is applied only to regions newly revealed by each subsequent crop. Crops are resized to a common display size for visualization only; their original spatial coordinates are used during training.

	Warm-up	Base Pretrain	Long-Video	High-Res.
			Post-train	Cooldown
Input Configuration
Video Frames	16	16	64	64
Video Sampling Rate	4 FPS
Video Tubelet Size	2 frames
Resolution	$256^{2}$	$256^{2}$	$256^{2}$	$384^{2}$

Appendix

Table 8: Multi-stage training strategy. The base model is first trained with 16-frame clips at

256\times 256

, followed by long-video post-training with 64-frame clips and a final high-resolution stage.

Figure 9: Qualitative short-term anticipation results on Ego4D STA v2. Each example pairs the top-scoring predictions of VANE (left) with the ground truth (right). Each box marks a next-active object and is labeled with the interaction verb, the object noun, and the time to contact (TTC). Predictions are produced by a probe trained on frozen VANE features.

Figure 10: Qualitative semantic segmentation results on ADE20K. Each example shows the input image, ground truth, and predictions from dense probes trained on frozen features. All compared encoders use the same downstream protocol.

Figure 11: Qualitative semantic segmentation results on Cityscapes. Each example shows the input image, ground truth, and the prediction from a dense linear probe trained on frozen VANE features.

Figure 12: Qualitative monocular depth estimation on NYUv2. The error column shows $\Delta e=\left|\hat{d}_{VANE{}}-d_{\mathrm{gt}}\right|-\left|\hat{d}_{\text{V-JEPA 2.1}}-d_{\mathrm{gt}}\right|$ ; blue favors VANE, while red favors V-JEPA 2.1.

Figure 13: Qualitative monocular depth estimation results on KITTI. Predicted depth maps are obtained from dense readouts trained on frozen visual features using the same protocol as the quantitative evaluation.

Method	Objective	Target	IN1K	SSv2	ADE20K
V-JEPA 2.1	Masked	EMA	81.8	72.0	38.4
V-JEPA 2.1	Masked	Frozen DINOv2	82.7	70.6	41.5
VANE	Next frame	Frozen DINOv2	84.5	72.3	45.1

Appendix

Table 9: Masked versus next-frame prediction under matched pretraining. All models use ViT-L encoders trained on VISTA for 50K iterations. The frozen-target V-JEPA 2.1 variant uses the same DINOv2 ViT-L semantic tokenizer as VANE. We report top-1 accuracy (%) on IN1K and SSv2 and mIoU (%) on ADE20K.

Number of Crops $K$	IN1K	SSv2	ADE20K
2	84.3	72.1	44.4
3 (default)	84.5	72.3	45.1
4	84.4	71.9	44.7

Appendix

Table 10: Ablation of the number of image crops. We vary the number of local-to-global views

K

while keeping the remaining training configuration fixed.

Predictor Attention	IN1K	SSv2	ADE20K
Causal self-attention	81.2	66.8	35.5
Context-isolated cross-attention	84.5	72.3	45.1

Appendix

Table 11: Ablation of state-predictor attention. Context-isolated prediction treats historical encoder states as read-only context rather than jointly updating them with prediction tokens inside the predictor.

Video Representation Learning

Video representation learning develops features that capture appearance, motion, and temporal structure for recognition, prediction, and control.

See all 143 papers

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
Mar 15, 2026Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar +6Self-Supervised Visual Representation Learning Visual Representation Learning
FAIR at Meta · Universidad de Zaragoza
What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction
Jun 5, 2026Jewon Yeom, Hanseul Kim, Jeongjae Park +3Video Representation Learning Predictive Representation Learning
Graduate School of Data Science, Seoul National University
Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction
Jun 4, 2026Tianxiang Jiang, Linquan Wu, Sheng Xia +5Large Vision-Language Models Latent Visual Reasoning
University of Science and Technology of China · Shanghai AI Laboratory · City University of Hong Kong +4

Mar 15, 2026J/K move · Enter open · S save

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mido Assran+5

FAIR at Meta · Universidad de Zaragoza

Self-Supervised Visual Representation Learning Visual Representation Learning Video Representation Learning Joint-Embedding Predictive Architecture

Open paper AskPDF arXiv

We present V-JEPA 2.1, a family of self-supervised models that learn dense, high-quality visual representations for both images and videos while retaining strong global scene understanding. The approach combines four key components. First, a dense predictive loss uses a masking-based objective in which both visible and masked tokens contribute to the training signal, encouraging explicit spatial and temporal grounding. Second, deep self-supervision applies the self-supervised objective hierarchically across multiple intermediate encoder layers to improve representation quality. Third, multi-modal tokenizers enable unified training across images and videos. Finally, the model benefits from effective scaling in both model capacity and training data. Together, these design choices produce representations that are spatially structured, semantically coherent, and temporally consistent. Empirically, V-JEPA 2.1 achieves state-of-the-art performance on several challenging benchmarks, including 7.71 mAP on Ego4D for short-term object-interaction anticipation and 40.8 Recall@5 on EPIC-KITCHENS for high-level action anticipation, as well as a 20-point improvement in real-robot grasping success rate over V-JEPA-2 AC. The model also demonstrates strong performance in robotic navigation (5.687 ATE on TartanDrive), depth estimation (0.307 RMSE on NYUv2 with a linear probe), and global recognition (77.7 on Something-Something-V2). These results show that V-JEPA 2.1 significantly advances the state of the art in dense visual understanding and world modeling.

Abstract

Video Representation Learning

Explore similar work

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction

Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction