cs.CVAug 27, 2026

LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

Authors: Lukas Kuhn, Lucas Maes, Giuseppe Serra, Quentin Le Lidec, Yann LeCun, Randall Balestriero, Florian Buettner

Organizations: German Cancer Research Center · German Cancer Consortium · Mila · Université de Montréal · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Video carries the temporal structure of the physical world, yet learning representations from it has remained computationally expensive: prevailing self-supervised methods either prevent representation collapse through architectural asymmetries, coupling an exponential-moving-average target encoder, a stop-gradient, and a capacity-limited predictor, or circumvent it by reconstructing masked content in pixel space. We introduce LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective, which dispenses with both. A single encoder is trained with an invariance loss over global and local views of a clip, regularized by SIGReg, which excludes collapse with a provable guarantee. The architecture reduces to an encoder and a projector, and the objective to a single hyperparameter. This formulation admits two properties. First, the cost of pretraining is governed by the number of tokens the encoder observes; uniform random token dropping renders this number small while simultaneously improving downstream accuracy. At matched epochs on identical data, LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute, and at matched total FLOPs it exceeds the strongest video baseline by 7.6 points on ImageNet-1K while remaining competitive on motion-centric benchmarks. Second, since no asymmetry between branches is required, the encoder can be trained with block-causal attention at no measurable accuracy cost: temporal ordering becomes a property of the encoder itself. Against a compute-matched DINOv2 trained on frames of the same videos, LeVJEPA approaches the image-pretrained encoder on appearance-centric evaluation while nearly doubling its motion-centric accuracy. These results indicate that, once its computational overhead is removed, video becomes a viable and in several respects preferable substrate for general-purpose visual pretraining.

Figures & tables

Figure 2: Accuracy against total pretraining compute at matched epochs. ImageNet-1K attentive-probing accuracy of ViT-S, ViT-B, and ViT-L encoders pretrained for 240240 epochs on the identical 20%20\% subsample of K710 (marker size indicates model size; the horizontal axis is logarithmic and reversed).
Figure 3: Patch-token PCA across methods. The three leading principal components of the patch-token representations, visualized as RGB, for the same input image. LeVJEPA yields a decomposition that cleanly separates the animal from the surrounding furniture and background, comparable to that of V-JEPA 2.1, which obtains its dense structure through an explicitly introduced auxiliary patch-level loss [ 21 ] . V-JEPA 2, trained without such a loss, exhibits no comparable token-level organization. In LeVJEPA, this structure emerges although the training objective supervises only the [cls] token and no patch-level supervision is applied.
Figure 4: Ablations on view construction. ( 4(a) ) Effect of the masking ratio and ( 4(b) ) effect of the number of local views on ImageNet linear probing accuracy.
Patch embeddingIN1KSSv2
τ=2\tau=2 , ρ=0.90\rho=0.9047.428.8
τ=1\tau=1 , ρ=0.95\rho=0.9550.730.4
Table 1: Temporal patch aggregation at a matched token budget. Top-1 accuracy of a frozen attentive probe. Both configurations retain an equal number of tokens per view during pretraining and are evaluated on sequences of 88 temporal slots.
AttentionIN1K top-1
Bidirectional50.7
Block-causal51.2
Table 2: Effect of the attention topology. ImageNet-1K top-1 accuracy of a frozen attentive probe; both configurations use τ=1\tau=1 , ρ=0.95\rho=0.95 , V=4V=4 , and uniform random dropping.
Figure 5: Further patch-token visualizations. ( 5(a) ) Cosine similarity between a query patch placed on the object and all patch tokens of the image: high similarity is confined sharply to the object, indicating that the token representations are not only semantically organized but spatially precise. ( 5(b) ) Patch-token PCA as in Figure 3 , for a different scene: the decomposition again groups tokens by semantic region, separating the animal from furniture, floor, and background, indicating highly semantic patch tokens.
MethodIN1KSSv2K400
VideoMAEv253.443.637.4
V-JEPA 251.642.540.7
LeVJEPA61.040.444.6
Table 3: FLOP-matched comparison on identical pretraining data. ViT-B encoders pretrained on the 20%20\% subsample of K710 at equal total pretraining FLOPs, evaluated frozen; IN1K and SSv2 report attentive-probing top-1 accuracy, K400 reports linear-probing top-1 accuracy.
MethodIN1KSSv2
DINOv253.816.9
LeVJEPA50.730.4
Table 4: Comparison to image-based pretraining at matched compute. ViT-B encoders pretrained at equal total FLOPs on [identical source data]; top-1 accuracy of frozen attentive probes.
More in this topic

Causal Attention

See all 15 papers

Explore similar work

CardsList
Sep 27, 2026

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

Comparisons in video self-supervised learning often evaluate complete training recipes rather than isolating the method itself: architecture, objective, data exposure, schedule, scale, and decoder capacity can all vary at once. This makes it hard to identify which choices yield motion-prioritized representations, whose gains concentrate on frame-to-frame change while retaining useful appearance. We address this with a matched 4×6=244 \times 6 = 24 architecture-objective study at roughly 170M ~ 190M encoder scale on ∼\sim1.7M OpenVid and Moments-in-Time v2 clips for 8 epochs, and propose TT-VidT. TT-VidT combines a DINOv3-initialized ViT-B/16 per-frame spatial path with a compact Temporal Transfer Layer, trained by Diff Compression to reconstruct target frames from a first-frame appearance anchor and frame-specific motion tokens. The sweep shows that TT3D with Diff Compression, not either component alone, enters the strongest motion-sensitive regime, and decoder ablations favor a compact video-pretrained decoder. In final comparison, TT-VidT leads Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously, improving over the strongest non-TT row by 54% ~ 121%, while using 48% fewer encoder FLOPs than DisMo and 55% fewer than VideoMAE or V-JEPA2. HMDB51, IARD, and EPIC-Kitchens bound the claim.

Shih-Ying Yeh, Daniel Z. Kaplan, Xuehai Wang +3

National Tsing Hua University · Comfy Org Research · Karolinska Institutet +2

Efficient ViTsVideo Representation LearningMotion Representation Learning
AskPDF
Oct 5, 2026

LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning

Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes audio, video, and joint audio-video inputs. Modality dropout treats a missing modality as another view of the same event, making cross-modal alignment implicit in the objective. The model aligns global embeddings with modality-specific local embeddings, and SIGReg prevents representational collapse. A controlled ablation identifies modality dropout as the key mechanism for audio-visual alignment. Despite the architectural simplicity, LeAVJEPA reaches 36.0 mAP on AudioSet-20K and 91.3% accuracy on ESC-50 under frozen evaluation. After fine-tuning, it reaches 61.1% accuracy on VGGSound, and its embeddings support zero-shot audio-visual retrieval.

Benjamin Robson, Santeri Mentu, Wenshuai Zhao +1

ELLIS Institute Finland Aalto University

Audio-Visual Representation LearningJoint-Embedding Predictive ArchitectureAudio-Visual AlignmentMultimodal SSL
AskPDF
May 16, 2026

Factorized Latent Dynamics for Video JEPA: An Empirical Study of Auxiliary Objectives

Joint-Embedding Predictive Architectures (JEPA) are a promising framework for self-supervised video representation learning, yet the behavior of auxiliary objectives in small-scale Video-JEPA training is not well characterized. We report a small-scale empirical study of 18 auxiliary objective variants for Video-JEPA across two pretraining regimes: single-dataset (UCF-101) and mixed-dataset (UCF-101 + Something-Something V2 + ImageNet-100). We evaluate frozen representations on three complementary benchmarks: Diving-48 (fine-grained motion), SomethingSomething V2 (temporal reasoning), and ImageNet-100 (appearance). Our experiments suggest that many auxiliary objectives exhibit capacity trade-offs: gains on one downstream capability often coincide with degradation on another. We then study FWM-HW-LD (Factorized World-Model with Hard-Region-Weighted Latent Dynamics), a training-time objective that separates the latent representation into appearance and dynamics subspaces and applies hard-region weighting to both JEPA prediction errors and latent dynamics errors. In our mixed-dataset setting, FWM-HW-LD improves ImageNet-100 by +5.92 and SSv2 by +3.21 percentage points relative to the reference baseline, while remaining within 0.30 percentage points on Diving-48. These results indicate that latent factorization is a useful direction for studying auxiliary-objective trade-offs in Video-JEPA.

Santosh Premi

Disentangled Representation LearningLatent Dynamics ModelingJoint-Embedding Predictive ArchitectureVideo Action RecognitionVideo Representation Learning
AskPDF