JEPA Learns What the Mask Leaves Unrecoverable

Peng Xie; Amr Alanwar

cs.CVSep 26, 2026

JEPA Learns What the Mask Leaves Unrecoverable

Authors: Peng Xie, Amr Alanwar

Organizations: TUM School of Computation, Information and Technology, Department of Computer Engineering, Technical University of Munich, 74076 Heilbronn, Germany

Abstract

Joint-embedding predictive architectures are unusually sensitive to how the input is masked: block masks work, scattered masks do not, and the explanations are empirical. We give a measurement account. A mask is a linear measurement, and in a compactly supported wavelet basis every atom whose support lies inside the hidden region falls in the measurement's null space and leaves no trace in the data. The JEPA loss asks only that the encoded context suffice for the target, so a target that a low-level prior can recover admits a shortcut, one the moving-average target encoder can make self-consistent. What removes the shortcut is the coarse-scale content the mask leaves unrecoverable, provided enough context stays within reach of each target. We score that content before training and test the account's distinctive predictions in 151 pre-training runs. On ImageNet-100, strip masks match blocks in area and contiguity yet are recoverable, and they land at 40.3% linear top-1, beside random masks at 40.8%, against 64.3% for blocks; within one geometry family, the placements that leave the least unrecoverable content lose 6.5 points to those that leave the most, over five seed pairs; pixel targets span 7 points where latent targets span 25; and against a frozen target the gap between random and block masks, 19 points on the same kind of GPU, closes to 1.5, so the geometry acts through the target the encoder produces for itself. On UCF101 the masking ratio decides which condition, content or reach, binds; removing whole frames, unrecoverable in space but recoverable from neighbouring frames, is worst at both ratios; and on V-JEPA's own masks, batching them intact instead of truncated changes little (36.0% against 35.1%), whereas making 100 target tokens inside the blocks visible lifts them to 48.7% and hiding 100 context tokens outside the blocks does not (33.7%).

Figures & tables

Figure 1: Top: an image rebuilt from atoms of decreasing scale. Middle: where its detail lives at each scale. Bottom: an atom of scale $s$ on its $s\times s$ support, and two 64-pixel atoms against a 128-pixel hole: the red one lies inside it and is annihilated, the blue one protrudes and is not. Right: detail energy by scale over 512 validation images.

Figure 2: Eight families on one image at a matched budget, from scattered to deep; grey is everything the encoder does not see. Strips are as contiguous as a block and cover the same area, yet on this image they are the most recoverable family (per-image PSNR).

	Family	$\nu$	Insc. sq.	Harmonic	TV	Wavelet- $\ell_{1}$	Linear top-1
F2	dispersed	0.023	2.39	18.78	17.65	16.36	42.00 $\pm$ 0.51
F1	random	0.033	2.58	18.36	17.20	15.98	40.77 $\pm$ 1.99
F7	strips	0.027	2.44	18.25	17.03	15.73	40.29 $\pm$ 2.70
F3	12 blocks	0.099	3.50	16.87	15.70	14.97	49.14 $\pm$ 1.90
F4	4 blocks $\dagger$	0.264	6.39	14.87	13.77	13.94	64.30 $\pm$ 0.62
F5	2 blocks	0.264	6.73	14.87	13.81	13.92	65.34 $\pm$ 0.08

Table 1: Depth decides recoverability; area and contiguity do not. Eight families at a matched budget on 5,000 ImageNet-100 validation images, with the probe after 300 epochs of pre-training on the right. Family-level

\rho(\nu,\text{TV})=-0.95

;

\rho(\nu,\text{linear})=+0.71

Figure 3: The 22 configurations placed by annihilated content $\nu\!\cdot\!n_{T}$ and reachability, coloured by linear-probe accuracy; dashed lines are the thresholds and the shaded quadrant is where both hold. Right: mean and range of each group.

Family	$\nu$	Pixel target	I-JEPA	Pixel $-$ I-JEPA	Pixel probe
strips	0.02	52.93 $\pm$ 0.01	40.29	$+12.64$	24.40
dispersed	0.02	53.10 $\pm$ 0.40	42.00	$+11.10$	23.90
random	0.03	55.09 $\pm$ 0.30	40.77	$+14.32$	23.73
random 75% $\ddagger$	0.08	58.48 $\pm$ 0.23	—	—	23.60
12 blocks	0.10	59.19 $\pm$ 0.95	49.14	$+10.05$	23.94
4 blocks $\dagger$	0.18	59.76 $\pm$ 0.54	64.30	$-4.54$	24.62

Table 2: Mask geometry matters nearly four times more for a latent target than for a pixel target, and the two curves cross. Same encoder, predictor and masks (

\ddagger

excepted); the pixel objective uses MAE’s optimiser settings and no moving average (Appendix N ). Mean of two seeds.

Figure 4: Both

y

-axes start at 30. Left: at 90% masking the families are ordered by the share of deep targets; red marks the two that remove whole time steps. Right: V-JEPA’s block geometry intact, with 100 context tokens outside the blocks turned hidden, and with 50, 100 and 200 target tokens inside them made visible (grey: the public collator); bars carry the share of deep targets.

	90% masked ( $n_{C}=157$ )				50% masked ( $n_{C}=784$ )
Family	$\nu^{3d}$	$R_{\text{low}}$	Deep	Linear	$R_{\text{low}}$	Linear
random tokens	0.30	17.43	0.1%	40.56 $\pm$ 0.92	17.92	38.05 $\pm$ 2.07
frame blocks	0.38	16.97	5.5%	40.33 $\pm$ 2.19	17.54	39.36 $\pm$ 0.22
random tube	0.54	15.99	11.8%	37.51 $\pm$ 0.11	—	—
tube, 1 block	0.52	16.45	20.9%	36.99 $\pm$ 3.98	—	—
tube, 4 blocks	0.54	15.84	30.5%	36.17 $\pm$ 1.48	17.07	41.67 $\pm$ 2.22

Table 3: The masking ratio decides which condition binds. UCF101 split 1, ViT-S, 200 epochs, mean of two seeds.

V-JEPA’s 8+2 masks, as batched	Density	Deep targets	Linear	$k$ -NN
public collator, batch-min truncation	1.14	59%	35.06 $\pm$ 0.58 (2)	26.74
official masks intact, batched by padding	1.76	55%	36.04 $\pm$ 1.36 (3)	28.20
100 context tokens outside the blocks hidden	2.11	51%	33.73 $\pm$ 0.45 (3)	27.41
50 target tokens inside the blocks made visible	2.71	2%	44.34 $\pm$ 3.83 (2)	34.79
100 inside	3.65	0.2%	48.72 $\pm$ 1.35 (3)	37.89
200 inside	5.55	0%	47.25 $\pm$ 0.73 (2)	34.77

Table 4: Where the visible tokens sit decides: V-JEPA’s 8+2 block geometry as batched by six collators, UCF101 split 1, ViT-S, 200 epochs, seeds in parentheses.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Masks, as batched	Density	Anchored	Deep	AvgDist	Linear
public collator (batch-min truncation) $\ast$	1.14	17%	59%	3.83	35.1
official masks intact	1.76	23%	55%	3.52	36.0
context thinned to 349 / 102	0.92	22%	55%	3.53	36.4
100 context tokens outside hidden	2.11	28%	51%	3.39	33.7
50 target tokens inside visible	2.71	71%	2%	1.31	44.3
100 inside visible	3.65	88%	0.2%	1.12	48.7

Appendix

Table 5: Mask statistics of the collator variants: density (mean context tokens among the 26 space-time neighbours of a target) averaged over the two sets, and, over all targets of both sets, the share with a visible token among their 26 neighbours, the share three or more cells from any visible token, and the mean chessboard distance in cells from a target to the nearest visible token (AvgDist). Rows marked

\ast

were trained on H100/H200, the others on RTX 5090.

Figure 5: Left: each family under the two prediction targets; a line points right when the latent target wins. Right: every mixture of easy and hard masks lands below the linear interpolation of the pure runs (shaded).

Joint-Embedding Predictive Architecture

Joint-Embedding Predictive Architecture (JEPA) learns representations by predicting latent features of hidden or future data from observed context.

See all 240 papers

Sep 19, 2026

Mask-Aware Execution for Efficient JEPA Training

Joint Embedding Predictive Architectures (JEPAs) are becoming a core representation-learning primitive and a building block for latent world models across vision, video, audio, brain dynamics, and time series. Despite (potential of) wide deployment, current JEPA training pipelines are inefficient: each input is executed through multiple mask-specific branches, with redundant target-side work, and memory-bound token routing. These costs grow with the number of masks and limit GPU efficiency. We present M-JEPA, a mask-aware execution architecture that restructures JEPA training without changing the learning objective. M-JEPA separates mask-independent computation from mask-dependent routing, enabling shared context encoder execution, fused token routing and slicing with backward support, sparse target encoder execution over the union of target tokens, and masked patch embedding for sparse inputs. The resulting pipeline preserves training semantics while reducing computation, memory traffic, and synchronization overhead. We implement M-JEPA for five JEPA variants and evaluate it on NVIDIA A100 GPUs. Compared against the state-of-the-art baselines, M-JEPA achieves up to 1.7x end-to-end training speedup for 2-10 masks. Separately, with masked patch embedding, 4.75x patch-embedding speedup at high sparsity. These results show that execution restructuring, rather than changes to the JEPA objective, is a key lever for efficient JEPA training.

Md Musfiqur Rahman Sanim, Zhihao Shu, Bahram Afsharmanesh +3

University of Georgia Athens, GA, USA

Joint-Embedding Predictive Architecture Memory-Efficient Training Deep Learning Optimization

Ask PDF

Oct 23, 2026

Shared Is Not Enough: Predictor Width and Factor Readability Under Masked Latent Prediction

Masked latent prediction methods such as I-JEPA aim to learn predictable features, but what determines which information they retain is less clear. We built a simulator in which every factor behind an image is known, so a probe can measure how readable each factor remains after training. We trained small encoders (0.7M parameters) with I-JEPA, SimCLR and BYOL on the same pairs of views and probed a factor that both views share. I-JEPA had a predictor with half the encoder's width and depth, the proportions it uses for ViT-B. Its median score was about 40% of SimCLR's. Widening the predictor to the encoder's width reversed this ordering at every training seed, against both other methods. We release the simulator and the trained models.

Nilotpal Kakati

Weizmann Institute of Science, Rehovot, Israel

Representation Learning Predictive Representation Learning Self-Supervised Learning Joint-Embedding Predictive Architecture

Ask PDF

Jul 26, 2026

The JEPA Paradox in Language: The Geometry of Linguistic Alternatives

Joint-Embedding Predictive Architectures (JEPAs) are effective for images, video, and audio, yet deterministic JEPA-style latent prediction has not become a standard objective for text encoders. We argue that this gap reflects a mismatch between squared-error latent prediction and the conditional structure of language. The key requirement is conditional concentration: given a context and target location, the target representation should lie near a single meaningful point. Local image prediction often satisfies this through spatial continuity, whereas masked text can admit multiple valid token or span completions whose representations need not share a coherent center. We formalize this mismatch through three conditions---predictability, non-collapse, and low conditional variance---and show how their failure creates centroid degeneracy and collapse pressure in text. Matched I-JEPA and T-JEPA experiments reveal the predicted sequence: mutual-information saturation and elevated target variance precede train--validation instability, effective-rank degeneration, cosine collapse, and poor downstream transfer. The same pattern appears across five independent data seeds, indicating that it is not a sampling artifact. These results do not rule out predictive learning for language; they show that text-compatible JEPA objectives must preserve multiple plausible completions rather than compress them into a single latent point.

Anh Trac Duc Dinh, Khang Nhat Hoang Vo

Center for AI Research (CAIR), VinUniversity, Hanoi, Vietnam · Faculty of Computer Science and Engineering Ho Chi Minh City University of Technology (HCMUT), VNU-HCM Ho Chi Minh City, Vietnam · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates

Representation Collapse Masked Language Modeling Joint-Embedding Predictive Architecture Predictive Representation Learning Representation Geometry in Language Models

Ask PDF

Abstract

Joint-Embedding Predictive Architecture

Explore similar work

Mask-Aware Execution for Efficient JEPA Training

Shared Is Not Enough: Predictor Width and Factor Readability Under Masked Latent Prediction

The JEPA Paradox in Language: The Geometry of Linguistic Alternatives