Joint-Embedding Predictive Architectures (JEPAs) predict the latent embedding of a target signal from a context signal, with an auxiliary variable z conditioning the predictor. Despite its prominent role in the original JEPA blueprint, the content of z has been carried by a fixed deterministic token in every prior instantiation, collapsing the genuinely multi-modal distribution over plausible completions to a point estimate. We present S-JEPA, which realises z as a proper stochastic latent variable for masked image prediction: a learned posterior infers z from context and target during training, while a learned context-conditioned prior approximates the distribution of plausible completions at inference, with no decoders and no reconstruction. In its deterministic limit the objective reduces to the I-JEPA loss in functional form, generalising that objective rather than strictly containing the I-JEPA model. Our central finding is that stochasticity adds capabilities, error-correlated predictive uncertainty and diverse completions, at no cost to representation quality: under matched architectures and schedules S-JEPA improves over the deterministic I-JEPA baseline and exceeds LeJEPA at matched architecture, while additionally producing spatially structured uncertainty maps and semantically diverse completions, neither of which the deterministic predictor produces.
Figures & tables
Figure 1: JEPAs are trained to predict the representation of an input y from the representation of another input x . The additional variable z provides the predictor with information about the transformation that computes y from x .
Figure 2: S-JEPA overview. The posterior network qψ(z∣sx,sy) infers z from both context and target representations; the prior network pθ(z∣sx) infers z from context alone. The sampled z is injected into the predictor gϕ to produce s^y . No arrows return to observation space: S-JEPA is non-generative by design.
Method
Arch.
Epochs
Top-1
Methods without view data augmentations
data2vec [ 6 ]
ViT-L/16
1600
77.3
MAE [ 42 ]
ViT-B/16
1600
68.0
ViT-L/16
1600
76.0
ViT-H/14
1600
77.2
I-JEPA [ 4 ]
ViT-B/16
600
72.9
Table 1: ImageNet linear evaluation. Top-1 linear probing accuracy on frozen features.
Method
Arch.
Epochs
Top-1
Methods without view data augmentations
data2vec [ 6 ]
ViT-L/16
1600
73.3
MAE [ 42 ]
ViT-L/16
1600
67.1
ViT-H/14
1600
71.5
I-JEPA [ 4 ]
ViT-B/16 ‡
600
68.9
ViT-L/16
600
69.4
Table 2: ImageNet low-shot evaluation. Top-1 accuracy with 1% of ImageNet-1K labels, using full fine-tuning.
Method
Arch.
Cifar100
Places205
iNat18
Methods without view data augmentations
data2vec [ 6 ]
ViT-L/16
59.6
36.9
10.9
MAE [ 42 ]
ViT-H/14
77.3
55.0
32.9
I-JEPA [ 4 ]
ViT-B/16 †
69.2
53.4
43.4
ViT-L/16 †
83.6
56.5
48.4
ViT-H/14
87.5
58.4
47.6
Table 3: Transfer classification. Top-1 linear probing accuracy on CIFAR-100, Places205, and iNaturalist 2018.
Method
Arch.
Clevr/Count
Clevr/Dist
Methods without view data augmentations
data2vec [ 6 ]
ViT-L/16
72.7
53.0
MAE [ 42 ]
ViT-H/14
90.5
72.4
I-JEPA [ 4 ]
ViT-B/16 †
82.2
70.7
ViT-L/16 †
85.6
71.2
ViT-H/14
86.7
72.4
Table 4: Transfer low-level tasks. Accuracy on object counting (Clevr/Count) and depth prediction (Clevr/Dist).
Figure 3: PCA and attention visualisations of encoder features. Each row shows the original image, the PCA projection (first three principal components of block-4 patch representations, foreground-masked, mapped to RGB), and the self-attention map (block 8, head 11, averaged over all query positions), across three ViT-B/16 encoders trained with different objectives (MAE [ 42 ] , I-JEPA [ 4 ] , and S-JEPA). All methods recover coherent object/background structure in the PCA projection; the predictive objectives (I-JEPA, S-JEPA) additionally show attention concentrated on the primary object, in contrast to the more diffuse attention of the pixel-reconstruction objective (MAE). Neither map is supervised by any localisation signal: both emerge purely from the pretraining objective. Additional examples with more baselines are in Appendix Figures D.1 and D.2 .
Figure 4: Stochastic predictor completions via RCDM. Each row shows a masked input and five completions, each from a distinct draw zi∼pθ(z∣sx) decoded via a lightweight diffusion decoder [ 14 ] trained on frozen S-JEPA predictor representations. Samples vary in object identity and configuration, reflecting genuine uncertainty over the masked region encoded in z . Decoding uses a fixed noise seed across all five draws, so pixel-level variation reflects only the sampled latent and not decoder-side stochasticity.
Figure 5: Uncertainty and error maps. Up : predictive variance (Eq. ( 9 ), N=50 , full image as context) for I-JEPA, whose deterministic predictor gives Up=0 everywhere, and S-JEPA (ViT-B/16). Error: squared distance of the mean S-JEPA prediction to the target, each 5×5 -patch block masked in turn. The I-JEPA column carries no information about the underlying image, since its flat colour reflects only the normalisation; S-JEPA’s Up , by contrast, consistently concentrates on the animal or object rather than the background.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Table C.1 : Ablation study of S-JEPA on ImageNet-100, ViT-B/16, 400 epochs, batch size 512. We report k -NN top-1 and logistic regression top-1 accuracy (%). We highlight the default hyperparameter.
Targets
Context
Mask
Type
Freq.
Selection
Type
Avg. Ratio ∗
knn
logistic
grid
Strided
1
Fixed uniform
Grid(stride=2)
0.25
26.9
38.2
random
Random(0.6)
1
Random patches
Complement
0.40
36.9
54.6
rasterized
Quadrant
3
Fixed quadrant
Complement
0.25
27.5
50.1
block
Block(0.6)
1
Random rectangle
Complement
0.40
58.4
67.9
multi-block
Block(0.15,0.2)
4
Random rectangle
Block(0.85,1.0) × Complement
0.25
38.7
63.9
Appendix
Table C.2 : Masking strategy ablation of S-JEPA on ImageNet-100, ViT-B/16, 400 epochs, S-JEPA. We report k -NN top-1 and logistic regression top-1 accuracy (%). We highlight the default strategy.
Table C.3 : Ablation study of I-JEPA on ImageNet-100, ViT-B/16, 400 epochs, batch size 512. We report k -NN top-1 and logistic regression top-1 accuracy (%), along with Δ=S-JEPA−I-JEPA at the same hyperparameter value. We highlight the default hyperparameter.
Method
EMA
sg
knn
logistic
eff. rank
I-JEPA [ 4 ]
✗
✗
1.1
1.3
8
✗
✓
12.2
21.3
35
✓
✓
36.1
61.1
187
S-JEPA (ours)
✗
✗
1.1
1.2
6
✗
✓
22.4
41.7
76
✓
✓
38.7
63.9
201
Appendix
Table C.4 : Role of EMA and stop-gradient ( sg ). ImageNet-100, ViT-B/16, 400 epochs. Effective rank is a collapse diagnostic computed on ℓ2 -normalised encoder features over the validation set; low values indicate representation collapse independently of accuracy. We highlight the default configuration.
S-JEPA
I-JEPA
Pooled Spearman ρ
0.327
–
Within-image Spearman ρ (mean)
0.161
–
Images with within-image ρ>0
76%
–
ECEreg , raw ↓
0.180
0.237
ECEreg , after τ↓
0.068
–
NLL after τ↓
0.63 (0.98)
–
Appendix
Table D.1 : Uncertainty quality of ViT-B/16 under the pretraining masks. Spearman correlations between Up and error on 3000 validation images ( N=8 ); calibration on 500 held-out images ( N=50 ), with the temperature τ fit on a disjoint split. NLL in parentheses: constant-variance baseline with the same mean. I-JEPA has Up=0 .
N
Spearman ρ
ECEreg↓
5
0.300
0.184
10
0.272
0.182
25
0.289
0.181
50
0.279
0.181
I-JEPA
–
0.237
Appendix
Table D.2 : Sample count. Pooled Spearman ρ between Up and error, and ECEreg (Appendix B.3 ), of S-JEPA ViT-B/16 against the number of prior draws N , on 300 validation images under the pretraining masks; each row is reseeded. I-JEPA has Up=0 .
Figure D.1: PCA visualisations of encoder features (ViT-B/16). Each row shows the original image followed by the PCA projection of patch features for DINO [ 16 ] (400 epochs), iBOT [ 81 ] (400 epochs), MAE [ 42 ] (1600 epochs), MSN [ 2 ] (600 epochs), I-JEPA [ 4 ] (600 epochs) and S-JEPA (600 epochs), all ViT-B/16 encoders pretrained on ImageNet-1K. All baseline visualisations are reproduced from publicly released checkpoints.
Figure D.2: Attention visualisations of encoder features (ViT-B/16). Each row shows the original image followed by the self-attention map for DINO [ 16 ] (400 epochs), iBOT [ 81 ] (400 epochs), MAE [ 42 ] (1600 epochs), MSN [ 2 ] (600 epochs), I-JEPA [ 4 ] (600 epochs) and S-JEPA (600 epochs), all ViT-B/16 encoders pretrained on ImageNet-1K. All baseline visualisations are reproduced from publicly released checkpoints.