Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures

José Lucas De Melo Costa; Seong Woo Ahn; Fabrice Popineau; Arpad Rimmel; Bich-Liên Doan

cs.LGOct 1, 2026

Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures

Authors: José Lucas De Melo Costa, Seong Woo Ahn, Fabrice Popineau, Arpad Rimmel, Bich-Liên Doan

Organizations: Université Paris-Saclay, CNRS, CentraleSupélec, LISN, Orsay, France

Abstract

Joint-Embedding Predictive Architectures (JEPAs) are prone to representation collapse, typically mitigated through empirical heuristics. We develop an early-training stability theory that unifies these heuristics. Linearising the coupled JEPA gradient flow around the trivial fixed point reveals two competing effects: a driving force ( $γ$ ) and a decay effect ( $σ$ ). Under approximate spectral decoupling, a per-mode stability ratio $μ_i = γ_i / σ_i$ factorises into independent data-side and predictor-side terms and the count of unstable modes tracks the rank of representations that can emerge. The framework predicts a phase boundary, which we confirm empirically across more than 800 Tabular-JEPA configurations. It also unifies predictor scaling, masking ratio, and EMA as distinct mechanisms for shifting $μ$ . Guided by this analysis, we introduce ResidualPred, a transformer predictor whose attention is biased toward the identity at initialisation; it improves both effective rank and downstream accuracy on tabular benchmarks and in I-JEPA pretraining on CIFAR-10, CIFAR-100, STL-10, and ImageNet. Our framework connects empirical collapse-avoidance heuristics to an explicit dynamical picture, yielding theory-driven stabilizers. Code is available at https://github.com/jose-melo/drive-vs-decay.

Figures & tables

Figure 1: Phase boundary. Learning where the drive wins (

\mu=\gamma/\sigma>1

), collapse where the decay wins (

\mu<1

Panel of Figure 2 — Figure 2: Linear JEPA architecture.

Figure 3: Inverse

\mu(\alpha)

scaling. Empirical

\mu

vs. predictor scale

\alpha

for

p_{m_{1}}\!\in\!\{0.2,\dots,0.7\}

Figure 4: Empirical phase diagram of Tabular JEPA stability predicted by the drive-decay framework. Each panel reports a two–dimensional sweep over the predictor scale $\log_{10}(\alpha)$ and the context masking ratio $p_{m_{1}}$ , for three tabular datasets. Colored heatmaps show the downstream normalized linear probe score . The thick red curve highlights the critical boundary $\mu=1$ , which our theory identifies as the transition between a learning regime and a collapsing regime.

Model	Context ratio	Predictor scale	Epoch 0				Last epoch				Final loss	Test Acc.
Model	Context ratio	Predictor scale	$\mu_{25}$	$\mu_{50}$	$\mu_{75}$	$\mathrm{Erank}$	$\mu_{25}$	$\mu_{50}$	$\mu_{75}$	$\mathrm{Erank}$	Final loss	Test Acc.
Tabular–JEPA variants	low	low	$1.49$	$2.46$	$3.92$	$11.05$	$0.76$	$2.71$	$7.91$	$4.56$	$0.0180$	$0.509$
	medium		$1.54$	$2.71$	$4.17$	$12.69$	$0.58$	$1.64$	$4.62$	$14.49$	$0.1798$	$0.737$
	high		$0.96$	$1.76$	$3.17$	$16.18$	$0.42$	$1.63$	$7.39$	$16.26$	$0.0984$	$0.872$
	low	medium	$0.53$	$0.88$	$1.82$	$9.93$	$0.41$	$0.78$	$4.44$	$3.92$	$0.0149$	$0.433$
	medium		$0.67$	$1.14$	$1.93$	$9.97$	$0.42$	$1.12$	$7.46$	$5.92$	$0.0248$	$0.603$

Table 1: Aggregated results for Tabular–JEPA variants (aggregated over ALOI, Helena, and Jannis) and T-JEPA (Jannis). For Tabular–JEPA, only test accuracy is normalized within each dataset before aggregation. Metrics are reported as mean values over 5 runs.

Panel of Figure 5 — Figure 5: $\Erank$ vs $\tilde{r}_{\mathit{tight}}$ on helena (one point per run); colour is $\log_{10}\max_{k\ell}\tilde{\mu}_{k\ell}$ .

Mechanism	Derived dose	Mode escapes iff
Weight decay	additive floor: $\sigma_{i}+\eta_{wd}$	$\mu_{i}>1+\eta_{wd}/\sigma_{i}$
SIGReg	lowers the predictor factor: $p_{\ell}-\rho$	$\mu>1-\rho/p_{\ell}$
VICReg hinge	lowers it by $\tilde{\lambda}$ , its small-weight dose	$\mu>1-\tilde{\lambda}/p_{\ell}$
Mimetic initialisation	$q_{\ell}/p_{\ell}$ dosed once, washed out at rate $\zeta$	integrated growth clears $\log 10^{3}$
Identity predictor	$\mathbf{P},\mathbf{Q}$ fixed; a data-spectrum condition	$s_{k}>(1-p)/(p_{tgt}-p)$ , $p_{tgt}>p$

Table 3: Five further anti-collapse mechanisms as doses on

\mu

, derived in Appendix J .

p

is the context keep probability and

\{s_{k}\}

the spectrum of the unit-diagonal data covariance.

Tabular brittle regime			Image
Variant	Jannis	Helena	Variant	CIFAR-10	CIFAR-100	STL-10	IN-100
T-JEPA	$43.4\pm 1.7$	$\phantom{0}8.6\pm 1.1$	I-JEPA	$64.55\pm 1.71$	$36.83\pm 1.49$	$71.25\pm 1.14$	$37.09\pm 0.68$
+ ResPred	$\mathbf{46.9\pm 3.5}$	$\mathbf{21.3\pm 3.0}$	+ ResPred	$\mathbf{68.61\pm 0.59}$	$\mathbf{40.85\pm 0.66}$	$\mathbf{75.31\pm 0.78}$	$\mathbf{42.73\pm 0.41}$

Table 4: ResidualPred across tabular and image datasets. Linear-probe accuracy (%). More ImageNet results: Appendix I.5 , Figure 6(c) .

Panel of Figure 6 — Figure 6: ResidualPred – aggregated brittle-vs-non-brittle diagnostics, image-domain training trajectory, and the ImageNet-1k pilot. Panel (a) carries the general tabular claim: the predictors start matched and Standard finishes below its own initialisation in both regimes. Panel (c): ViT-S/16, predictor depth $6$ , $30$ epochs, five seeds each, mean $\pm$ s.d.; every ResidualPred seed lies above every standard seed at both resolutions.

	Linear (%)	Effective rank
I-JEPA	$25.22\pm 0.80$	$107.2\pm 7.7$
+ SIGReg	$29.19\pm 0.54$	$185.6\pm 5.6$
+ ResPred	$31.69\pm 0.22$	$188.5\pm 3.3$
+ ResPred + SIGReg	$\mathbf{32.23\pm 0.54}$	$\mathbf{206.7\pm 1.0}$

Table 5: Against SIGReg on the ImageNet-1k pilot. Mean over five seeds.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Relative data-side commutator

C_{\Sigma}

as a function of context ratio

p_{m_{1}}

. Left : raw

C_{\Sigma}

. Right :

\|[\mathbf{\Sigma}_{11},\mathbf{\Sigma}_{X1}]\|_{F}/\bigl(p_{m_{1}}^{2}(1-p_{m_{1}})\bigr)

– flat in

p_{m_{1}}

(residual

<10^{-12}

relative), confirming the closed-form identity ( 39 ). The CIFAR-10 raw-patch curve sits well below the synthetic benchmarks at all operating points.

$\mathbf{\Sigma}_{XX}$	$C_{\Sigma}$ at $p_{m_{1}}=0.85$	$\max_{p_{m_{1}}}C_{\Sigma}$
isotropic ( $\mathbf{I}$ )	$0.0$	$0.0$
diagonal power-law	$0.0$	$0.0$
CIFAR-10 raw patches ( $n{=}128{,}000$ , $D{=}48$ )	$\mathbf{9.7\times 10^{-5}}$	$\mathbf{2.7\times 10^{-3}}$
synthetic rotated power-law	$6.7\times 10^{-3}$	$7.6\times 10^{-2}$
synthetic strong rank-decay	$9.7\times 10^{-3}$	$1.9\times 10^{-1}$

Appendix

Table 6: Relative data-side commutator

C_{\Sigma}

across spectra at

p_{m_{1}}=0.85

, and the maximum over

p_{m_{1}}\in[0.05,0.99]

. The masking factor

f(0.85)=0.108

. Real CIFAR-10 raw-patch covariance gives

C_{\Sigma}

on the order of

10^{-4}

, two to three orders of magnitude tighter than rotated power-law surrogates.

	aloi ( $n{=}264$ )	helena ( $n{=}270$ )	jannis ( $n{=}270$ )
$\tilde{r}_{\mathit{tight}}$ (abs., Cor. 3.3 )	$\mathbf{+0.905}$ $[.87,.93]$	$\mathbf{+0.906}$ $[.88,.93]$	$+0.619$ $[.54,.70]$
$\tilde{r}_{\mathit{loose}}=\|\widetilde{\mathcal{U}}\|$ (abs.)	$+0.766$ $[.71,.81]$	$+0.897$ $[.86,.92]$	$+0.574$ $[.49,.65]$
$r_{\mathit{tight}}$ (signed)	$+0.672$ $[.61,.72]$	$+0.908$ $[.88,.93]$	$+0.648$ $[.59,.70]$
$r_{\mathit{loose}}=\|\mathcal{U}\|$ (signed)	$+0.772$ $[.72,.81]$	$+0.897$ $[.87,.92]$	$+0.697$ $[.65,.73]$
$\log_{10}\!\max_{k\ell}\tilde{\mu}_{k\ell}$ (scalar)	$+0.493$	$+0.844$	$+0.470$

Appendix

Table 7: Per-dataset Spearman

\rho(\Erank,\cdot)

for the absolute-value and signed formulations and the loose counts, with

1000

-resample bootstrap

95\%

CIs.

Figure 8: On helena the two formulations agree. The signed

\mu_{k\ell}

and the absolute-value

\tilde{\mu}_{k\ell}

give essentially identical scatter against

\Erank

Panel of Figure 9 — Figure 9: Monotone effect of $\beta_{\text{init}}$ on ERank and downstream accuracy.

	100 epochs		300 epochs
Variant	Acc. (%)	$\Delta$ / $p$ vs Std	Acc. (%)	$\Delta$ / $p$ vs Std
Standard	$36.83\pm 1.49$	–	$46.89\pm 0.77$	–
ZeroInit only	$37.29\pm 0.65$	$+0.46$ / $0.55$	$47.20\pm 0.80$	$+0.32$ / $0.69$
ResidualPred (bias only)	$\mathbf{41.14\pm 1.07}$	$\mathbf{+4.31}$ / $\mathbf{0.011}$	$\mathbf{50.36\pm 0.53}$	$\mathbf{+3.48}$ / $\mathbf{0.046}$
ResidualPred $+$ ZeroInit	$40.85\pm 0.66$	$+4.02$ / $0.005$	$49.62\pm 0.13$	$+2.73$ / $0.12$

Appendix

Table 8: Linear-probe accuracy across the

2\times 2

grid, CIFAR-100, ViT-Small/4,

p_{m_{1}}=0.85

. Mean

\pm

standard deviation over seeds.

\Delta

and two-sided

p

-values are computed against the Standard cell. ResidualPred (the design retained in the main paper) and ResidualPred + ZeroInit are statistically indistinguishable in mean.

Figure 10: Adiabatic stability. Assuming the target encoder tracks the context encoder proportionally, the parameter $\tau$ can be used to avoid representational collapse also yielding better performance; see the analysis in Appendix F.4 .

Figure 11: Bifurcation Phenomenon. Empirical analysis of JEPA collapse metrics as a function of the parameter $\tau$ . The plot displays the mean Frobenius Norm (right axis) and mean Condition Number (left axis, log scale) of the encoder weights $\mathbf{W}$ , averaged over 20 experimental runs. The shaded area represents the standard deviation. A critical threshold, empirically determined at $\tau_{crit}\approx 0.551$ , delineates the transition between the collapsed and non-collapsed operational regimes.

Figure 12: Loss landscape projection. Two-dimensional projection of a JEPA-like loss landscape in the Jannis dataset. An empirical critical value of $\tau_{\text{crit}}\approx 0.400$ is noted for this illustration. ( Left ) Subcritical regime ( $\tau=0.200<\tau_{\text{crit}}$ ): The loss landscape exhibits a global minimum closer to the origin ( $r_{1}\approx 0,r_{2}\approx 0$ ), indicating the stability of the collapsed solution $\mathbf{W}\approx\mathbf{0}$ . ( Right ) Supercritical regime ( $\tau=0.600>\tau_{\text{crit}}$ ): The origin has transformed into a region of higher loss, and new, distinct minima with $\mathcal{L}>0$ have emerged away from the origin.

Library	Description
Python v3.12	Core programming language used for all experiments
pytorch $\geq$ 2.0.0	Deep learning framework used to implement all models
numpy $\geq$ 1.24.0	Numerical computing and array operations
sciPy $\geq$ 1.10.0	Scientific computing utilities and linear algebra
pandas $\geq$ 2.0.0	Tabular data processing and experiment management
scikit-learn $\geq$ 1.2.0	Baseline models and evaluation utilities

Appendix

Table 9: Main libraries used in the project.

	Jannis	Helena	ALOI
Samples	83,733	65,196	108,000
Numerical	$54$	$27$	$128$
Classes	$4$	$100$	$1{,}000$
Metric	Accuracy	Accuracy	Accuracy

Appendix

Table 10: Dataset characteristics

Dataset	Context	Predictor scale	Config ID	Epoch 0 $\mu_{50}$		Epoch 0 Erank		Last epoch $\mu_{50}$		Last epoch Erank		Final loss	Test Acc.
Dataset	Context	Predictor scale	Config ID	Mean	SEM	Mean	SEM	Mean	SEM	Mean	SEM	Final loss	Test Acc.
ALOI	0.2	0.01	0	$1.46$	$0.029$	$19.44$	$0.17$	$0.64$	$0.006$	$9.45$	$0.06$	$0.0350$	$0.699$
Helena	0.2	0.01	0	$20.06$	$1.520$	$1.40$	$0.00$	$4.84$	$0.361$	$2.01$	$0.07$	$0.0106$	$0.229$
Jannis	0.2	0.01	0	$2.01$	$0.039$	$11.41$	$0.22$	$1.92$	$0.015$	$6.12$	$0.19$	$0.0425$	$0.658$
ALOI	0.2	0.05	1	$0.68$	$0.007$	$20.50$	$0.30$	$0.64$	$0.003$	$7.65$	$0.08$	$0.0219$	$0.668$
Helena	0.2	0.05	1	$5.83$	$0.737$	$1.52$	$0.02$	$6.48$	$1.336$	$1.51$	$0.03$	$0.0041$	$0.179$

Appendix

Table 11: Detailed results for Tabular–JEPA variants across ALOI, Helena, and Jannis datasets. Each row shows results for a specific context ratio and predictor scale configuration, with means and standard errors (SEM) aggregated over seeds. Collapse statistics (

\mu

quartiles), effective rank (Erank), final training loss, and test accuracy are reported.

Dataset	Context	Predictor scale	Config ID	Epoch 0 $\mu_{50}$		Epoch 0 Erank		Last epoch $\mu_{50}$		Last epoch Erank		Final loss	Test Acc.
Dataset	Context	Predictor scale	Config ID	Mean	SEM	Mean	SEM	Mean	SEM	Mean	SEM	Final loss	Test Acc.
ALOI	0.2	0.50	5	$0.09$	$0.002$	$11.11$	$0.04$	$0.06$	$9.17\cdot 10^{-4}$	$5.47$	$0.05$	$0.0198$	$0.689$
Helena	0.2	0.50	5	$1.14$	$0.091$	$1.37$	$0.01$	$13.63$	$3.757$	$1.06$	$0.00$	$1.73\cdot 10^{-4}$	$0.128$
Jannis	0.2	0.50	5	$0.09$	$0.002$	$8.26$	$0.13$	$0.08$	$0.001$	$3.09$	$0.06$	$0.0203$	$0.585$
ALOI	0.2	1.00	6	$0.05$	$9.99\cdot 10^{-4}$	$9.52$	$0.22$	$0.03$	$3.51\cdot 10^{-4}$	$3.96$	$0.15$	$0.0157$	$0.625$
Helena	0.2	1.00	6	$0.67$	$0.153$	$1.36$	$0.02$	$24.84$	$1.856$	$1.04$	$0.00$	$1.34\cdot 10^{-4}$	$0.119$

Appendix

Table 12: Detailed results for Tabular–JEPA variants across ALOI, Helena, and Jannis datasets (continued).

Dataset	Context	Predictor scale	Config ID	Epoch 0 $\mu_{50}$		Epoch 0 Erank		Last epoch $\mu_{50}$		Last epoch Erank		Final loss	Test Acc.
Dataset	Context	Predictor scale	Config ID	Mean	SEM	Mean	SEM	Mean	SEM	Mean	SEM	Final loss	Test Acc.
ALOI	0.3	0.20	10	$0.23$	$0.007$	$13.72$	$0.27$	$0.18$	$7.53\cdot 10^{-4}$	$7.90$	$0.08$	$0.0228$	$0.738$
Helena	0.3	0.20	10	$1.08$	$0.019$	$1.51$	$0.02$	$4.30$	$1.980$	$1.15$	$0.00$	$3.37\cdot 10^{-4}$	$0.153$
Jannis	0.3	0.20	10	$0.23$	$9.50\cdot 10^{-4}$	$9.38$	$0.11$	$0.22$	$0.001$	$4.01$	$0.02$	$0.0283$	$0.625$
ALOI	0.3	0.30	11	$0.15$	$0.001$	$11.82$	$0.05$	$0.11$	$0.001$	$6.66$	$0.04$	$0.0213$	$0.706$
Helena	0.3	0.30	11	$0.78$	$0.063$	$1.38$	$0.01$	$5.41$	$2.414$	$1.11$	$0.00$	$2.12\cdot 10^{-4}$	$0.142$

Appendix

Table 13: Detailed results for Tabular–JEPA variants across ALOI, Helena, and Jannis datasets (continued).

Dataset	Context	Predictor scale	Config ID	Epoch 0 $\mu_{50}$		Epoch 0 Erank		Last epoch $\mu_{50}$		Last epoch Erank		Final loss	Test Acc.
Dataset	Context	Predictor scale	Config ID	Mean	SEM	Mean	SEM	Mean	SEM	Mean	SEM	Final loss	Test Acc.
ALOI	0.4	0.20	17	$0.22$	$0.007$	$13.60$	$0.18$	$0.18$	$1.10\cdot 10^{-4}$	$8.57$	$0.15$	$0.0252$	$0.747$
Helena	0.4	0.20	17	$1.18$	$0.078$	$1.46$	$0.01$	$4.61$	$0.305$	$1.16$	$0.02$	$3.61\cdot 10^{-4}$	$0.148$
Jannis	0.4	0.20	17	$0.22$	$0.001$	$9.57$	$0.12$	$0.22$	$4.25\cdot 10^{-4}$	$4.19$	$0.04$	$0.0263$	$0.623$
ALOI	0.4	0.30	18	$0.16$	$5.16\cdot 10^{-4}$	$11.63$	$0.17$	$0.11$	$0.002$	$7.11$	$0.08$	$0.0225$	$0.736$
Helena	0.4	0.30	18	$0.89$	$0.151$	$1.40$	$0.01$	$11.67$	$4.567$	$1.10$	$0.02$	$1.98\cdot 10^{-4}$	$0.127$

Appendix

Table 14: Detailed results for Tabular–JEPA variants across ALOI, Helena, and Jannis datasets (continued).

Dataset	Context	Predictor scale	Config ID	Epoch 0 $\mu_{50}$		Epoch 0 Erank		Last epoch $\mu_{50}$		Last epoch Erank		Final loss	Test Acc.
Dataset	Context	Predictor scale	Config ID	Mean	SEM	Mean	SEM	Mean	SEM	Mean	SEM	Final loss	Test Acc.
ALOI	0.7	0.01	35	$0.79$	$0.018$	$26.39$	$0.09$	$0.40$	$0.005$	$16.88$	$0.21$	$0.0309$	$0.850$
Helena	0.7	0.01	35	$12.34$	$0.464$	$2.67$	$0.06$	$1.62$	$0.188$	$2.83$	$0.07$	$0.0233$	$0.272$
Jannis	0.7	0.01	35	$0.47$	$0.021$	$30.97$	$0.02$	$0.17$	$0.003$	$29.25$	$0.50$	$0.2604$	$0.690$
ALOI	0.7	0.05	36	$0.53$	$0.003$	$22.37$	$0.01$	$0.37$	$0.006$	$13.36$	$0.23$	$0.0217$	$0.821$
Helena	0.7	0.05	36	$3.48$	$0.272$	$2.23$	$0.06$	$4.95$	$0.874$	$2.11$	$0.03$	$0.0106$	$0.217$

Appendix

Table 15: Detailed results for Tabular–JEPA variants across ALOI, Helena, and Jannis datasets (continued).

Context	Predictor scale	Config ID	Epoch 0 $\mu_{50}$	Epoch 0 Erank	Last epoch $\mu_{50}$	Last epoch Erank	Final loss	Test Acc.
$0.1$	$0.01$	$0$	$0.48$	$6.54$	$0.48$	$5.53$	$5.74\times 10^{-5}$	$0.464$
$0.1$	$0.05$	$1$	$0.09$	$7.43$	$0.08$	$15.09$	$5.185\times 10^{-1}$	$0.654$
$0.1$	$0.10$	$2$	$0.03$	$7.02$	$0.04$	$3.93$	$1.34\times 10^{-6}$	$0.464$
$0.1$	$0.20$	$3$	$0.02$	$7.00$	$0.02$	$5.18$	$3.93\times 10^{-8}$	$0.464$
$0.2$	$0.01$	$4$	$1.11$	$5.74$	$0.96$	$4.61$	$8.87\times 10^{-5}$	$0.464$
$0.2$	$0.05$	$5$	$0.21$	$7.67$	$0.20$	$17.10$	$2.556\times 10^{-1}$	$0.673$

Appendix

Table 16: Detailed results for T-JEPA on Jannis dataset. Each row shows results for a specific context ratio and predictor scale configuration, with means aggregated over seeds.

Figure 13: Helena ERank trajectories at deliberately brittle configurations, where the criterion predicts collapse (mean across six runs; per-seed thin lines). The observable range is about

[1.1,9]

, since min–max preprocessing leaves a large common mean (collapse attractor near

1.1

, fresh encoder

1.3

1.9

, raw features near

8.8

); Standard lands at

1.11

1.12

on all six seeds, every ResidualPred seed above, at

1.14

3.03

, and the probe moves from

8.6\%

21.3\%

Variant	Dataset	$p_{m_{1}}$	$\alpha$	Seed	ERank 0	ERank last	Final loss	Probe acc. (%)
Standard	jannis	$0.075$	$0.20$	$0$	$1.71$	$1.27$	$4.78\!\times\!10^{-7}$	$46.01$
Standard	jannis	$0.075$	$0.20$	$1$	$1.79$	$1.29$	$2.54\!\times\!10^{-5}$	$34.38$
Standard	jannis	$0.075$	$0.20$	$2$	$1.79$	$1.29$	$6.33\!\times\!10^{-9}$	$46.01$
Standard	jannis	$0.150$	$0.50$	$0$	$1.65$	$1.28$	$2.37\!\times\!10^{-9}$	$46.01$
Standard	jannis	$0.150$	$0.50$	$1$	$1.71$	$1.28$	$1.09\!\times\!10^{-8}$	$34.38$
Standard	jannis	$0.150$	$0.50$	$2$	$1.66$	$1.27$	$5.32\!\times\!10^{-9}$	$46.01$

Appendix

Table 17: Per-seed T-JEPA + ResidualPred results on jannis and helena , on the subcritical stress grid, where the criterion places both predictors below threshold: no per-seed prevention is claimed, and the recurring probe values are class priors (

46.01\%

and

34.38\%

on jannis ). ERank values are reported at epoch

0

and at the last training epoch; final loss is the training loss at the last epoch; probe accuracy is the linear-probe test score evaluated on the frozen encoder at the last epoch.

Variant	Dataset	$p_{m_{1}}$	$\alpha$	ERank last		Final loss		Probe acc. (%)
				Mean	Std	Mean	Std	Mean	Std
Standard	jannis	$0.075$	$0.20$	$1.28$	$0.01$	$8.61\!\times\!10^{-6}$	$1.45\!\times\!10^{-5}$	$42.13$	$6.71$
Standard	jannis	$0.150$	$0.50$	$1.28$	$0.01$	$6.21\!\times\!10^{-9}$	$4.35\!\times\!10^{-9}$	$42.13$	$6.71$
Standard	jannis	$0.150$	$1.00$	$1.27$	$0.00$	$5.00\!\times\!10^{-9}$	$6.11\!\times\!10^{-9}$	$46.01$	$0.00$
Standard	helena	$0.150$	$0.50$	$1.12$	$0.00$	$3.31\!\times\!10^{-3}$	$3.81\!\times\!10^{-3}$	$\phantom{0}9.25$	$2.81$
Standard	helena	$0.150$	$1.00$	$1.12$	$0.01$	$8.79\!\times\!10^{-4}$	$1.04\!\times\!10^{-3}$	$\phantom{0}7.85$	$2.97$

Appendix

Table 18: Per-configuration aggregates (mean

\pm

standard deviation across

3

seeds) for the runs of Table 17 .

Dataset	Variant	Seed	$\mu_{50}^{(0)}$	ERank 0	ERank 100	Probe (%)
CIFAR-10	Standard	$0$	$0.0076$	$4.14$	$34.16$	$62.89$
CIFAR-10	Standard	$1$	$0.0083$	$4.29$	$40.99$	$64.44$
CIFAR-10	Standard	$2$	$0.0093$	$4.16$	$36.93$	$66.31$
CIFAR-10	+ResPred	$0$	$0.0076$	$4.14$	$50.93$	$68.78$
CIFAR-10	+ResPred	$1$	$0.0083$	$4.29$	$55.90$	$69.10$
CIFAR-10	+ResPred	$2$	$0.0093$	$4.16$	$63.09$	$67.95$

Appendix

Table 19: Per-seed I-JEPA results on CIFAR-10, CIFAR-100, and STL-10 at

p_{m_{1}}=0.85

100

epochs.

\mu_{50}^{(0)}

is the median per-mode initialisation stability ratio;

\mathrm{ERank}_{0}

\mathrm{ERank}_{100}

are the effective rank at epoch

0

and epoch

100

; Probe is the linear-probe test accuracy at epoch

100

Dataset	Variant	$n$	ERank 0		ERank 100		Probe (%)
			Mean	Std	Mean	Std	Mean	Std
CIFAR-10	Standard	$3$	$4.20$	$0.08$	$\phantom{0}37.36$	$\phantom{0}3.44$	$64.55$	$1.71$
CIFAR-10	+ResPred	$3$	$4.20$	$0.08$	$\phantom{0}56.64$	$\phantom{0}6.12$	$68.61$	$0.59$
CIFAR-100	Standard	$5$	$3.64$	$0.05$	$\phantom{0}38.96$	$\phantom{0}6.35$	$36.83$	$1.49$
CIFAR-100	+ResPred	$5$	$3.63$	$0.04$	$\phantom{0}44.28$	$\phantom{0}1.76$	$40.85$	$0.66$
STL-10	Standard	$3$	$7.08$	$0.26$	$121.93$	$\phantom{0}8.63$	$71.25$	$1.14$

Appendix

Table 20: Per-cell aggregates of Table 19 (mean

\pm

standard deviation across seeds). The Probe column reproduces the image rows of Table 4 of the main text exactly to two decimal places.

Readout	Depth 2	Depth 6
Linear probe	$+3.6$ pp	$+5.6$ pp
Nearest neighbour, $k=20$	$+8.3$ pp	$+5.5$ pp
Attentive probe	$+2.8$ pp	$+5.5$ pp
Effective rank	$+49.1$ ( $90.7\to 139.8$ )	$+26.9$

Appendix

Table 21: ImageNet-100, I-JEPA ViT-S/16,

128

pixels,

150

epochs, three seeds per predictor: ResidualPred minus the standard predictor. Every ResidualPred seed exceeds every standard seed on all three probes at both depths; exact paired permutation

p=0.031

per probe.

Readout at epoch 30	Standard	ResidualPred	$\Delta$
Linear probe	$0.2522$	$\mathbf{0.3169}$	$+6.48$ pp
Nearest neighbour, $k=20$	$0.0916$	$\mathbf{0.1774}$	$+8.58$ pp
Attentive probe	$0.2847$	$\mathbf{0.3292}$	$+4.45$ pp
Effective rank	$107.2$	$\mathbf{188.5}$	$+81.3$

Appendix

Table 22: The thousand-class pilot (ImageNet-1k, ViT-S/16,

128

pixels, depth

6

30

epochs, matched budget), seed means over five seeds per predictor. On the linear probe and effective rank every ResidualPred seed exceeds every standard seed (

p=0.008

); the nearest-neighbour and attentive readouts use the class-balanced bank.

Setting	Standard	ResidualPred	$\Delta$
$224$ pixels, $30$ epochs	$0.3398$ ( $0.3366$ to $0.3419$ )	$\mathbf{0.4014}$ ( $0.3834$ to $0.4091$ )	$+6.16$ pp
$128$ pixels, $90$ epochs	$0.3279$ ( $0.3208$ to $0.3348$ )	$\mathbf{0.3624}$ ( $0.3507$ to $0.3727$ )	$+3.45$ pp

Appendix

Table 23: The two stresses, recipe otherwise unchanged. Linear-probe top-1, seed mean and range.

224

pixels: five seeds per predictor.

90

epochs: three seeds per predictor, complete separation.

Predictor and loss	$n$	Linear	Range	Nearest neighbour	Attentive	Effective rank
Standard	5	$0.2522$	$0.2409$ to $0.2628$	$0.0916$	$0.2847$	$107.2$
Standard $+$ SIGReg, $\lambda=0.02$	2	$0.2821$	$0.2789$ to $0.2853$			$154$ to $180$
Standard $+$ SIGReg, $\lambda=0.05$ (default)	7	$0.2919$	$0.2830$ to $0.3001$	$0.1600$	$0.3100$	$179$ to $196$
Standard $+$ SIGReg, $\lambda=0.15$	2	$0.2865$	$0.2861$ to $0.2870$			$190$ to $201$
ResidualPred , $\beta_{init}=10$	5	$\mathbf{0.3169}$	$0.3143$ to $0.3203$	$0.1774$	$0.3292$	$188.5$
ResidualPred $+$ SIGReg, $\lambda=0.05$	5	$\mathbf{0.3223}$	$0.3159$ to $0.3304$	$0.1903$	$0.3341$	$205$ to $208$

Appendix

Table 24: Loss-side comparison at the pilot setting (ImageNet-1k,

128

pixels,

30

epochs, depth

6

);

n

is the number of seeds. SIGReg beats the standard predictor by

3.98

points and ResidualPred beats SIGReg by

2.50

points, each with complete separation; all

35

pairwise seed differences between ResidualPred and SIGReg are positive, from

+1.42

+3.73

points.

Mechanism	Derived effect on $\mu=\gamma/\sigma$	Predicted vs measured
Weight decay on the encoder	Raises every mode’s decay by $\eta_{wd}$ : escape iff $\gamma_{i}-\sigma_{i}>\eta_{wd}$ .	Five doses, three seeds ( $15$ runs); predicted counts $24/19/8/2/0$ , matched in every run.
Predictor scaling (Mechanism 1)	$\mu(\alpha)=\mu(1)/\alpha$ ; nothing escapes above $\alpha_{ext}=\max_{i}\mu_{i}(1)$ .	$\alpha$ ladder bracketing $\alpha_{ext}=4.16$ , three seeds ( $30$ runs); count exact in every run.
SIGReg [ Balestriero and LeCun, 2025 ]	Lowers the decay’s predictor factor $p_{\ell}$ by $\rho$ : escape iff $\mu>1-\rho/p_{\ell}$ .	Fifteen doses, three seeds ( $45$ runs), plus six teacher-free doses ( $p_{tgt}=0$ ); counts $224$ to $512$ and $[0,64,160,288,384,512]$ , exact in every run.
VICReg hinge [ Bardes et al., 2022 ]	Lowers the decay’s predictor factor by $\lambda_{v}/s$ for a mode of standard deviation $s$ ; rescue dose $\propto p_{\ell}(1-\mu)$ .	Four doses, three seeds ( $12$ runs); $12$ of $12$ exact.
Mimetic initialisation [ Trockman and Kolter, 2023 ]	Doses the alignment ratio $q_{\ell}/p_{\ell}$ once, washed out at rate $\zeta$ .	Eight wash-out rates, three seeds ( $24$ runs); $24$ of $24$ exact.
Identity predictor	With $\mathbf{W}_{p}=\mathbf{I}$ frozen, mode $k$ escapes iff $s_{k}>(1-p)/(p_{tgt}-p)$ .	Four configurations plus a BYOL-style setting, three seeds ( $15$ runs); $5/8/4/8$ and $0$ , exact.

Appendix

Table 25: Seven anti-collapse mechanisms as doses on one ratio, verified on the linear model: number of escaping modes predicted from the derived

\gamma

and

\sigma

at initialisation against the number that grows when Eq. ( 2 ) is integrated with the mechanism added. All

152

runs are simulations of the linear model, not deep networks.

Explore similar work

May 31, 2026cs.LG

UR-JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures

A central difficulty in training Joint-Embedding Predictive Architectures (JEPAs) is preventing representation collapse. LeJEPA addresses this by enforcing an isotropic Gaussian target on the embeddings via Sketched Isotropic Gaussian Regularization (SIGReg). This target is in tension with the manifold hypothesis, which expects embeddings to concentrate on a low-dimensional subset of the ambient space. We propose \emph{UR-JEPA}, which targets a uniformly

n

-rectifiable measure of local tangent dimension

n

at small scales, realized through a Gaussian-kernel smoothed Carleson-type square function

\mathcal{L}^{\text{CGLT}}

, with a complementary Jones

β

-number formulation. On Inet10, UR-JEPA(

\mathcal{L}^{\text{CGLT}}

) attains

0.9141 \pm 0.0014

for a

+0.83

,pp gain over LeJEPA(

\mathcal{L}^{\text{SIGReg}}

) with

\sim 30\%

lower seed standard deviation; on matched-recipe Galaxy10~SDSS, a single-seed ImageNet-

100

run, and a

3

-seed EuroSAT remote-sensing run, the two methods lie in the same peak-accuracy band at convergence, with UR-JEPA retaining its lower-seed-variance signature. On EuroSAT the in-domain pair is competitive at

96.0

96.1\%

with large remote-sensing foundation-model transfer at a

25\times

smaller backbone. The distinction is geometric: direct visualization of the projector output distribution shows that on all four datasets UR--JEPA(

\mathcal{L}^{\text{CGLT}}

) produces a global PCA spectrum with a

4

5

order-of-magnitude drop at index

\sim 20

25

out of

D = 32

, while LeJEPA's spectrum is near-flat (top-to-bottom ratio at most

3.6

). Per-dimension marginals are simultaneously near-Gaussian for both methods (mean Shapiro-Wilk

W \in [0.992, 0.996]

) as a Diaconis-Freedman consequence. At matched accuracy the two regularizers therefore yield structurally distinct projected representations.

Triet M. Le

Joint-Embedding Predictive Architecture Geometric Representation Learning Self-Supervised Visual Representation Learning

Ask PDF

Aug 4, 2026cs.LG

SJEPA: Learning Elegant Latent Dynamics with Hybrid Symbolic-Neural Predictors

Joint-embedding predictive architectures learn abstract states by predicting target embeddings from context embeddings, but their transition models are typically opaque neural maps. We introduce SJEPA, a reconstruction-free JEPA framework that learns predictive representations whose induced dynamics admit compact symbolic descriptions. Its hybrid transition combines a symbolic law with a regularised neural correction for dynamics outside the selected grammar. The central principle is to learn the simplest adequate dynamics: representation constraints preserve informative, non-collapsed predictive coordinates, while operator compression favours low-complexity symbolic-neural transitions that remain predictively adequate. We formalise this principle through induced-dynamics complexity, analyse predictive-coordinate non-identifiability, and show that unconstrained operator compression creates a direct shortcut to representation collapse. The framework supports both alternating representation-equation learning and symbolic dynamics fitted to fixed representations. In controlled pendulum experiments, joint learning discovers substantially simpler symbolic dynamics with lower long-horizon rollout error and divergence than post-hoc fitting, while an unconstrained one-step diagnostic realises the predicted collapse shortcut. Under grammar misspecification, correction regularisation preserves the representable symbolic mechanism and directs the neural component towards residual dynamics. The results expose a controllable trade-off among predictive fidelity, representation quality, symbolic parsimony, and symbolic-neural allocation.

Yongchao Huang

Latent Dynamics Modeling Joint-Embedding Predictive Architecture Predictive Representation Learning Neurosymbolic Learning

Ask PDF

Sep 16, 2026cs.LG

Does Joint-Embedding Predictive Architecture Pretraining Help Time Series Forecasting?

Joint-embedding predictive architectures (JEPA) have emerged as a promising self-supervised pretraining paradigm for time series, learning representations by predicting target embeddings in latent space rather than reconstructing raw signals. Yet evidence on their benefits remains mixed, and most studies test only a single backbone or a narrow set of architectures, leaving unclear whether JEPA pretraining is a reliable improvement or one that depends heavily on the downstream model. We address this gap through a large scale evaluation of one JEPA instantiation across nine backbones and eleven benchmarks spanning temporal and spatio-temporal forecasting, the most extensive cross architecture assessment of JEPA for time series to date. We find that the benefit of this instantiation varies sharply across backbones, producing consistent gains for some architectures and consistent degradation for others, even on the same dataset. This pattern holds across both task families, indicating the variability is a general property of this instantiation rather than a dataset specific artifact worth accounting for when choosing a backbone in practice.

Yutong Feng, Bowen Liao, See Kiong Ng +1

National University of Singapore · South China University of Technology · Hong Kong University of Science and Technology (Guangzhou)

Joint-Embedding Predictive Architecture Self-Supervised Time Series Representation Learning Time Series Forecasting

Ask PDF