Good Views Are Not Enough: Global Counterexamples for Gaussian-Regularized JEPAs
Abstract
Good views do not suffice to guarantee scene invariance or uniqueness up to rotation at the global optima of Gaussian-regularized JEPAs. We give globally certified counterexamples showing that the uniqueness premise of the Gaussian extension in Good Views fails for specified soft objectives. Gaussian marginal matching can reward retaining independently resampled observation variation, and this benefit can exceed its prediction cost. In a four-state model, comparison against the entire scene-only class proves that every global minimizer splits a scene and admits an equal-value partner outside its rotation orbit. In a 512-dimensional model with a complete scene basis, a spectral gap, and exact population whitening, pooled Epps-Pulley regularization still forces every global minimizer to retain nuisance and yields positive cross-modal alignment error even after the best rotation when modality nuisances are independent. Better scene alignment can also accompany worse prediction. With matched 512-label readout budgets and 1% observation noise, whitened prediction JEPA aligns better to the scene than next-token prediction but incurs 0.2872 nats/token higher linear-readout test NLL. A Bayes-risk decomposition distinguishes information lost through the observation channel from the fitted readout's difficulty using retained information. These results connect three determinants of learned representations: what views share, what regularization favors, and what a prediction task can use.
Figures & tables
| Primary setting | Whitened prediction JEPA | NTP |
|---|---|---|
| Clean best-sign scene MSE | 0.003530 | 1.130827 |
| Linear test NLL (nats/token) | 0.674001 | 0.386775 |
| MLP16 test NLL (nats/token) | 0.617590 | 0.388431 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Object | Frequency measure | Sampling and objective |
|---|---|---|
| Scalar analytic | , or ; two views, scalar outputs. | |
| Scalar quadrature | ; trapezoidal weights times | Per-view iid size ; EP averaged over two views. Mixed weight gives . |
| Pooled quadrature | ; symmetry-doubled trapezoidal weights times | Eight same-scene views, two global views, , ; , . |
| Quantity | All scene-only whitened encoders | Cube witness |
|---|---|---|
| Direction-averaged discrepancy | 0.0497120941 | 0.000000450006 |
| Expected pooled EP | 108.79885252 | 1.05385614 |
| Expected prediction | 0 | 1.25 |
| Objective | 2.17597705 | 1.27107712 |
| Quantity | Outward enclosure |
|---|---|
| Population gap | |
| Population coefficient | |
| Batch quadratic | |
| Batch quadratic | |
| Batch quadratic | |
| Scene lower bound at |
| Training | Linear, | Linear, 0.01 | MLP16, | MLP16, 0.01 |
|---|---|---|---|---|
| 32 | 0.271382 | 0.271181 | 0.229097 | 0.229478 |
| 128 | 0.290776 | 0.290745 | 0.237620 | 0.237916 |
| 512 | 0.287282 | 0.287226 | 0.228746 | 0.229160 |
| 2048 | 0.286032 | 0.285984 | 0.226737 | 0.226463 |
| Encoder seed | Primary linear difference, mean of three repeats |
|---|---|
| 0 | 0.285893 |
| 1 | 0.285844 |
| 2 | 0.285899 |
| 3 | 0.292616 |
| 4 | 0.285879 |
| Representation | Observation loss | Linear head gap | MLP16 head gap |
|---|---|---|---|
| Whitened prediction | 0.225499 | 0.059349 | 0.003077 |
| Soft pooled EP | 0.027999 | 0.165827 | 0.002191 |
| Whitening plus pooled EP | 0.019571 | 0.207490 | 0.007651 |
| NTP | 0.000365 | 0.002184 | |
| Scene-only control | 0.281874 | 0.001552 | 0.002657 |
| Task-sufficient control | 0.000301 | 0.002358 |
| No. | Lower bound on | ||||
|---|---|---|---|---|---|
| 1 | |||||
| 2 | |||||
| 3 | |||||
| 4 | |||||
| 5 | |||||
| 6 |
Joint-Embedding Predictive Architecture
Joint-Embedding Predictive Architecture (JEPA) learns representations by predicting latent features of hidden or future data from observed context.
See all 277 papers