SADL: On the Demand-Supply Geometry of Representation Collapse
Organizations: State Key Laboratory of CAD&CG, Zhejiang University · RoboParty Lab · Alibaba Group · AIR, Tsinghua University ·
Abstract
In Joint-Embedding Predictive Architectures (JEPAs), a well-constrained projector output does not by itself prevent representation collapse. We find that, from initialization, the encoder representation is dominated by a component shared by different inputs. A batch-normalized projector removes this component, so regularization after the projector cannot correct it, and the differences between representations shrink until they collapse. We propose SADL (Supply And Demand Loss), which keeps the original training objective and constrains the encoder directly when needed. SADL compares the representation differences that the projector output demands with the differences the encoder can retain and pass on to the projector, and adds a constraint on the encoder only when supply falls short of demand. It works with different Demand Regularizers and introduces no auxiliary networks, additional trainable parameters, or architectural changes. In single-task and jointly trained four-task LeWorldModel (LeWM) and INTACT experiments, SADL lowers the collapse rate over ten random seeds from 50% to 0%. After one epoch of training, encoder effective rank is 3.4-6.4 times that of the corresponding baseline without SADL, and latent prediction error falls to 1/20-1/303 of it. Without additional tuning, SADL stays robust as its loss weight spans orders of magnitude, and with several Demand Regularizers INTACT exceeds 95% mean search-free control success. On four image datasets, SADL also consistently improves accuracy and stability across random seeds. These results show that matching the representation differences a regularizer demands with the differences the encoder can supply leads to faster and more stable representation learning in control and image classification.
Figures & tables
| Model | Setting | SR | N/O/D/C | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| LeWM | initialization | – | ||||||||
| MATH, -only | ||||||||||
| Flash, no | ||||||||||
| Flash, -only | ||||||||||
| Flash, + SADL | ||||||||||
| INTACT | initialization | – |
| Ablation cell | Weight range | Weights | N/O/D/C | Best SR | Worst seed |
| Supply term | |||||
| SADL | 1 | ||||
| tr-only | 1 | ||||
| SADL | – | 2 | ( ) | ( ) | |
| exc-only | – | 2 | ( ) | ( ) | |
| Gate and potential | |||||
| SIGReg control | VISReg control | Image accuracy | |||||
|---|---|---|---|---|---|---|---|
| Demand | Ep. 1 | Ep. 2 | Ep. 1 | Ep. 2 | Imagenette | ImageNet-LT | CIFAR-10 |
| baseline | – | – | – | – | |||
| Model | |||||||||||||
| Single-task | |||||||||||||
| LeWM | SIGReg | ||||||||||||
| VISReg | |||||||||||||
| QQWorld | |||||||||||||
| INTACT | SIGReg | ||||||||||||
| VISReg | |||||||||||||
| Model | Target | PushT | Cube | Reacher | TwoRoom | Macro | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| LeWM | G | SIGReg | 0.0114 | 56.00 | 67.11 | 82.33 | 80.78 | 1.289 | .1772 | |
| VISReg | 1.5000 | 47.33 | 68.44 | 81.00 | 89.56 | 1.184 | .1717 | |||
| QQWorld | 1.2729 | 60.89 | 68.00 | 80.33 | 88.22 | 1.179 | .1699 | |||
| Energy-Gaussian | 2.3383 | 49.44 | 61.56 | 79.44 | 90.56 | 1.161 | .1712 | |||
| NG | VIC moments | 1.9130 | 35.56 | 62.00 | 79.67 | 92.89 | 1.308 | .2228 | ||
| Sphere | 1.1318 | 73.11 | 59.56 | 81.89 | 86.78 | 1.189 | .1660 |
Appendix figures & tables49 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning |
|---|---|
| ; | Input batch of aligned samples; encoder and projector. |
| , | Encoder representation and endpoint (the projector output), , ; are their sample rows. |
| , | Jacobians of the projector and of the encoder. |
| Endpoint predicted by the forward model. | |
| Mean, centered features, centered covariance, and uncentered mean energy of . | |
| Fraction of energy carried by sample variation, . |
| Parameter | Value |
|---|---|
| Demand interpolation weight | |
| Capacity parameter | |
| Ridge coefficient | |
| Numerical floor and log smoothing | |
| Supply Regularizer weight | |
| SIGReg |
| Configuration | Setting |
|---|---|
| Tasks | PushT, Cube, Reacher, TwoRoom |
| Training seeds | 0, 42, 3072 |
| Encoder | ViT-Tiny, trained from scratch |
| Encoder representation dimension | 192 |
| Input image size | |
| Patch size |
| Task | Moderate | Strict |
|---|---|---|
| PushT | Joint agent-and-block position error native pixel units; block angle error ; hold one step. | Block position error ; block angle error ; hold three steps. |
| Cube | Position error m; no orientation constraint; hold one step. | Position error m and cube-symmetry-aware orientation error ; hold three steps. |
| Reacher | Maximum joint error rad; periodic shoulder and unwrapped elbow error; hold one step. | End-effector distance m; hold two steps. |
| TwoRoom | Goal distance native units; hold one step. | Goal distance , valid collision-free route, doorway crossing, and correct goal room; hold one step. |
| Visual encoder | Latent predictor | Bitwise match | First divergence | Max. parameter gap at step 100 |
|---|---|---|---|---|
| MATH | Flash | Not observed | ||
| Flash | MATH | Step 1, | – |
| Run | Encoder MATH layers | |||
|---|---|---|---|---|
| A | All 12 | 21.13 | 50.18 | 49.14 |
| A | None (all Flash) | 4.65 | 7.65 | 7.74 |
| A | 1–6 | 10.58 | 20.72 | 23.00 |
| A | 7–12 | 20.62 | 51.22 | 51.32 |
| B | All 12 | 26.84 | 66.63 | 64.78 |
| B | None (all Flash) | 13.80 | 14.86 | 4.73 |
| Encoder | Constraint | |||
|---|---|---|---|---|
| MATH | SIGReg | 21.13 | 50.18 | 49.14 |
| Flash | SIGReg | 4.65 | 7.65 | 7.74 |
| Flash | barrier , no constraint | 27.90 | 9.67 | 2.96 |
| Flash | barrier + SIGReg | 2.12 | 6.14 | 3.33 |
| Training condition | Paired rank-drop events | Prediction loss | Official SR | |
|---|---|---|---|---|
| MATH + SIGReg (reference) | – | |||
| Flash, no Demand Regularizer | ||||
| Flash + SIGReg | ||||
| Flash + SIGReg + exc-only SADL | ||||
| Flash + SIGReg + SADL |
| Setting | Ep. 1/Ep. 2 SR | Ep. 2 by seed | N/O/D/C |
|---|---|---|---|
| LeWM, MATH, -only | 32.02/48.85 (cohort) | .168, .157, .0933, .168, .220, .177, .126, .219, .0949, .118 | |
| LeWM, Flash, no | 24.11/23.03 (cohort) | , , , .00138, , , , , , | |
| LeWM, Flash, -only | 31.92/39.96 (cohort) | .000239, .149, , .000395, .0177, .139, .0283, .287, .0984, .0928 | |
| LeWM, Flash, + SADL | 64.97/69.20 (cohort) | .990, .990, .989, .990, .990, .986, .989, .990, .989, .989 | |
| INTACT, MATH, -only | 49.50/76.92; 39.17/67.17; 32.75/65.83 | .074, .0767, .114 | |
| INTACT, Flash, -only | 43.42/92.33; 48.50/79.00; 45.33/27.00 | .0532, .0714, .00193 |
| Condition | Threshold | dropped | Encoder earlier / same / later / from start | Encoder only |
|---|---|---|---|---|
| SIGReg ( -only), Flash | 16 | 5 / 7 / 0 / 4 | 5 | |
| SIGReg ( -only), Flash | 4 | 0 / 3 / 0 / 1 | 9 | |
| SIGReg with MATH; exc-only; SADL | both | 0 | – | 0 |
| Model | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LeWM | SIGReg s42 | ||||||||||||
| SIGReg s3072 | |||||||||||||
| QQWorld s3072 | |||||||||||||
| INTACT | SIGReg s0 | ||||||||||||
| SIGReg s3072 | |||||||||||||
| QQWorld s3072 | |||||||||||||
| Method | Epoch | PushT | Cube | Reacher | TwoRoom | Macro | |
|---|---|---|---|---|---|---|---|
| SIGReg | -only | Ep. 1 | |||||
| SIGReg | +SADL | Ep. 1 | |||||
| VISReg | -only | Ep. 1 | |||||
| VISReg | +SADL | Ep. 1 | |||||
| QQWorld | -only | Ep. 1 | |||||
| QQWorld | +SADL | Ep. 1 |
| Method | Epoch | PushT | Cube | Reacher | TwoRoom | Macro | |
|---|---|---|---|---|---|---|---|
| SIGReg | -only | Ep. 1 | |||||
| SIGReg | +SADL | Ep. 1 | |||||
| VISReg | -only | Ep. 1 | |||||
| VISReg | +SADL | Ep. 1 | |||||
| QQWorld | -only | Ep. 1 | |||||
| QQWorld | +SADL | Ep. 1 |
| Method | Epoch | PushT | Cube | Reacher | TwoRoom | Macro | |
|---|---|---|---|---|---|---|---|
| SIGReg | -only | Ep. 1 | |||||
| SIGReg | +SADL | Ep. 1 | |||||
| SIGReg | -only | Ep. 5 | |||||
| SIGReg | +SADL | Ep. 5 | |||||
| SIGReg | -only | Ep. 10 | |||||
| VISReg | -only | Ep. 1 |
| Method | Epoch | PushT | Cube | Reacher | TwoRoom | Macro | |
|---|---|---|---|---|---|---|---|
| SIGReg | -only | Ep. 1 | |||||
| SIGReg | +SADL | Ep. 1 | |||||
| SIGReg | -only | Ep. 5 | |||||
| SIGReg | +SADL | Ep. 5 | |||||
| SIGReg | -only | Ep. 10 | |||||
| VISReg | -only | Ep. 1 |
| Criterion | Method | Epoch | PushT | Cube | Reacher | TwoRoom | Macro | |
|---|---|---|---|---|---|---|---|---|
| Moderate | SIGReg | -only | Ep. 1 | |||||
| SIGReg | +SADL | Ep. 1 | ||||||
| VISReg | -only | Ep. 1 | ||||||
| VISReg | +SADL | Ep. 1 | ||||||
| QQWorld | -only | Ep. 1 | ||||||
| QQWorld | +SADL | Ep. 1 |
| Criterion | Method | Epoch | PushT | Cube | Reacher | TwoRoom | Macro | |
|---|---|---|---|---|---|---|---|---|
| Moderate | SIGReg | -only | Ep. 1 | |||||
| SIGReg | +SADL | Ep. 1 | ||||||
| VISReg | -only | Ep. 1 | ||||||
| VISReg | +SADL | Ep. 1 | ||||||
| QQWorld | -only | Ep. 1 | ||||||
| QQWorld | +SADL | Ep. 1 |
| Method | Epoch | PushT | Cube | Reacher | TwoRoom | Macro | |
|---|---|---|---|---|---|---|---|
| SIGReg | -only | Ep. 1 | |||||
| SIGReg | +SADL | Ep. 1 | |||||
| SIGReg | -only | Ep. 5 | |||||
| SIGReg | +SADL | Ep. 5 | |||||
| SIGReg | -only | Ep. 10 | |||||
| VISReg | -only | Ep. 1 |
| Method | Epoch | PushT | Cube | Reacher | TwoRoom | Macro | |
|---|---|---|---|---|---|---|---|
| SIGReg | -only | Ep. 1 | |||||
| SIGReg | +SADL | Ep. 1 | |||||
| SIGReg | -only | Ep. 5 | |||||
| SIGReg | +SADL | Ep. 5 | |||||
| SIGReg | -only | Ep. 10 | |||||
| VISReg | -only | Ep. 1 |
| Method | Epoch | PushT | Cube | Reacher | TwoRoom | Macro | |
|---|---|---|---|---|---|---|---|
| SIGReg | -only | Ep. 1 | |||||
| SIGReg | +SADL | Ep. 1 | |||||
| SIGReg | -only | Ep. 5 | |||||
| SIGReg | +SADL | Ep. 5 | |||||
| SIGReg | -only | Ep. 10 | |||||
| VISReg | -only | Ep. 1 |
| Method | Epoch | PushT | Cube | Reacher | TwoRoom | Macro | |
|---|---|---|---|---|---|---|---|
| SIGReg | -only | Ep. 1 | |||||
| SIGReg | +SADL | Ep. 1 | |||||
| SIGReg | -only | Ep. 5 | |||||
| SIGReg | +SADL | Ep. 5 | |||||
| SIGReg | -only | Ep. 10 | |||||
| VISReg | -only | Ep. 1 |
| Configuration | INTACT Mod. | INTACT Strict | LeWM Mod. | LeWM Strict |
|---|---|---|---|---|
| exc-only | ||||
| tr-only | ||||
| -only | ||||
| SADL |
| Potential | Epoch | Moderate | Strict |
|---|---|---|---|
| LeWM | |||
| Canonical | Ep. 1 | ||
| Two-sided | Ep. 1 | ||
| Linear | Ep. 1 | ||
| Canonical | Ep. 2 | ||
| Two-sided | Ep. 2 | ||
| Setting | E | (%) | SR (%) | |||||
| LeWM | ||||||||
| Endpoint † | 10 | .8049 | .6541 | .7174 | .3177 | .6948 | 100.0 | |
| -only | 2 | .0000 | .0781 | .0000 | .0439 | .1858 | 0.0 | |
| Trans. | 2 | .5416 | .3150 | .3886 | .2578 | .5062 | 41.7 | |
| Two-sided | 2 | .9496 | .5307 | .7098 | .5882 | .6944 | 100.0 | |
| Linear | 2 | .9479 | .5439 | .7179 | .6640 | .7287 | 100.0 | |
| Configuration | (%) | SR (%) | ||||
| LeWM | ||||||
| Endpoint † | .8180 | .25432 | .00756 | 3.843 | -.0251 | |
| -only | .0000 | .00092 | .00000 | .000 | n/a | |
| Trans. | .0233 | 15.74698 | .00023 | .017 | .0987 | |
| Two-sided | .1886 | .16440 | .00196 | 1.763 | .0815 | |
| Linear | .0790 | .14240 | .00408 | 3.697 | .1885 | |
| (A) Controlled capacity optimization, 20 seeds | ||||
|---|---|---|---|---|
| Initial deficit | Supply Regularizer target | , 0 200 | , 0 200 | , 0 200 |
| Excitation | tr-only | |||
| Excitation | SADL | |||
| Transmission | exc-only | |||
| Transmission | SADL | |||
| Configuration | Ep. 20 demand | Ep. 100 Top-1 (%) | vs. baseline (pp) |
|---|---|---|---|
| VISReg baseline | n/a | 22.034 | 0.000 |
| SADL | .772 | 22.904 | |
| SADL | .772 | 22.348 | |
| SADL | .667 | 23.026 | |
| SADL | .554 | 22.144 | |
| SADL | .460 | 22.064 |
| Model | Recipe | Ep. 1 PushT | Ep. 1 Cube | Ep. 1 Reacher | Ep. 1 TwoRoom | Ep. 1 Macro | Late Macro | |
|---|---|---|---|---|---|---|---|---|
| LeWM | SIGReg | Dual+floor | ||||||
| SADL | ||||||||
| VISReg | Dual+floor | |||||||
| SADL | ||||||||
| QQWorld | Dual+floor | |||||||
| SADL |
| A. Representation geometry: mean seed SD | |||||
|---|---|---|---|---|---|
| (%) | |||||
| SIGReg | |||||
| VISReg | |||||
| QQWorld | |||||
| Energy-Gaussian | |||||
| VIC moments | |||||
| SIGReg | ||||
|---|---|---|---|---|
| VISReg | ||||
| QQWorld | ||||
| Energy-Gaussian | ||||
| VIC moments | ||||
| Sphere |
| PushT 0 | Cube 0 | Reacher 0 | TwoRoom 0 | Macro 0 | Macro SADL | pp | |
|---|---|---|---|---|---|---|---|
| SIGReg | 3.44 | 49.89 | 20.00 | 36.44 | |||
| VISReg | 53.00 | 55.11 | 27.33 | 55.33 | |||
| QQWorld | 19.22 | 52.56 | 18.78 | 40.33 | |||
| Energy-Gaussian | 3.78 | 45.11 | 14.78 | 36.89 | |||
| VIC moments | 5.67 | 47.44 | 15.56 | 29.56 | |||
| Sphere | 34.33 | 54.33 | 23.22 | 47.11 |
| Dataset ( ) | Epoch | Baseline | +SADL | Wins | ||
|---|---|---|---|---|---|---|
| ImageNet-LT (3) | VISReg | 100 | 3/3 | |||
| Galaxy10 (3) | VISReg | 400 | 3/3 | |||
| CIFAR-10 (10) | SIGReg | 1 | 10/10 | |||
| 10 | 9/10 | |||||
| 50 | 9/10 | |||||
| 100 | 4/10 |
| Batch | Backend and path | Results using the batch | |
|---|---|---|---|
| Matched backend | MATH/Flash, original INTACT backend | .03 | Table 40 ; direct regularizer scan |
| Fast ablation | Flash, optimized attention backend | .03 | transmission sweep and sensitivity tables |
| Ten-seed audit | Flash/MATH LeWM diagnostic recipe | .03 | paired rank-drop, , and temporal diagnostics |
| Mechanism audit | Flash, mechanism-audit configuration | .03 | tr-only, gate and potential rows of Table 2 ; Table 46 |
| Historical control | MATH -only, Flash + SADL | .03 | Appendix E ; long-training reference |
| Model | Backend | -only Ep. 1 | SADL Ep. 1 | -only Ep. 2 | SADL Ep. 2 |
|---|---|---|---|---|---|
| INTACT | MATH | ||||
| Flash | |||||
| LeWM | MATH | ||||
| Flash |
| Setting | SR (Ep. 1) | N/O/D/C | |||||||
|---|---|---|---|---|---|---|---|---|---|
| LeWM, MATH, + SADL |
| Ep. 1 Reacher | Ep. 1 TwoRoom | Ep. 2 Reacher | Ep. 2 TwoRoom | |
|---|---|---|---|---|
| 0.01 | ||||
| 0.1 | ||||
| 0.3 | ||||
| 1.0 |
| Stage | PushT | Cube | Reacher | TwoRoom |
|---|---|---|---|---|
| Ep. 1, steps 0–2k | ||||
| Ep. 1, steps 4k–6k | ||||
| Ep. 2, steps 4k–6k |
| Demand | Imagenette | ImageNet-LT | CIFAR-10 |
|---|---|---|---|
| Baseline (no SADL) | |||
| Constant | |||
| Constant | |||
| Constant | |||
| Constant | |||
| Constant |
| Term on | Weight | Ep. 1 SR |
|---|---|---|
| SADL, | 0.01 (default) | (95.25) |
| 0.03 | (94.33) | |
| 0.1 | (93.17) | |
| 0.3 | (93.50) | |
| 1 | (91.17) | |
| MCR 2 expansion | 0.03 | (23.17) |
| Variant | Weight or audit setting | Ep. 1 SR (best / worst) | N/O/D/C |
|---|---|---|---|
| SADL | – | ||
| exc-only | |||
| tr-only | mechanism audit, | ||
| Two-sided (no gate) | mechanism audit | ||
| One-sided linear | mechanism audit | ||
| -only | mechanism audit |
| Setting | Ep. 1 SR | Ep. 2 SR | selected, Ep. 1/2 | , Ep. 2 |
|---|---|---|---|---|
| Default ( ) | 0 / 0 | .736–.752 | ||
| 0 / 0 | .735–.751 | |||
| .37 / .43 | .733–.753 | |||
| 0 / 0 | .771–.781 | |||
| 0 / 0 | .718–.740 |
| Ridge | Ep. 1 SR | Ep. 2 SR | selected, Ep. 1/2 |
|---|---|---|---|
| 0 / .01 | |||
| (default) | .33 / .35 | ||
| 1.00 / .97 |
| Normalization | ImageNet-LT | Galaxy10, epoch 400 | CIFAR-10 | Imagenette, epoch 800 |
|---|---|---|---|---|
| Original: | +.794 (3/3) | +.348 (3/3) | +.026 (4/10) | +.444 (6/7) |
| Finite batch: | +.694 | +.470 (2/3) | +.186 (7/10) | +.401 (6/7) |
| Dataset | |||||
|---|---|---|---|---|---|
| Imagenette | baseline | ||||
| SADL | |||||
| speed-up | |||||
| ImageNet-LT | baseline | reached by 2 of 3 seeds | |||
| SADL | (3 of 3 seeds) | ||||
| speed-up | only SADL reaches it on every seed |
| Model / workload | Step B | Step N | Step P | Peak B/N/P |
| H20 (H20-3e) | ||||
| LeWM / PushT | 21.49/21.65/12.10 | |||
| LeWM / 4 tasks | 22.16/22.17/12.35 | |||
| INTACT / PushT | 21.51/21.67/12.73 | |||
| INTACT / 4 tasks | 22.57/22.58/12.62 | |||
| H200 | ||||
Representation Collapse
Representation collapse is the loss of diversity in learned features, causing distinct inputs to map to similar or low-dimensional embeddings.
See all 54 papers