Predictive Closure: A Set-Level Characterization of JEPA's Feature Selection

Mosh Levy; Aly Lidayan; Yoav Goldberg

cs.LGOct 19, 2026

Predictive Closure: A Set-Level Characterization of JEPA's Feature Selection

Authors: Mosh Levy, Aly Lidayan, Yoav Goldberg

Organizations: Bar-Ilan University · University of California, Berkeley · Allen Institute for AI

Abstract

In joint-embedding predictive architectures (JEPA), an encoder and a predictor are trained together to produce a representation that minimizes the error in predicting the representation of the next observation. Because the representations of both the current and the next observation are learned, it is unclear which features they will include. Earlier characterizations describe the features JEPA selects in terms of their individual predictability, raising the concern that unpredictable features, like a traffic light, will not be encoded into the representation. However, the prediction error is also affected by the relationships between features, e.g., the light's color helps predict whether the traffic will stop. We propose the predictive closure characterization, which accounts for these relationships. Based on a reformulation of the JEPA training objective, we show that selection depends on properties of feature sets. This view agrees with earlier works when features evolve independently, but makes new predictions when they do not, for instance that a feature like the traffic light can still be retained. We test this characterization in controlled experiments, which show that unpredictable features can become better represented when they help predict other features, and that changing one feature's predictive relationships can affect how well others are represented. We also demonstrate implications for steering feature selection: targeting one feature with an auxiliary objective, or supplying it to the predictor as side information, can strengthen the representation of another feature whose future values it helps predict, so steering can be indirect.

Figures & tables

Figure 1: Will the traffic light be included in the latent representation? The top row shows successive observations, and the rows below show their latent representations with the light kept or dropped; crosses and dashed frames mark the resulting prediction errors. In the first step, the light changes color at random, so keeping it costs an error on its own next color. In the second step, the light’s current color helps predict whether the cyclist will ride or stop next, so dropping the light costs an error on the cyclist. Earlier characterizations judge the light only by whether its own next color can be predicted, and so would drop it; they disregard what its current color contributes to predicting the cyclist’s next move, which is what can make the light worth keeping. Cyclist artwork adapted from Scott McCloud’s Understanding Comics (1993).

Figure 2: Cross-subsidization: the unpredictable light becomes more decodable when it helps predict the images. Increasing

q

makes the current light more useful for predicting which images persist, while its future remains unpredictable. Dashed line:

50\%

chance accuracy.

Figure 3: Set switching: changing predictive relationships reverses which pair is more strongly represented. Increasing

p_{D}

strengthens decoding of

C/D

and weakens

A/B

, whose predictive relationships remain fixed; neither pair helps predict the other. Dashed line:

50\%

chance.

Figure 4: Feasibility dependence: adding images to a cycle can reverse which images’ features are more strongly represented. Mean decoding accuracy across the competing cycle’s positions decreases as

N

grows, despite unchanged average copying probabilities; at

N=16

, the fixed

A/B

pair is better represented. Dashed line:

50\%

chance.

Figure 5: Reconstructing one image can strengthen representation of its predictive partner. The changes are not uniformly monotonic, and the baseline already favors

C/D

at high

p_{D}

. Each cell uses one optimization seed; all panels use the same

50

–

90\%

color scale.

Figure 6: Supplying

C

to the predictor can strengthen representation of its predictive partner

D

. Data and probe tasks are matched; probes use only the main latent.

D

becomes more represented for

p_{D}\geq 0.75

and less at

p_{D}=0.60

, where

C

is least often copied to

D

;

C

becomes less represented at the highest

p_{D}

. Dotted line:

50\%

chance.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	No side information		Side information
Setting	Prediction	Shuffled	Prediction	Shuffled
Traffic light ( $q=1$ )	$77.13\pm 0.51$	$49.97\pm 0.43$	—	—
Two pairs, $p_{D}=0.60$	$88.88\pm 0.35$	$50.10\pm 0.38$	$91.27\pm 0.19$	$49.95\pm 0.28$
Two pairs, $p_{D}=0.75$	$90.65\pm 0.14$	$50.27\pm 0.23$	$93.64\pm 0.04$	$50.13\pm 0.11$
Two pairs, $p_{D}=0.85$	$92.22\pm 0.08$	$50.56\pm 0.16$	$95.39\pm 0.07$	$49.70\pm 0.06$
Two pairs, $p_{D}=0.95$	$94.53\pm 0.36$	$49.83\pm 0.33$	$97.15\pm 0.04$	$49.59\pm 0.07$

Appendix

Table 1: Predictors distinguish actual next observations from incompatible alternatives. Values are percentage correct, reported as mean

\pm

standard error across three training runs. Chance is

50\%

for choosing between the actual next observation and the alternative. Each system is evaluated in its own learned latent space. The side-information condition supplies the current

C

image while keeping it visible in the main input.

Figure 7: Cross-subsidization. The current light becomes more strongly represented as

q

increases, although stronger regularization weakens this effect. Dashed lines mark

50\%

chance light-decoding accuracy.

Figure 8: Changing predictability. Increasing

p_{D}

from

0.60

1

changes the better-represented pair from

A/B

C/D

in all three runs at every regularization weight. Stronger regularization reduces the separation between the pairs. The change need not be monotonic: at

r=0.20

, mean decoding accuracy for

C/D

decreases from

p_{D}=0.95

p_{D}=1

. Dashed lines mark

50\%

chance accuracy.

Figure 9: Changing group size. Enlarging the

C/D

group reduces its decoding accuracy in every run at every regularization weight. At

N=16

A/B

becomes better represented in all three runs at

r=0.10,0.20

, but only one of three at

r=0.05

. We average decoding accuracy across positions within each group; dashed lines mark

50\%

chance accuracy.

Figure 10: Auxiliary reconstruction. Reconstructing

C

can also strengthen its predictive partner

D

, which receives no reconstruction supervision. At

\beta=0.36

C/D

is better represented than

A/B

at every

p_{D}

and regularization weight. Increasing

\beta

does not always improve decoding: at

r=0.20,p_{D}=0.75

, both

C

and

D

decline from

\beta=0.135

\beta=0.36

. All four images remain visible to the encoder. Cells show decoding accuracy from one training run, using a shared

50

–

90\%

color scale; numerical values below

50\%

remain visible.

Figure 11: Predictor side information. Supplying

C

strengthens representation of

A

and

D

while weakening

B

p_{D}=0.75,0.85

in all three runs at every regularization weight. The effect varies across the sweep and does not uniformly replace the

A/B

pair with

C/D

. Colored curves supply

C

to the predictor; gray curves omit side input. All probes read the main latent without access to the side input; supplying

C

also adds encoder capacity. Panels show decoding accuracy with standard errors across three runs; dotted lines mark

50\%

chance accuracy.

Figure 12: Worst- and best-position readouts show the same reversal. Both measures favor the competing cycle at the smallest size and

A/B

at the largest, showing that the reversal also holds for the least and most decodable positions within each group. Curves show means across three training runs; error bars show standard errors. Probes are trained on collages from images held out from encoder training and evaluated on separate collages. The dashed line marks the chance image-detection accuracy of

50\%

Joint-Embedding Predictive Architecture

Joint-Embedding Predictive Architecture (JEPA) learns representations by predicting latent features of hidden or future data from observed context.

See all 245 papers

Jun 29, 2026

Predictive Objectives Discard Exogenous Control-Relevant Features: A Controlled Mechanistic Study

Joint-embedding predictive (JEPA-style) objectives learn representations by predicting future latents. In doing so they can discard features that are exogenous (uncontrollable by the agent) yet control-relevant, even when those features are trivially encodable. This occurs because the objective optimizes temporal predictability rather than control-relevance. We isolate this failure mode in a controlled 2x2 experimental design that varies feature controllability and relevance independently, using a predictability knob that decouples a feature's temporal predictability from its control-relevance. Comparing six objectives: reconstruction, JEPA, action-conditioned JEPA, controllability-based JEPA, inverse dynamics under a random policy, and reward-grounded JEPA, we observe that all evaluated reward-free predictive objectives leave the exogenous control-relevant feature near chance accuracy, while a reward-grounded variant retains it selectively. The remedy is label-efficient and robust: as little as 2% of reward-labeled transitions recovers the feature, the effect holds across two environments with different surface forms, and it persists across latent dimensions from 16 to 1024. Comparing the learned latent geometry against bisimulation theory's prediction, the JEPA latent realizes only a small fraction of the class separation a supervised reference attains.

Ayan Pendharkar

Representation Learning Joint-Embedding Predictive Architecture Predictive Representation Learning State Representation Learning

Ask PDF

Oct 1, 2026

Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures

Joint-Embedding Predictive Architectures (JEPAs) are prone to representation collapse, typically mitigated through empirical heuristics. We develop an early-training stability theory that unifies these heuristics. Linearising the coupled JEPA gradient flow around the trivial fixed point reveals two competing effects: a driving force (

γ

) and a decay effect (

σ

). Under approximate spectral decoupling, a per-mode stability ratio

μ_i = γ_i / σ_i

factorises into independent data-side and predictor-side terms and the count of unstable modes tracks the rank of representations that can emerge. The framework predicts a phase boundary, which we confirm empirically across more than 800 Tabular-JEPA configurations. It also unifies predictor scaling, masking ratio, and EMA as distinct mechanisms for shifting

μ

. Guided by this analysis, we introduce ResidualPred, a transformer predictor whose attention is biased toward the identity at initialisation; it improves both effective rank and downstream accuracy on tabular benchmarks and in I-JEPA pretraining on CIFAR-10, CIFAR-100, STL-10, and ImageNet. Our framework connects empirical collapse-avoidance heuristics to an explicit dynamical picture, yielding theory-driven stabilizers. Code is available at https://github.com/jose-melo/drive-vs-decay.

José Lucas De Melo Costa, Seong Woo Ahn, Fabrice Popineau +2

Université Paris-Saclay, CNRS, CentraleSupélec, LISN, Orsay, France

Joint-Embedding Predictive Architecture Representation Collapse Self-Supervised Pre-Training

Ask PDF

Mar 14, 2026

Is the reconstruction loss culprit? An attempt to outperform JEPA

We evaluate JEPA-style predictive representation learning versus reconstruction-based autoencoders on a controlled "TV-series" linear dynamical system with known latent state and a single noise parameter. While an initial comparison suggests JEPA is markedly more robust to noise, further diagnostics show that autoencoder failures are strongly influenced by asymmetries in objectives and by bottleneck/component-selection effects (confirmed by PCA baselines). Motivated by these findings, we introduce gated predictive autoencoders that learn to select predictable components, mimicking the beneficial feature-selection behavior observed in over-parameterized PCA. On this toy testbed, the proposed gated model is stable across noise levels and matches or outperforms JEPA.

Alexey Potapov, Oleg Shcherbakov, Ivan Kravchenko

SingularityNET

Predictive Representation Learning Autoencoders Joint-Embedding Predictive Architecture Feature Selection

Ask PDF

Abstract

Joint-Embedding Predictive Architecture

Explore similar work

Predictive Objectives Discard Exogenous Control-Relevant Features: A Controlled Mechanistic Study

Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures

Is the reconstruction loss culprit? An attempt to outperform JEPA