A joint-embedding predictive architecture (JEPA) that applies prediction supervision only at masked positions struggles to clearly capture object shapes and boundaries in its latent representations. We further observe that some existing methods improve spatial structure while also retaining rich variations in texture and color. However, preserving more low-level visual information does not necessarily make representations better suited to prediction and action. This raises a key question: how can we strengthen the spatial structure in latent representations that supports prediction and action without over-encoding these details? We introduce LEF-JEPA, which learns spatial structure through target-guided latent recovery and transfers this supervision to the visual encoder through a relative-energy constraint; the energy module is used only during pretraining and adds no inference overhead. To reduce the cost of energy optimization during pretraining, we further construct a conditional Ising Hamiltonian that allows low-energy-state search to be performed by specialized Ising solvers. On physical hardware, we validate the feasibility of achieving lower forward latency with fewer local updates. Under a unified CIFAR-10 pretraining and frozen-encoder evaluation protocol, both realizations outperform V-JEPA2.1 in image classification, foreground segmentation, and visual planning.
Figures & tables
Figure 2: Overview of LEF-JEPA. (a) Visible tokens from the online encoder are partitioned into disjoint candidate and anchor sets. The anchors condition the latent energy landscape over perturbed candidate representations. Multi-level target representations both supervise the displacement induced by K energy-gradient updates ( Lfield ) and provide the target-energy reference for visual-encoder supervision ( Lenc ), together with the masked-prediction objective ( Lpred ). (b) The neural energy realization contextualizes candidate representations with multi-level anchors and aggregates the energy contribution of each token. (c) The conditional Ising realization predicts Hamiltonian coefficients (h,J) , invokes a physical photonic CIM to search for low-energy spin states, and differentiably rescores them to form a smooth scalar energy.
Figure 3: Patch-token PCA on CIFAR-10 samples. LEF-JEPA produces coherent regions aligned with object extent and boundaries. PCA is fitted independently per image, so colors are not comparable across methods.
Method
ViT-S
ViT-B
Acc. (%) ↑
Fg. IoU (%) ↑
SR (%) ↑
Acc. (%) ↑
Fg. IoU (%) ↑
SR (%) ↑
V-JEPA2 [ 4 ] (no LEF)
89.40±0.025
53.54±0.037
65
89.51±0.024
64.29±0.038
73
EB-JEPA [ 26 ]
81.49±0.040
60.29±0.089
59
85.89±0.017
71.15±0.007
72
V-JEPA2.1 [ 23 ]
83.99±0.024
65.30±0.012
53
88.77±0.005
67.47±0.045
77
LEF-JEPA-CIM
91.80±0.005
68.44±0.039
79
92.27±0.010
72.11±0.044
86
LEF-JEPA
91.63±0.016
70.04±0.040
71
92.17±0.009
71.44±0.021
84
Table 1: Frozen-encoder transfer across classification, foreground segmentation, and visual planning. Results aggregate three independently pretrained encoders; classification accuracy and foreground IoU are reported as mean ± standard deviation across encoders. Each encoder is evaluated on 100 Two-Room episodes, and SR is the mean success rate across the three experiments, rounded to the nearest percentage point. Best and second-best within each backbone are bold and underlined.
Figure 4: Spatial representations and foreground segmentation on PartImageNet. Each method is shown with patch-token PCA and foreground predictions. Green, red, and blue denote true positives, false positives, and false negatives. LEF-JEPA yields more complete masks and sharper boundaries.
Figure 5: Two-Room planning trajectories. Each column fixes the same task across methods. Yellow, green, cyan, and red denote the start, goal, executed path, and final state.
Figure 6: Effect of the number of energy-gradient updates K on pretraining latency and downstream transfer. Each CIM trajectory uses one physical solve and reuses its low-energy state pool across all K updates. CIM latency excludes cloud communication and queueing; bars report means over three independently pretrained encoders.
LEF-JEPA
LEF-JEPA-CIM
λenc
Acc. (%) ↑
Fg. IoU (%) ↑
SR (%) ↑
λenc
Acc. (%) ↑
Fg. IoU (%) ↑
SR (%) ↑
0.001
92.40±.010
69.20±.014
69
5×10−5
89.09±.026
66.67±.024
79
0.005
91.63±.016
70.04±.040
71
10−4
91.80±.005
68.44±.039
79
0.009
84.04±.050
61.11±.060
80
0.005
71.82±.054
47.19±.083
53
0.05
39.38±.070
31.66±.014
0
0.05
68.25±.032
48.83±.013
81
Table 2: Encoder-energy coefficient ablation with ViT-S. LEF-JEPA uses K=3 , whereas LEF-JEPA-CIM uses K=1 . Classification and foreground IoU are mean ± standard deviation across three independently pretrained encoders; each encoder is evaluated on 100 Two-Room episodes, and SR is averaged across the three runs and rounded to the nearest percentage point. Bold marks the best result within each realization.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Symbol
Meaning
x
Input image
Ω
Set of all patch positions
V,M
Visible and masked patch positions
Q,A
Disjoint candidate and anchor subsets of V
fθ
Online visual encoder
fθˉ
EMA-updated target encoder
pψ
Masked latent predictor
Hθ(ℓ)
Online patch tokens at encoder layer ℓ
Ns
Spatial centering followed by one image-level RMS scaling
S
Clean online candidate representation
C
Multi-level online anchor condition
T
Multi-level target representation at candidate positions
Appendix
Table S1 : Notation used in the main paper and supplement.
Objective
Online encoder θ
Predictor ψ
Energy ϕ/ω
Lpred
✓
✓
–
Lfield
–
–
✓
Lenc
✓
–
–
Appendix
Table S2 : Gradient recipients for the three training objectives. A check mark indicates that the corresponding parameters are updated by that loss.
Hyperparameter
LEF-JEPA
LEF-JEPA-CIM
Corruption magnitude σ
U(0.1,0.5)
U(0.1,0.5)
Update step size α
0.1
0.1
RMS clipping threshold τ
1.0
1.0
Reachable fraction β
0.8
0.8
Pretraining epochs
300
300
Batch size (ViT-S / ViT-B)
256/256
256/256
Appendix
Table S3 : Pretraining hyperparameters for the two energy realizations.
Component
Width
Blocks / heads
MLP
ViT-S encoder
384
12 / 6
4
ViT-B encoder
768
12 / 12
4
Masked predictor
160
2 / 8
2
Conditional energy Transformer
160
2 / 4
4
Appendix
Table S4 : Network architecture. The encoder, predictor, and energy network apply two-dimensional rotary position encoding to attention queries and keys.
Figure S1 : Effect of the number of energy-gradient updates K on pretraining latency and downstream transfer. Each CIM recovery trajectory uses one physical solve and reuses its low-energy state pool across all updates. CIM latency excludes cloud communication and queueing; bars report means over three independently pretrained encoders.
Method
Backbone
K=1
K=3
K=5
Acc. ↑
Fg. IoU ↑
SR ↑
Acc. ↑
Fg. IoU ↑
SR ↑
Acc. ↑
Fg. IoU ↑
SR ↑
LEF-JEPA
ViT-S
89.40±.025
66.88±.043
70
91.63±.016
70.04±.040
71
92.38±.004
69.71±.024
63
LEF-JEPA
ViT-B
91.00±.033
68.28±.021
85
92.17±.009
71.44±.021
84
93.05±.020
70.21±.035
82
LEF-JEPA-CIM
ViT-S
91.80±.005
68.44±.039
79
89.17±.009
69.44±.011
76
91.15±.009
68.71±.027
78
LEF-JEPA-CIM
ViT-B
92.27±.010
72.11±.044
86
92.20±.004
70.76±.021
85
89.89±.036
64.94±.075
83
Appendix
Table S5 : Complete ablation over the number of energy-gradient updates. Classification and foreground IoU are mean ± standard deviation across three independently pretrained encoders (%); each encoder is evaluated on 100 Two-Room episodes, and SR is averaged across the three encoder runs and reported to the nearest percentage point (%).