Cross-modal translation requires models to capture shared semantics across modalities while preserving the fine-grained structure required by the target modality. Yet in real-world scientific domains, paired data are often imperfect, while obtaining clean pairs at scale is costly. Supervised fine-tuning (SFT) directly optimizes cross-modal translation through categorical token-level supervision, but this objective is sensitive to errors in the target sequences. JEPA offers a complementary representation-space objective, but most existing formulations focus on prediction within a modality rather than translation across modalities. We introduce Anchored Cross-Modal JEPA (AC-JEPA), an SFT-compatible plug-in that predicts masked intermediate states in both cross-modal directions using paired data. The joint-embedding objective targets cross-modal structure predictable from context, while the causal SFT path retains direct token-level supervision for the target sequence. During self-training, AC-JEPA learns from continuous latent targets derived from imperfect pseudo-labeled pairs, offering intrinsic tolerance to local data errors without promoting pseudo-tokens to hard categorical supervision. Using optical chemical structure recognition as a testbed, AC-JEPA trained on clean pairs achieves 76.07% accuracy on the Simple set of the WildMol benchmark, compared with 67.40% for SFT alone. Using the same pseudo-labeled data for both approaches, AC-JEPA-based self-training further raises accuracy to 80.23%, whereas accuracy under direct SFT drops from 67.40% to 63.47%. These results support SFT-anchored joint-embedding prediction as an effective strategy for cross-modal translation under both curated and imperfect supervision.
Figures & tables
Figure 1: Macro view of AC-JEPA. Paired image and SMILES views are processed by one image-to-SMILES VLM, while SFT anchors causal generation and the JEPA objective regularizes intermediate hidden states through latent MSE and SIGReg. The latent self-training loop is shown only as an abstract training extension; data-source and curriculum details are separated from this core mechanism view.
Figure 2: Model structure and training losses of AC-JEPA. SFT supervises the causal decoder, while JEPA predicts intermediate latents under structured masking. The clean latent is detached only as the MSE target, while SIGReg regularizes the undetached clean image and SMILES latents separately.
Figure 3: Training curriculum. Base SFT first learns from the complete synthetic MolParser-7M corpus and is then fine-tuned on Real-SFT 100K. After JEPA-module warm-up with that backbone frozen, gold-pair joint training and latent self-training start independently from the shared checkpoint. Gold-pair joint training reuses the already-seen 7M pairs for latent prediction while retaining Real-SFT for token supervision. Self-training adds open-environment images through model-generated pseudo-SMILES while gold-pair SFT remains active as the anchor; pseudo-label sets are refreshed between rounds.
Method
Simple Acc. (%)
Markush Acc. (%)
Published reference
MolParser-Base
76.90
38.10
Our curriculum
Base SFT
67.40
36.48
Gold-pair AC-JEPA
76.07
42.28
AC-JEPA self-training, round 1
78.51
42.78
Table 1: Accuracy (%) across the AC-JEPA curriculum on WildMol Simple and Markush. Every row in our curriculum inherits the same MolParser-7M pretraining and Real-SFT fine-tuning; gold-pair AC-JEPA reuses existing pairs through JEPA. MolParser-Base is a published reference from Fang et al. [8] , with a different architecture and optimization recipe.
Figure 4: Component losses from three independent AC-JEPA runs initialized from the same CPT-trained, JEPA-warmed checkpoint. Curves show every 100 local optimization steps for gold-pair training (6,000 steps), self-training R1 (4,500 steps), and self-training R2 (4,000 steps). The blocks are juxtaposed for compact comparison and are not connected across runs. All curves report unweighted loss components.
Pseudo-label route
Simple
Markush
Direct pseudo-label SFT
63.47
36.04
AC-JEPA self-training, round 1
78.51
42.78
AC-JEPA self-training, round 2
80.23
48.28
Table 2: Accuracy (%) when model-generated pseudo-labels are used for direct token-level SFT or AC-JEPA latent self-training.
Objective
ρseq
Simple
Markush
Direct SFT
20%
67.08
21.85
Direct SFT
50%
66.03
21.62
Direct SFT
80%
51.37
4.45
AC-JEPA joint training
20%
73.67
45.98
AC-JEPA joint training
50%
73.26
45.45
AC-JEPA joint training
80%
73.07
43.01
Table 3: Accuracy (%) under controlled SMILES corruption. ρseq denotes the fraction of corrupted MolGrapher labels; 5–15% of characters are perturbed in each selected label.
Latent-interface layer
Simple Acc. (%)
12
76.07
17
77.67
23
75.61
26
71.31
Table 4: Preliminary Simple accuracy (%) for the recorded decoder-layer choices used as the JEPA latent interface.
Masked modalities
Ratio
Simple Acc. (%)
Image + SMILES
0.20
75.20
Image + SMILES
0.35
76.07
Image + SMILES
0.50
74.60
SMILES only
0.35
73.12
Image only
0.35
70.54
Table 5: Preliminary Simple accuracy (%) for the recorded masking-ratio and masked-modality variants.
LeJEPA derives an isotropic Gaussian embedding target by minimizing, under its task assumptions, a Fisher information criterion that controls the task-averaged squared prediction bias. This enables unimodal self-supervised learning without reconstruction decoders, exponential-moving-average teachers, or contrastive negatives. However, its isotropic task prior does not account for the cross-modal structure of multimodal tasks that read shared content. We introduce MM-LeJEPA by extending LeJEPA's task assumptions to multimodal learning. Specifically, we model multimodal tasks as strength-weighted combinations of components that read independent information from the two modalities and components that read the same content from both. Under explicit gradient-moment, correspondence, and symmetry assumptions, we derive a correlated Gaussian target that minimizes a weighted Fisher information functional under a fixed total variance constraint. An analytic transform adapts Sketched Isotropic Gaussian Regularization (SIGReg) to this target, yielding a two-term objective that combines distributional regularization with local-to-global prediction. The extension adds a single scalar hyperparameter, the target cross-modal correlation, and retains the simplicity of LeJEPA. On public audio-visual classification benchmarks, our results are competitive with representative open-source methods. Code will be released.
Haowen Bai, Rui Gong$, Meichen Liu +3
Nanyang Technological University · Hyundai Motor Group Innovation Center Singapore (HMGICS)
Large-scale self-supervised pretraining has reshaped modern machine learning, substantially advancing the ability of language and vision models to generalize across downstream tasks. While deep learning has driven considerable progress in modeling atomistic systems in recent years, self-supervised pretraining in this domain has not yet achieved comparable downstream generalization. To address this, we introduce Atom-JEPA, a self-supervised pretraining framework that learns latent representations from unlabeled 3D structures through complementary atom-level and substructure-level objectives inspired by joint-embedding predictive architectures. We pretrain Atom-JEPA on large-scale molecular and crystalline datasets and evaluate its transfer performance by fine-tuning on a diverse set of downstream property prediction tasks. Atom-JEPA achieves state-of-the-art performance on molecular ADMET and quantum-chemical property prediction tasks, and is highly competitive in predicting the physical properties of crystalline materials. These results demonstrate the potential of latent-space predictive pretraining to support broad downstream generalization from structural data alone. Code and pretrained model checkpoints are publicly available at https://github.com/khelverskovp/atom-jepa
Kasper Helverskov Petersen, Rasmus Hannibal Tirsgaard, François R J Cornet +2
Department of Applied Mathematics and Computer Science Technical University of Denmark
Despite recent advances in molecular foundation models, several limitations remain, such as chemically invalid augmentations, modality collapse, and incomplete representation of biochemical environments. To address these challenges, we present Mol-JEPA, a scalable framework for learning molecular world models. Rather than relying on suboptimal molecular perturbations, our model uses modality masking to exploit information from molecular structures, cellular phenotypes, binding affinities, ADMET profiles, quantum chemistry simulations and other drug discovery data. Across various benchmarks, we show that the representations learned by Mol-JEPA deliver strong performance, demonstrating the value of incorporating biochemical context through latent space prediction.
Florian Rottach, Sebastian Schieferdecker, William Rudman +2
University of Tübingen · 2Boehringer Ingelheim · 3The University of Texas at Austin +1