JEPA for astronomy: multi-survey spectral alignment puts stellar parameters on the leading principal axes
Authors: Xiaosheng Zhao, Rosemary F. G. Wyse, Yuan-Sen Ting, Alexander S. Szalay
Organizations: William H. Miller III Department of Physics & Astronomy, The Johns Hopkins University, Baltimore, MD 21218, USA · Department of Astronomy, The Ohio State University, 140 West 18th Avenue, Columbus, OH 43210, USA · Center for Cosmology and AstroParticle Physics (CCAPP), The Ohio State University, Columbus, OH 43210, USA · Max-Planck-Institut für Astronomie, Königstuhl 17, D-69117 Heidelberg, Germany · Asia School of Business, No. 1 & 11, Jalan Dato Onn, 50480 Kuala Lumpur, Malaysia · Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21218, USA
Spectra from different surveys carry different systematics and are not calibrated against one another. Multi-survey foundation models for stellar spectra are then built by aligning embeddings of the same star observed across surveys. Most in astronomy follow the recipes established for language and vision, aligning with a contrastive loss that separates every star from the others. However, this does not ensure that the stellar physics is encoded in a form that a simple read-out can recover. Holding the cross-survey architecture fixed and varying only the alignment loss, we measure what an objective encodes from the principal-component geometry of the Gaia BP/RP (XP) spectra and from a linear probe of the stellar parameters. In a properly aligned embedding, the leading two principal components form a plane that recovers an analog of the Kiel diagram, the third component tracks [Fe/H], and [alpha/Fe] varies coherently across the plane, such that the chemically defined Galactic disks and accreted populations occupy distinct loci. We show that legibility, the ease of reading the information out, is highest for a training objective that never pushes stars apart, and instead constrains only the overall distribution of each survey's embeddings (JEPA). JEPA also exceeds the contrastive alternative at the same architecture, and the performance improves further as more surveys (Gaia XP+LAMOST LRS+DESI+APOGEE) are included. Predicted abundances for 8.0 million XP giants reproduce the main [alpha/Fe]-[Fe/H] structure of the Milky Way. These results indicate that the choice of alignment objective, not alignment alone, determines whether stellar labels are linearly recoverable.
Figures & tables
Figure 1: Multi-survey alignment. Embeddings from reconstructive autoencoders are mapped by a survey-specific head into a shared embedding space. Alignment is performed in a subsequent projection space (not shown for simplicity), which is discarded after training. Since the views share the same star, the common information is dominated by the underlying physical state (center), while instrument-specific information is suppressed. PCA and linear probes are then applied to aligned embeddings from a specific survey (e.g., Gaia).
Survey pair
Matched pairs
Used in
Gaia XP
LAMOST LRS
820,568
all models
Gaia XP
APOGEE
454,355
four-survey
Gaia XP
DESI
313,212
four-survey
DESI
LAMOST LRS
260,063
four-survey
APOGEE
LAMOST LRS
145,195
four-survey
APOGEE
DESI
16,732
four-survey
Table 1: Cross-matched observations used for alignment, and the labeled sample used to evaluate the Gaia XP embeddings.
Figure 2: Linear legibility of [Fe/H]. Each panel shows the first two principal components, PC1 and PC2, colored by PC3, from embeddings of two alignment models (JEPA, top left; CLIP, bottom left) and one unaligned autoencoder (AE, bottom right). The APOGEE Kiel diagram colored by [Fe/H] (top right) is shown as the physical anchor. Components are computed independently for each panel and are in arbitrary units. The same 40,000 stars are shown throughout. Reconstruction alone (AE) leaves [Fe/H] unordered. In contrast, alignment recovers the ordering, and the objective determines how sharply it is recovered (JEPA vs CLIP).
parameter
N
JEPA (4 survey)
CLIP (4 survey)
JEPA (2 survey)
CLIP (2 survey)
AE
ridge / MLP
ridge / MLP
ridge / MLP
ridge / MLP
ridge / MLP
Teff
200
0.93 / 0.97
0.86 / 0.96
0.94 / 0.96
0.68 / 0.97
0.74 / 0.93
10,000
0.95 / 0.98
0.93 / 0.98
0.96 / 0.99
0.93 / 0.98
0.95 / 0.98
logg
200
0.95 / 0.96
0.93 / 0.95
0.93 / 0.95
0.88 / 0.94
0.70 / 0.90
10,000
0.98 / 0.98
0.97 / 0.98
0.97 / 0.98
0.96 / 0.98
0.94 / 0.96
[Fe/H]
200
0.90 / 0.91
0.87 / 0.90
0.86 / 0.90
0.82 / 0.88
0.69 / 0.75
Table 2: Recovery of stellar parameters from each 768-d embedding, at two labeled-sample sizes.
Figure 3: Gravity and chemistry encoding. The same PC1–PC2 plane as in Figure 2 is shown for the JEPA model, colored by surface gravity logg (left), by [ α /Fe] (middle), and by chemical category (right, where gray marks unclassified stars and thin/thick/accreted/VMP/low- α & MP contain 46,627/19,340/9,292/143/5,470 stars). The plane is ordered by surface gravity. The chemical populations occupy distinct bands whose [ α /Fe] matches their chemical definition.
Figure 4: The [ α /Fe]–[Fe/H] plane of 8.0 million Gaia XP red giant stars ( Teff<5300 K, logg<3.5 ) with median per-pixel S/N ≥64 and full phase-space information, in bins of Galactocentric radius R in the midplane and height ∣z∣ . The elemental abundances are predicted from the aligned XP embedding of the JEPA model. The main chemical structures of the MW are recovered from resolving power ∼50 XP spectra alone.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure B1: Architecture of four-survey JEPA. Each survey spectrum is encoded by a frozen reconstructive backbone (an overcomplete autoencoder for Gaia XP, masked transformers for LAMOST LRS and DESI, and a wavelength-aware autoencoder built on the universal spectral tokenizer for APOGEE; labeled AE, MT and UST, respectively) and is then mapped onto a common 768-dimensional representation by a trainable head: an MLP for the single XP latent vector, and cross-attention pooling for the token sequences of the other three surveys. A single rectified projector, shared by all four surveys, maps these representations into a 128-dimensional non-negative space. A mean-squared invariance term aligns the representations across the six cross-matched survey edges, pulling different views of the same star towards a common point. A per-survey rectified distribution-matching regularization (RDMReg) prevents collapse constraint without requiring negative pairs. Only the heads and the shared projector are trained (33.8 M parameters, against 142 M held frozen). All downstream probes read the head output before the projector (dashed arrow), so the recovered stellar parameters come from the 768-dimensional head representation rather than from the loss-space projection.
Latent space alignment maps learned representations into compatible coordinates to enable comparison, transfer, and interoperability across models or data sources.
In this paper we investigate whether geometrically aligned astronomical representations are also scientifically interchangeable. We analyze every layer's representation from 36 pretrained models using cross-matched optical images, infrared images, and spectra. All 139 analyzed model-survey comparisons show significant local-neighborhood alignment somewhere in the network after accounting for the search over depth. However, alignment does not generally increase with network depth and can be stronger for the changes between consecutive computational blocks than for the block outputs themselves. We find that geometry-selected stages under-perform label-selected stages in every image-transfer comparison; moreover, every final HSC-COSMOS-Web model pair is aligned while every redshift transfer has negative R^2. Models can therefore recover a similar cross-survey geometry among astronomical objects without recovering a survey-invariant linear encoding of the physical properties tested here.
Kshitij Duraphe, Aravind Kannappan, Dun Li Chan +4
UniverseTBD · EleutherAI · INTI International College Penang, Malaysia +4
Cross-survey generalization is a critical challenge in stellar spectral analysis, particularly in cases such as transferring from low- to moderate-resolution surveys. We investigate this problem using pre-trained models, focusing on simple neural networks such as multilayer perceptrons (MLPs), with a case study transferring from LAMOST low-resolution spectra (LRS) to DESI medium-resolution spectra (MRS). Specifically, we pre-train MLPs on either LRS or their embeddings and fine-tune them for application to DESI stellar spectra. We compare MLPs trained directly on spectra with those trained on embeddings derived from transformer-based models (self-supervised foundation models pre-trained for multiple downstream tasks). We also evaluate different fine-tuning strategies, including residual-head fine-tuning, LoRA, and full fine-tuning. We find that MLPs pre-trained on LAMOST LRS achieve strong performance, even without fine-tuning, and that modest fine-tuning with DESI spectra further improves the results. For iron abundance, embeddings from a transformer-based model yield advantages in the metal-rich ([Fe/H] > -1.0) regime, but underperform in the metal-poor regime compared to MLPs trained directly on LRS. We also show that the optimal fine-tuning strategy depends on the specific stellar parameter under consideration. These results highlight that simple pre-trained MLPs can provide competitive cross-survey generalization, while the role of spectral foundation models for cross-survey stellar parameter estimation requires further exploration.
Xiaosheng Zhao, Yuan-Sen Ting, Rosemary F. G. Wyse +5
Department of Physics & Astronomy, The Johns Hopkins University, Baltimore, MD 21218, USA · Department of Astronomy, The Ohio State University, 140 West 18th Avenue, Columbus, OH 43210, USA · Center for Cosmology and AstroParticle Physics (CCAPP), The Ohio State University, Columbus, OH 43210, USA +6
Good views do not suffice to guarantee scene invariance or uniqueness up to rotation at the global optima of Gaussian-regularized JEPAs. We give globally certified counterexamples showing that the uniqueness premise of the Gaussian extension in Good Views fails for specified soft objectives. Gaussian marginal matching can reward retaining independently resampled observation variation, and this benefit can exceed its prediction cost. In a four-state model, comparison against the entire scene-only class proves that every global minimizer splits a scene and admits an equal-value partner outside its rotation orbit. In a 512-dimensional model with a complete scene basis, a spectral gap, and exact population whitening, pooled Epps-Pulley regularization still forces every global minimizer to retain nuisance and yields positive cross-modal alignment error even after the best rotation when modality nuisances are independent. Better scene alignment can also accompany worse prediction. With matched 512-label readout budgets and 1% observation noise, whitened prediction JEPA aligns better to the scene than next-token prediction but incurs 0.2872 nats/token higher linear-readout test NLL. A Bayes-risk decomposition distinguishes information lost through the observation channel from the fitted readout's difficulty using retained information. These results connect three determinants of learned representations: what views share, what regularization favors, and what a prediction task can use.