OmniSpectra: A Unified Foundation Model for Universal Spectra Representation Learning

Md Khairul Islam; Zeyu Xia; Adam S. Bolton; Judy Fox

astro-ph.IMJan 21, 2026

OmniSpectra: A Unified Foundation Model for Universal Spectra Representation Learning

Authors: Md Khairul Islam, Zeyu Xia, Adam S. Bolton, Judy Fox

Organizations: Computer Science Dept, University of Virginia · School of Data Science, University of Virginia

Abstract

Standard sequence models encode position by token rank rather than physical coordinate, typically forcing practitioners to destructively resample continuous physical signals such as spectra observed across heterogeneous instruments. We identify this coordinate mismatch as a structural limitation in cross-instrument representation learning, and propose OmniSpectra, a wavelength-native foundation model for spectral representation learning: a continuous-coordinate masked autoencoder that discards the integer pixel indexing of sequence modeling. Mapping physical coordinates to a logarithmic basis turns the multiplicative scaling induced by cosmological redshift into a uniform additive shift. In continuous rotary position embeddings, this shared shift cancels from relative token distances rather than producing grid-dependent positional distortion. We further counter the spectral bias inherent to implicit neural decoders with a physics-grounded, resolution-gated Fourier decoder that provides anti-aliasing. Pretrained on eight surveys and benchmarked against seven baselines, our model reaches state-of-the-art performance with only 12.2M parameters while outperforming substantially larger models, indicating that encoding physical structure can substitute for parameter scale.

Figures & tables

Dimension	Std Transformer	OmniSpectra
Content	Semantic tokens	Local line shape
Position Anchor	Integer index $i$	Physical $\log\lambda$ with RoPE
Mapping	Sequence steps	Continuous $\log\lambda$ axis
Attention	Distance $(i-j)$	Energy ratio $\log(\lambda_{A}/\lambda_{B})$
Decoder bias	Low-frequency bias	Resolution gating

Table 1: The core inductive biases: standard Transformer sequence modeling vs. OmniSpectra.

Figure 1: OmniSpectra architecture. The encoder extracts local morphology and places tokens in continuous physical log-wavelength ( $\log\lambda$ ) space, where redshift dilation becomes a translation. The cross-attention Fourier decoder reconstructs at native coordinates, with an LSF gate tied to the instrument’s resolution.

Panel of Figure 2 — Figure 2: (a) Pixel-index PE ( left ) gives one wavelength a different code in each survey; $\log\lambda$ PE ( right ) gives it the same code. (b) Mean pooling ( left ) weights all $P$ pixels equally, diluting a narrow feature with the continuum; attention pooling ( right , Eq. 2 ) concentrates the token on it.

Figure 3: Redshift changes index distance (a) , but adds the same offset to every

\log\lambda

coordinate (b) , shifting a windowed line profile uniformly on the log grid (c) ; continuous RoPE preserves the relative angle between tokens under this shift (d) .

Panel of Figure 4 — Figure 4: The implicit decoder. (a) Linear and Fourier decoding from one shared encoder context. (b) The same model’s ungated (left half) and gated (right half) reconstruction.

Figure 5: OmniSpectra’s four-survey reconstructions with a single set of shared weights.

Figure 6: Low-data regime: average downstream performance.

SDSS
	$T_{\mathrm{eff}}$ (K)		$\log g$ (dex)		$[\mathrm{Fe/H}]$ (dex)
Model	$R^{2}$	$\sigma^{\mathrm{MAD}}$	$R^{2}$	$\sigma^{\mathrm{MAD}}$	$R^{2}$	$\sigma^{\mathrm{MAD}}$
OmniSpectra	0.93	158	0.31	0.22	0.58	0.21
AstroCLIP	0.90	168	0.24	0.22	0.46	0.21
Shen et al.	0.89	200	0.19	0.21	0.41	0.26
Perceiver	0.90	233	0.22	0.31	0.49	0.24

Table 2: Stellar parameter estimation on SDSS (optical) and APOGEE (near-infrared). Ours is shaded ; best/second best per block in bold / underline (all tables).

	sSFR		$t_{\mathrm{age}}$		$Z_{\mathrm{MW}}$		$\log M_{*}$
Model	$R^{2}$	$\sigma^{\mathrm{MAD}}$	$R^{2}$	$\sigma^{\mathrm{MAD}}$	$R^{2}$	$\sigma^{\mathrm{MAD}}$	$R^{2}$	$\sigma^{\mathrm{MAD}}$
OmniSpectra	0.72	1.00	0.55	0.57	0.69	0.41	0.89	0.20
AION-1	0.64	1.13	0.47	0.62	0.60	0.49	0.88	0.20
AstroCLIP	0.62	1.15	0.46	0.60	0.61	0.48	0.86	0.22
Shen et al.	0.65	1.11	0.49	0.67	0.63	0.46	0.86	0.21
Perceiver	0.63	1.18	0.49	0.67	0.60	0.48	0.84	0.23

Table 3: Galaxy property estimation on DESI PROVABGS (

R^{2}

\uparrow

\sigma^{\mathrm{MAD}}

\downarrow

Model	Galaxy	Quasar	Star
OmniSpectra	0.99	0.93	0.97
AION-1	0.87	0.05	0.40
AstroCLIP	0.98	0.92	0.92
Shen et al.	0.98	0.89	0.92
Perceiver	0.98	0.89	0.93
GalSpecNet	0.94	0.54	0.45

Table 4: Per-class source classification. F1 (

\uparrow

) on SDSS.

Panel of Figure 7 — Figure 7: Transfer robustness (left) and computational efficiency (right).

Model	$R^{2}$ ( $\uparrow$ )	$\sigma^{\mathrm{MAD}}$ ( $\downarrow$ )	Outlier % ( $\downarrow$ )
OmniSpectra	0.99	0.0001	0.06
AION-1	0.99	0.01	0.07
AstroCLIP	0.98	0.01	0.06
Shen et al.	0.95	0.01	0.11
Perceiver	0.95	0.01	0.14
GaSNet-III	0.97	0.0001	0.28

Table 5: Redshift estimation on DESI PROVABGS (

Z_{\mathrm{HP}}

). Top: pretrained; bottom: supervised.

Figure 8: Redshift estimation on DESI PROVABGS.

Task Performance
Component	Base	$-$ Tok	$-$ PE	$-$ Dec	$-$ Gate	Full
Tokenizer	–	–	✓	✓	✓	✓
Positional Code	–	✓	–	✓	✓	✓
Implicit Decoder	–	✓	✓	–	✓	✓
LSF Gate	–	✓	✓	–	–	✓
Stellar $R^{2}$	0.544	0.568	0.560	0.561	0.581	0.599

Table 6: Leave-One-Component-Out Ablation. ✓ enables a component and – substitutes its standard alternative.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 799 papers

Self-Supervised Representation Learning: From Spectral Foundation Models to Auroral Emission Spectra
Sep 25, 2026Matthieu Le Lain, Gaël Cessateur, Sébastien LefèvreRepresentation Learning Self-Supervised Pre-Training
IRISA, UMR CNRS 6074, Université Bretagne Sud, Vannes, France · Royal Belgian Institute for Space Aeronomy, Brussels, Belgium · UiT - The Arctic University of Norway, Tromsø, Norway
AstroSpecLM: A Spectrum-Language Model for Evidence-Grounded Astronomical Spectral Analysis
Sep 7, 2026Jinghang Shi, Yanxia Zhang, Ali Luo +2Language Modeling Scientific QA
Attributing Preprocessing Invariance in Spectral Foundation Models
Aug 14, 2026Dongjun Wei, Hongyi WuTransfer Learning Raman Spectroscopy
ESCP Business School, Paris · OpluxCare Tech Group Ltd. · The University of Hong Kong, Hong Kong

Sep 25, 2026J/K move · Enter open · S save

Self-Supervised Representation Learning: From Spectral Foundation Models to Auroral Emission Spectra

Matthieu Le Lain, Gaël Cessateur, Sébastien Lefèvre

IRISA, UMR CNRS 6074, Université Bretagne Sud, Vannes, France · Royal Belgian Institute for Space Aeronomy, Brussels, Belgium · UiT - The Arctic University of Norway, Tromsø, Norway

Representation Learning Self-Supervised Pre-Training Vision Transformer

Open paper AskPDF arXiv

Auroral spectrographs such as the Auroral Spectrograph In Skibotn (ASIS) record hundreds of thousands of emission spectra, but only a few hundred can be labelled by an expert. To exploit the rest, we pretrain a 1D Vision Transformer with a masked autoencoder on 223,000 unlabelled spectra. Without labels, its representation recovers the emission-line intensity ratios that physicists use to diagnose the precipitating particles (R^2 0.91 vs. 0.77 for an untrained control) and, under one linear probe, classifies as well as 13 features designed by experts. Fine-tuned, the model outperforms the previous supervised auroral classifier on its own benchmark (macro-AP 88.5 vs. 77.8), reaches 0.870 mAP, and exceeds the same architecture trained from scratch by +0.159 with 10% of the labels; attribution shows that it uses both N2+ bands. Could an existing pretrained model replace it? Two astronomical spectral foundation models and a time-series model transfer according to their spectral window: SpectraFM, trained in the infrared, falls below the untrained control, whereas SpecFormer, trained in the optical, approaches in-domain pretraining without reaching it.

Abstract

Representation Learning

Explore similar work

Self-Supervised Representation Learning: From Spectral Foundation Models to Auroral Emission Spectra

AstroSpecLM: A Spectrum-Language Model for Evidence-Grounded Astronomical Spectral Analysis

Attributing Preprocessing Invariance in Spectral Foundation Models