OmniSpectra: A Unified Foundation Model for Universal Spectra Representation Learning

Md Khairul Islam; Zeyu Xia; Adam S. Bolton; Judy Fox

astro-ph.IMJan 21, 2026

OmniSpectra: A Unified Foundation Model for Universal Spectra Representation Learning

Authors: Md Khairul Islam, Zeyu Xia, Adam S. Bolton, Judy Fox

Organizations: Computer Science Dept, University of Virginia · School of Data Science, University of Virginia

Abstract

Standard sequence models encode position by token rank rather than physical coordinate, typically forcing practitioners to destructively resample continuous physical signals such as spectra observed across heterogeneous instruments. We identify this coordinate mismatch as a structural limitation in cross-instrument representation learning, and propose OmniSpectra, a wavelength-native foundation model for spectral representation learning: a continuous-coordinate masked autoencoder that discards the integer pixel indexing of sequence modeling. Mapping physical coordinates to a logarithmic basis turns the multiplicative scaling induced by cosmological redshift into a uniform additive shift. In continuous rotary position embeddings, this shared shift cancels from relative token distances rather than producing grid-dependent positional distortion. We further counter the spectral bias inherent to implicit neural decoders with a physics-grounded, resolution-gated Fourier decoder that provides anti-aliasing. Pretrained on eight surveys and benchmarked against seven baselines, our model reaches state-of-the-art performance with only 12.2M parameters while outperforming substantially larger models, indicating that encoding physical structure can substitute for parameter scale.

Figures & tables

Dimension	Std Transformer	OmniSpectra
Content	Semantic tokens	Local line shape
Position Anchor	Integer index $i$	Physical $\log\lambda$ with RoPE
Mapping	Sequence steps	Continuous $\log\lambda$ axis
Attention	Distance $(i-j)$	Energy ratio $\log(\lambda_{A}/\lambda_{B})$
Decoder bias	Low-frequency bias	Resolution gating

Table 1: The core inductive biases: standard Transformer sequence modeling vs. OmniSpectra.

Figure 1: OmniSpectra architecture. The encoder extracts local morphology and places tokens in continuous physical log-wavelength ( $\log\lambda$ ) space, where redshift dilation becomes a translation. The cross-attention Fourier decoder reconstructs at native coordinates, with an LSF gate tied to the instrument’s resolution.

Panel of Figure 2 — Figure 2: (a) Pixel-index PE ( left ) gives one wavelength a different code in each survey; $\log\lambda$ PE ( right ) gives it the same code. (b) Mean pooling ( left ) weights all $P$ pixels equally, diluting a narrow feature with the continuum; attention pooling ( right , Eq. 2 ) concentrates the token on it.

Figure 3: Redshift changes index distance (a) , but adds the same offset to every

\log\lambda

coordinate (b) , shifting a windowed line profile uniformly on the log grid (c) ; continuous RoPE preserves the relative angle between tokens under this shift (d) .

Panel of Figure 4 — Figure 4: The implicit decoder. (a) Linear and Fourier decoding from one shared encoder context. (b) The same model’s ungated (left half) and gated (right half) reconstruction.

Figure 5: OmniSpectra’s four-survey reconstructions with a single set of shared weights.

Figure 6: Low-data regime: average downstream performance.

SDSS
	$T_{\mathrm{eff}}$ (K)		$\log g$ (dex)		$[\mathrm{Fe/H}]$ (dex)
Model	$R^{2}$	$\sigma^{\mathrm{MAD}}$	$R^{2}$	$\sigma^{\mathrm{MAD}}$	$R^{2}$	$\sigma^{\mathrm{MAD}}$
OmniSpectra	0.93	158	0.31	0.22	0.58	0.21
AstroCLIP	0.90	168	0.24	0.22	0.46	0.21
Shen et al.	0.89	200	0.19	0.21	0.41	0.26
Perceiver	0.90	233	0.22	0.31	0.49	0.24

Table 2: Stellar parameter estimation on SDSS (optical) and APOGEE (near-infrared). Ours is shaded ; best/second best per block in bold / underline (all tables).

	sSFR		$t_{\mathrm{age}}$		$Z_{\mathrm{MW}}$		$\log M_{*}$
Model	$R^{2}$	$\sigma^{\mathrm{MAD}}$	$R^{2}$	$\sigma^{\mathrm{MAD}}$	$R^{2}$	$\sigma^{\mathrm{MAD}}$	$R^{2}$	$\sigma^{\mathrm{MAD}}$
OmniSpectra	0.72	1.00	0.55	0.57	0.69	0.41	0.89	0.20
AION-1	0.64	1.13	0.47	0.62	0.60	0.49	0.88	0.20
AstroCLIP	0.62	1.15	0.46	0.60	0.61	0.48	0.86	0.22
Shen et al.	0.65	1.11	0.49	0.67	0.63	0.46	0.86	0.21
Perceiver	0.63	1.18	0.49	0.67	0.60	0.48	0.84	0.23

Table 3: Galaxy property estimation on DESI PROVABGS (

R^{2}

\uparrow

\sigma^{\mathrm{MAD}}

\downarrow

Model	Galaxy	Quasar	Star
OmniSpectra	0.99	0.93	0.97
AION-1	0.87	0.05	0.40
AstroCLIP	0.98	0.92	0.92
Shen et al.	0.98	0.89	0.92
Perceiver	0.98	0.89	0.93
GalSpecNet	0.94	0.54	0.45

Table 4: Per-class source classification. F1 (

\uparrow

) on SDSS.

Panel of Figure 7 — Figure 7: Transfer robustness (left) and computational efficiency (right).

Model	$R^{2}$ ( $\uparrow$ )	$\sigma^{\mathrm{MAD}}$ ( $\downarrow$ )	Outlier % ( $\downarrow$ )
OmniSpectra	0.99	0.0001	0.06
AION-1	0.99	0.01	0.07
AstroCLIP	0.98	0.01	0.06
Shen et al.	0.95	0.01	0.11
Perceiver	0.95	0.01	0.14
GaSNet-III	0.97	0.0001	0.28

Table 5: Redshift estimation on DESI PROVABGS (

Z_{\mathrm{HP}}

). Top: pretrained; bottom: supervised.

Figure 8: Redshift estimation on DESI PROVABGS.

Task Performance
Component	Base	$-$ Tok	$-$ PE	$-$ Dec	$-$ Gate	Full
Tokenizer	–	–	✓	✓	✓	✓
Positional Code	–	✓	–	✓	✓	✓
Implicit Decoder	–	✓	✓	–	✓	✓
LSF Gate	–	✓	✓	–	–	✓
Stellar $R^{2}$	0.544	0.568	0.560	0.561	0.581	0.599

Table 6: Leave-One-Component-Out Ablation. ✓ enables a component and – substitutes its standard alternative.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 798 papers

Sep 25, 2026

Self-Supervised Representation Learning: From Spectral Foundation Models to Auroral Emission Spectra

Auroral spectrographs such as the Auroral Spectrograph In Skibotn (ASIS) record hundreds of thousands of emission spectra, but only a few hundred can be labelled by an expert. To exploit the rest, we pretrain a 1D Vision Transformer with a masked autoencoder on 223,000 unlabelled spectra. Without labels, its representation recovers the emission-line intensity ratios that physicists use to diagnose the precipitating particles (R^2 0.91 vs. 0.77 for an untrained control) and, under one linear probe, classifies as well as 13 features designed by experts. Fine-tuned, the model outperforms the previous supervised auroral classifier on its own benchmark (macro-AP 88.5 vs. 77.8), reaches 0.870 mAP, and exceeds the same architecture trained from scratch by +0.159 with 10% of the labels; attribution shows that it uses both N2+ bands. Could an existing pretrained model replace it? Two astronomical spectral foundation models and a time-series model transfer according to their spectral window: SpectraFM, trained in the infrared, falls below the untrained control, whereas SpecFormer, trained in the optical, approaches in-domain pretraining without reaching it.

Matthieu Le Lain, Gaël Cessateur, Sébastien Lefèvre

IRISA, UMR CNRS 6074, Université Bretagne Sud, Vannes, France · Royal Belgian Institute for Space Aeronomy, Brussels, Belgium · UiT - The Arctic University of Norway, Tromsø, Norway

Representation Learning Self-Supervised Pre-Training Vision Transformer

Ask PDF

Sep 7, 2026

AstroSpecLM: A Spectrum-Language Model for Evidence-Grounded Astronomical Spectral Analysis

Astronomical spectra encode rich physical information, but drawing scientific conclusions from spectral features typically requires expert interpretation. This paper presents AstroSpecLM, a spectrum-language model that connects one-dimensional DESI spectra with Qwen3-4B to answer questions and provide explanations grounded in spectral evidence. Instead of generating question-answer pairs directly from templates or raw catalog fields, we first distill each spectrum into a compact set of catalog- and spectrum-derived facts, then use these facts as references to generate instruction-following conversations. The resulting model is competitive with specialist supervised baselines on classification and redshift estimation, while additionally producing natural-language explanations that reference specific spectral features. Our results indicate that grounding a language model in one-dimensional scientific spectra is feasible, and that fact-mediated instruction data yields a model capable of both prediction and explanation.

Jinghang Shi, Yanxia Zhang, Ali Luo +2

Language Modeling Scientific QA LLM Grounding Astrophysics

Ask PDF

Aug 14, 2026

Attributing Preprocessing Invariance in Spectral Foundation Models

A spectral foundation model should remain useful when laboratories preprocess spectra differently. The standard test trains a classifier under one pipeline and evaluates under another, taking preserved accuracy as evidence of learned invariance. However, these models normalize each input before any learned parameter is applied. When normalization maps differently preprocessed spectra to the same vector, the encoder receives identical inputs and the measured invariance cannot be attributed to learning. We propose a normalization-only attribution control: compare the encoder against its normalization before interpreting transfer as learned invariance. On six Raman datasets the encoder does not measurably improve transfer over its normalization. A controlled experiment confirms invariance develops only when variation reaches the encoder past normalization. Across three systems, no encoder improves relative retention over its normalization. An audit of eighteen configurations across five modalities confirms the issue is widespread: the normalization-only control should be reported before crediting transfer to the encoder.

Dongjun Wei, Hongyi Wu

ESCP Business School, Paris · OpluxCare Tech Group Ltd. · The University of Hong Kong, Hong Kong

Transfer Learning Raman Spectroscopy Spectral Methods Invariant Representation Learning

Ask PDF

Abstract

Representation Learning

Explore similar work

Self-Supervised Representation Learning: From Spectral Foundation Models to Auroral Emission Spectra

AstroSpecLM: A Spectrum-Language Model for Evidence-Grounded Astronomical Spectral Analysis

Attributing Preprocessing Invariance in Spectral Foundation Models