Authors: Nicolò Oreste Pinciroli Vago, Rocco Di Tella, Carolina Cuesta-Lázaro, Michael J. Smith, Cecilia Garraffo, Rafael Martínez-Galarza
Organizations: Department of Electronics, Information and Bioengineering, Politecnico di Milano, Via Giuseppe Ponzio, 34, Milan, 20133, MI, Italy · Osservatorio Astronomico di Roma, INAF, Via Frascati 33, Monte Porzio Catone, 00078, RM, Italy · AstroAI, Center for Astrophysics | Harvard & Smithsonian, 60 Garden Street, Cambridge, 02138, MA, United States of America · Center for Computational Astrophysics, Institute for Advanced Study/The Flatiron Institute, 162 5th Ave 9th floor, New York, 10010, NY, United States of America
Astronomers have acquired vast repositories of multimodal data, including images, spectra, and time series, complemented by decades of literature that analyzes astrophysical sources. Still, these data sources are rarely systematically integrated. This work introduces a contrastive learning framework designed to align X-ray spectra with domain knowledge extracted from scientific literature, facilitating the development of shared multimodal representations. Establishing this connection is inherently complex, as scientific texts encompass a broader and more diverse physical context than spectra. We propose a contrastive pipeline that achieves a 20% Recall@1% when retrieving texts from spectra, proving that a meaningful alignment between these modalities is not only possible but capable of accelerating the interpretation of rare or poorly understood sources. Furthermore, the resulting shared latent space effectively encodes physically significant information. By fusing spectral and textual data, we improve the estimation of 20 physical variables by 16-18% over unimodal spectral baselines. Our results indicate that a Mixture of Experts (MoE) strategy, which leverages both unimodal and shared representations, yields superior performance. Finally, outlier analysis within the multimodal latent space identifies high-priority targets for follow-up investigation, including a candidate pulsating ULX (PULX) and a gravitational lens system. Importantly, this framework can be extended to other scientific domains where aligning observational data with existing literature is possible.
Figures & tables
Variable
Description
hard_hs
Hardness ratio between hard (2.0–7.0 keV) and soft (0.5–1.2 keV) bands
hard_ms
Hardness ratio between medium (1.2–2.0 keV) and soft (0.5–1.2 keV) bands
hard_hm
Hardness ratio between hard (2.0–7.0 keV) and medium (1.2–2.0 keV) bands
var_prob_b
Gregory–Loredo variability probability
var_index_b
Intra-observation Gregory-Loredo variability index
powlaw_gamma
Photon index from power-law spectral fit
Table 1: Physical variables and their descriptions from the Chandra Source Catalog. These 20 variables serve as the ground truth for evaluating the physical interpretability of the learned latent representations and are used in the multimodal regression tasks.
Figure 1: Pipeline overview. Spectra are encoded via a transformer-based autoencoder. Scientific papers are summarized using GPT-4o-mini, and the summaries are embedded from OpenAI’s Ada-002. Contrastive learning aligns modalities into a shared latent space for downstream tasks.
Figure 2: Recall@k% as a function of k , expressed as a percentage of the test set, for the ensemble model.
Latent Dim.
Variable
Correlation
L12
hard_hs
0.82
L1
hard_hs
0.82
L51
hard_hs
0.77
L26
hard_hs
0.76
L48
apec_kt
0.74
L29
hard_hs
0.74
Table 2: Top 10 latent-variable correlations (post-alignment spectra embeddings).
Figure 3: tSNE plots for a subset of the physical variables
Variable
Mean baseline
Pre-alignment
Post-alignment
MoE
Uncertainty
Improvement
Spectra
Text
Spectra
Text
Both
hard_hs
0.40
0.20
0.27
0.16
0.17
0.12
0.12
0.01
40%
hard_ms
0.28
0.17
0.22
0.14
0.17
0.12
0.12
0.01
28%
hard_hm
0.23
0.15
0.18
0.11
0.15
0.10
0.10
0.01
33%
var_prob_b
0.26
0.27
0.18
0.26
0 .20
0.23
0.23
0.02
−25 %
var_index_b
1.77
1.79
1.00
1.70
1.19
1.45
1.00
0.10
0%
Table 3: Comparison of MAE before and after alignment, obtained using a k -NN ( k=3 ) regressor. The “Mean baseline” column reports the MAE obtained when predicting the validation set mean for all samples. The “Improvement” column indicates the relative improvement of the MoE strategy compared to the best-performing pre-alignment modality. Underlined values represent the lowest MAE (best performance), while bold values indicate results within the confidence interval of the best value.
Spectra are a prevalent yet highly information-dense form of scientific imagery, presenting substantial challenges to multimodal large language models (MLLMs) due to their unstructured and domain-specific characteristics. Here we introduce SpecVQA, a professional scientific-image benchmark for evaluating multimodal models on scientific spectral understanding, covering 7 representative spectrum types with expert-annotated question-answer pairs. The aim comprises two aspects: spectra scientific QA evaluation and corresponding underlying task evaluation. SpecVQA contains 620 figures and 3100 QA pairs curated from peer-reviewed literature, targeting both direct information extraction and domain-specific reasoning. To effectively reduce token length while preserving essential curve characteristics, we propose a spectral data sampling and interpolation reconstruction approach. Ablation studies further confirm that the approach achieves substantial performance improvements on the proposed benchmark. We test the capability of prominent MLLMs in scientific spectral understanding on our benchmark and present a leaderboard. This work represents an essential step toward enhancing spectral understanding in multimodal large models and suggests promising directions for extending visual-language models to broader scientific research and data analysis.
Existing spectral benchmarks are limited in scale, modality alignment, and evaluation scope, and typically focus on either specialized models or multimodal language models (MLLMs). We introduce SpecX, a large-scale benchmark for multi-modal spectroscopy with cross-paradigm evaluation. SpecX contains 1.7M molecules with diverse spectral modalities, including NMR (1H, 13C, HSQC), IR, MS,UV,Raman and FL, and is organized into three tiers: a large-scale dataset for pretraining, an aligned multi-spectral subset for benchmarking, and a high-quality experimental subset for evaluation. SpecX supports a range of tasks such as molecular elucidation, spectrum simulation, and spectral understanding, and enables unified evaluation across both specialized spectral models and MLLMs. Experiments show that specialized models excel at signal-level modeling, while MLLMs exhibit strengths in high-level reasoning but lack precise spectral grounding. SpecX establishes a unified benchmark for spectral intelligence and highlights the need for spectrum-native foundation models.
Chengrui Xiang, Tengfei Ma, Yujie Chen +3
College of Computer Science and Technology, Hunan University, Changsha, China
The communication of scientific knowledge has become increasingly multimodal, spanning text, visuals, and speech through materials such as research papers, slides, and recorded presentations. These different representations collectively convey a study's reasoning, results, and insights, offering complementary perspectives that enrich understanding. However, despite their shared purpose, such materials are rarely connected in a structured way. The absence of explicit links across formats makes it difficult to trace how concepts, visuals, and explanations correspond, limiting unified exploration and analysis of research content. To address this gap, we introduce the Multimodal Conference Dataset (MCD), the first benchmark that integrates research papers, presentation videos, explanatory videos, and slides from the same works. We evaluate a range of embedding-based and vision-language models to assess their ability to discover fine-grained cross-format correspondences, establishing the first systematic benchmark for this task. Our results show that vision-language models are robust but struggle with fine-grained alignment, while embedding-based models capture text-visual correspondences well but equations and symbolic content form distinct clusters in the embedding space. These findings highlight both the strengths and limitations of current approaches and point to key directions for future research in multimodal scientific understanding. To ensure reproducibility, we release the resources for MCD at https://github.com/meghamariamkm2002/MCD
Megha Mariam K. M, Vineeth N. Balasubramanian, C. V. Jawahar
IIIT Hyderabad · Microsoft Research India & IIT Hyderabad