The Infinite-Dimensional Nature of Spectroscopy and Why Models Succeed, Fail, and Mislead

Umberto Michelucci; Francesca Venturini

cs.LGApr 6, 2026

The Infinite-Dimensional Nature of Spectroscopy and Why Models Succeed, Fail, and Mislead

Authors: Umberto Michelucci, Francesca Venturini

Organizations: Department of Computer Science Lucerne University of Applied Sciences and Arts Risch-Rotkreuz, 6343 Switzerland · Institute of Applied Mathematics and Physics ZHAW Zurich University of Applied Sciences Winterthur, 8400 Switzerland

Abstract

Machine learning (ML) models have achieved strikingly high accuracies in spectroscopic classification tasks, often without a clear proof that those models used chemically meaningful features. Existing studies have linked these results to data preprocessing choices, noise sensitivity, and model complexity, but no unifying explanation is available so far. In this work, we show that these phenomena arise naturally from the intrinsic high dimensionality of spectral data. Using a theoretical analysis grounded in the Feldman-Hajek theorem and the concentration of measure, we show that even infinitesimal distributional differences, caused by noise, normalisation, or instrumental artefacts, may become perfectly separable in high-dimensional spaces. Through a series of specific experiments on synthetic and real fluorescence spectra, we illustrate how models can achieve near-perfect accuracy even when chemical distinctions are absent, and why feature-importance maps may highlight spectrally irrelevant regions. We provide a rigorous theoretical framework, confirm the effect experimentally, and conclude with practical recommendations for building and interpreting ML models in spectroscopy.

Figures & tables

Figure 1: Illustration of the concentration of measure for multivariate Gaussian distributions. Shown are the empirical distributions of $\|x\|_{2}$ for samples drawn from $\mathcal{N}(0,\,1.0^{2}I_{n})$ (light blue) and $\mathcal{N}(0,\,1.1^{2}I_{n})$ (yellow), for increasing dimensionalities $n=2$ , $50$ , $500$ , and $5000$ (panels A–D). In low dimensions the two distributions overlap substantially, but as $n$ increases the probability mass concentrates sharply around the typical radius $\sigma\sqrt{n}$ , and even small variance differences cause almost complete separation. This illustrates how, in high-dimensional spaces, measures supported by Gaussian (and many non-Gaussian) distributions become effectively disjoint, providing an intuitive geometric basis for the Feldman–Hájek theorem and for the sensitivity of high-dimensional classifiers to minute statistical differences.

ID	Experiment	Experiment Details
N1	Gaussian noise: $\Delta\sigma$ sweep (QDA)	Classify white noise from two multivariate Gaussians with equal means ( $\mu_{1}=\mu_{2}=1$ ) and different isotropic and Toeplitz (with parameter $\rho=0.95$ ) covariances; we varied the variance gap $\Delta\sigma:=\|\sigma_{2}-\sigma_{1}\|$ from 0 to 2 while keeping $\sigma_{1}=1$ fixed. We have tested $n=\{5,10,50,500\}$ . We have measured classification accuracy.
N2	Gaussian noise: Bayes “oracle” boundary	Classify white noise from two multivariate Gaussians with equal means ( $\mu_{1}=\mu_{2}=1$ ) and different isotropic covariances; we varied the variance gap $\Delta\sigma:=\|\sigma_{2}-\sigma_{1}\|$ from 0 to 2 while keeping $\sigma_{1}=1$ fixed. We have tested $n=\{30,100,500,1000,5000\}$ . We classify white noise by using the ideal threshold on $\sum(x_{j}-\mu)^{2}$ .
N3	Gaussian noise: accuracy vs. dimension $n$	Classify white noise from two multivariate Gaussians with equal means ( $\mu_{1}=\mu_{2}=0$ ) and different isotropic covariances; we varied the variance gap in $\Delta\sigma:=\|\sigma_{2}-\sigma_{1}\|=\{0.1,0.3,0.6,0.9,1.2,1.5,2.0\}$ while keeping $\sigma_{1}=1$ fixed. We tested $n=\{1,5,10,20,50,100,200,500,1000,2000,5000\}$ . We calculate the accuracy of QDA for various values of $n$ .
N4	Skew–normal noise: 2D parameter sweeps	Classify two classes of white noise from multivariate skew–normal Azzalini and Valle (1996) . For this experiment, we used the parameters $n=50$ , $\mu_{1}=\mu_{2}=10$ , $\sigma_{1}=1$ , $\gamma_{1}=0.5$ . We then varied $\Delta\sigma/\sigma$ from 0 to 2, $\Delta\mu/\mu$ from 0 to 0.15, and $\Delta\gamma_{1}/\gamma_{1}$ from 0 to 8.
S1	Synthetic spectra: truly identical classes	Classify two classes of spectra each composed of one Lorentzian (both centres chosen according to a normal distribution $\mathcal{N}(50,10^{2})$ ) and an FWHM $\xi=7$ . We generated $N=500$ spectra for each class. We choose $n=\{5,10,50,100,1000,2000,5000,10000\}$ . These two sets of spectra are not distinguishable, as there are no differences in the data distributions.
S2	Synthetic spectra: FWHM difference	Classify two classes of spectra each composed of one Lorentzian (both centres chosen accordingly to a normal distribution $\mathcal{N}(50,10^{2})$ ) and two different FWHM $\xi_{1}=7$ and $\xi_{2}=9$ . We generated $N=500$ spectra for each class. We choose $n=\{5,10,50,100,1000,2000,5000,10000\}$ We study accuracy of classification for various value of $n$ .

Table 1: Overview of experiments and their core ideas.

Figure 2: Ten representative synthetic one-peak spectra per class used in the synthetic-spectra experiments. Each curve is a Lorentzian profile sampled on an

n=100

-point axis, with peak centre jittered as

c\sim\mathcal{N}(50,10^{2})

. Class 0 (blue) and Class 1 (orange) differ only through the FWHM

\xi_{1}=7

vs.

\xi_{2}=9

, illustrating that the two classes are visually difficult to distinguish despite being statistically separable in high dimension.

Figure 3: Fluorescence spectra of Spanish olive oil samples classified as Extra Virgin (EVOO), Virgin (VOO), and Lampante (LOO). The region 380–420 nm indicates the Rayleigh scattering peak from the excitation LED. The black line indicates the average spectrum for each class.

ID	Experiment	Key finding	Reference to Results
N1	Gaussian noise: $\Delta\sigma$ sweep (QDA)	Accuracy climbs monotonically with $\Delta\sigma$ and with dimension $n$ ; it reaches almost 1 already with modest gaps for $n$ high enough. In high $n$ white noise is easily and perfectly classifiable. We have tested $n\in\{5,10,50,500\}$ and $\Delta\sigma$ from 0 to 2 for an homogoenous covariance and for a Toeplitz one with $\rho=0.95$ .	Fig. 4 ; Methods in Section 3.1 ; Results in Section 4.1
N2	Gaussian noise: Bayes “oracle” boundary	White noise from the two distributions is almost perfectly classifiable in high enough $n$ ; accuracy goes to 1 quickly as $\Delta\sigma$ grows. We have tested $n\in\{30,100,500,1000,5000\}$ and we have varied $\Delta\sigma$ from 0 to 1.0.	Fig. 5 ; Methods in Section 3.1 ; Results in Section 4.1
N3	Gaussian noise: accuracy vs. dimension $n$	Even small $\Delta\sigma$ becomes highly separable as $n$ increases (dimensionality amplifies tiny distributional gaps). We have tested $n$ from 0 to 5000 (but visualisaed only until 100, to make the more steeper growing curves more visible) and tested $\Delta\sigma\in\{0.1,0.3,0.6,0.9,1.2,1.5,2.0\}$ .	Fig. 6 ; Methods in Section 3.1 ; Results in Section 4.1
N4	Skew–normal noise: 2D parameter sweeps	Tiny shifts in mean, variance or skewness give near-perfect accuracy for most models; random forest saturates the fastest.	Fig. 7 ; Methods in Section 3.2 ; Results in Section 4.2 ; Results in Section 4.3
S1	Synthetic spectra: truly identical classes	No classifier exceeds chance level accuracy (0.5) confirming that without distributional differences, classification is impossible.	Tab. 3 ; examples in Fig. 2
S2	Synthetic spectra: width difference	Validation accuracy rises as expected with $n$ ; linear and ensemble models approach $1.0$ for large $n$ similarity in the spectra.	Fig. 8 ; Results in Section 4.3

Table 2: Main findings and references to results.

Figure 4: Results from experiment N1. Classification accuracy of QDA (regularisation parameter equal to 0.4) as a function of the standard-deviation gap $\Delta\sigma$ between two white-noise classes with equal mean $\mu=1$ and baseline $\sigma_{1}=1$ . Each curve corresponds to a different number of points per array ( $n\in{[5,10,50,500]}$ ); at each $\Delta\sigma$ , $N$ arrays per class are generated and split 80/20 into train/test. The dashed line at $1.0$ marks perfect accuracy. The results are for the test dataset. Panel (A) has been obtained with a Toeptlitz covariance with $\rho=0.95$ , while panel (B) with a homogeneous covariance. It is an interesting observation that adding correlations between neighbouring values, slow down the growth to perfect accuracy, but it does not stop it altogether.

Figure 5: Results from experiment N2. Accuracy of the Bayes classifier for two Gaussian white-noise classes with common mean $\mu=10$ and variances $\sigma_{1}^{2}I_{n}$ vs. $\sigma_{2}^{2}I_{n}$ . The decision uses the sufficient statistic $S=\sum_{j=1}^{n}(x_{j}-\mu)^{2}$ with the LDA threshold $T$ .

Figure 6: Results from experiment N3. Test accuracy of QDA (reg_param $=0.4$ ) versus the number of points per array $n$ for two Gaussian white-noise classes with common mean $\mu$ and variances $\sigma_{1}^{2}I_{n}$ vs. $\sigma_{2}^{2}I_{n}$ . Each curve corresponds to a different standard-deviation gap $\Delta\sigma=\sigma_{2}-\sigma_{1}$ . Datasets contain $N$ arrays per class and are split $80/20$ into train/test. The dashed horizontal line at $1.0$ indicates perfect accuracy. Accuracy increases with both the number of points per array $n$ and the variance gap $\Delta\sigma$ ; even small $\Delta\sigma$ yields near-perfect accuracy as $n$ grows—showcasing how high dimensionality amplifies differences in distributions.

Figure 7: Results from experiment N4. Cross-validated classification accuracy for four models (from top to bottom: Random Forest, kNN, Decision Tree, Logistic Regression) on synthetic data drawn from a skew–normal distribution in dimension $n=50$ . Each column sweeps two parameters while holding the third at its baseline: (A) $\Delta\mu/\mu_{1}$ vs. $\Delta\sigma/\sigma_{1}$ , (B) $\Delta\mu/\mu_{1}$ vs. $\Delta\gamma/\gamma_{1}$ , (C) $\Delta\sigma/\sigma_{1}$ vs. $\Delta\gamma/\gamma_{1}$ , where $\gamma$ is the skew (shape) parameter of the skew–normal. Class 0 is fixed at $(\mu_{1},\sigma_{1},\gamma_{1})=(10,1,0.5)$ and Class 1 at $(\mu_{2},\sigma_{2},\gamma_{2})=(\mu_{1}+\Delta\mu,\ \sigma_{1}+\Delta\sigma,\ \gamma_{1}+\Delta\gamma)$ . For every grid point we generate $N=100$ samples per class and report the mean accuracy over $5$ -fold cross validation. Colour encodes accuracy (scale at right, 0.5–1.0); black contour lines mark iso-accuracy levels.

Model	Accuracy (mean $\pm$ SD)
Logistic Regression	$0.54\pm 0.03$
K-Neighbors Classifier	$0.50\pm 0.04$
Decision Tree Classifier	$0.53\pm 0.04$
Random Forest Classifier	$0.51\pm 0.01$

Table 3: Accuracy of multiple models for indistinguishable spectra generated as described in the text. Small deviations from

0.5

are attributable to finite-sample fluctuations.

Figure 8: Results from experiment S2. Model validation accuracy (mean $\pm$ 1 SD over 5-fold CV) versus spectrum length $n$ (log scale) for four classifiers: logistic regression, $k$ -NN, decision tree (max depth 5), and random forest (100 trees). Data consist of synthetic one–peak Lorentzian spectra: the two classes differ only in width ( $\xi_{1}=7$ vs. $\xi_{2}=9$ ) while peak centres are jittered $c\sim\mathcal{N}(50,10^{2})$ ; $500$ spectra per class are generated for each $n$ . Markers show the mean accuracy with error bars; the legend identifies the models.

Figure 9: Results from experiment S3. Validation accuracy (mean $\pm$ 1 SD over 5-fold CV) versus spectrum length $n$ (log scale) for four classifiers on synthetic one–peak spectra with identical signal distributions but class-specific additive noise. Each class contains $500$ spectra with Lorentzian FWHM $\xi=7$ and centres jittered $c\sim\mathcal{N}(50,10^{2})$ ; noise is i.i.d. Gaussian with mean $0$ (class 0) or $0.01$ (class 1) and SD $0.01$ . Accuracy rises with $n$ for all methods: random forests reach $\approx\!1.0$ with tens of intensity points, $k$ -NN stabilizes around $0.93$ – $0.95$ , logistic regression climbs steadily toward $1.0$ , while the single decision tree plateaus near $0.8$ .

Figure 10: Results for experiment Ra3/Rb3. LOO-CV classification accuracy as a function of the number of randomly selected pixels ( $k$ ) from the spectral noise region (pixels 0–50). Panel (A) shows the results for EVOO vs. LOO, and Panel (B) for EVOO vs. VOO. Each data point represents the mean accuracy over 20 independent random subsets, with error bars indicating the standard deviation. The rapid climb to accuracies above 80%-90% using only a handful of non-contiguous, chemically empty pixels serves as evidence: the model success is driven by high-dimensional statistical separability of instrumental artefacts rather than spectroscopic chemical features.

Figure 11: Empirical covariance matrices of the fluorescence spectra for the two olive oil classes (EXTRA and LAMPANTE). Bright red areas correspond to regions of strong inter-wavelength covariances, notably around the main fluorescence peak and stray-light regions. Such covariance mismatches are sufficient, in high-dimensional space, to enable nearly perfect classification even when chemically meaningful information is absent.

Figure 13: Results for experiment Ra5/Rb5. Regional Feature Attribution Map. Mean absolute SHAP values are presented in arbitrary units (a.u.) to facilitate the comparison of relative feature importance across varying window sizes ( $W$ ). Left Column (Panels A, C, E, G): Experiment EVOO vs. LOO. Right Column (Panels B, D, F, H): Experiment EVOO vs. VOO. The spectral region between 380–420 nm was explicitly removed to eliminate the Rayleigh scattering peak as a trivial discriminant. The persistence of high attribution in the noise-only region (pixels 0–400) across both experimental tasks indicates a reliance on high-dimensional statistical shortcuts rather than physical chemical signatures.

Classification

Classification maps observations to discrete labels, including familiar, unknown, hierarchical, or multiple categories.

See all 408 papers

May 4, 2026

Spectral Model eXplainer: a chemically-grounded explainability framework for spectral-based machine learning models

Spectral-based machine learning models have been increasingly deployed in chemometrics and spectroscopy, where predictive accuracy is as important as explainability. Current employed eXplainable Artificial Intelligence (XAI) methods are largely adapted from tabular or generic multivariate domains, assigning relevance to isolated spectral variables rather than to the chemically meaningful spectral zones. Widely adopted tools such as SHapley Additive exPlanations (SHAP), Permutation Feature Importance (PFI), and Variable Importance in Projection scores (VIP) were not designed for the physical continuity and high collinearity of spectral data, and their variable-level outputs require post-hoc aggregation to recover zone-level information. This study introduces the Spectral Model eXplainer (SMX), a post-hoc, global, model-agnostic XAI framework that explains spectral classifiers through expert-informed spectral zones. SMX summarizes each zone via PCA, defines quantile-based logical predicates, estimates predicate relevance with perturbation in stochastic subsamples, and aggregates bag-wise rankings in a directed weighted graph summarized by Local Reaching Centrality. A key component is threshold spectrum reconstruction, which back-projects predicate boundaries to the original spectral domain in natural measurement units, enabling direct visual comparison with measured spectra. The method was evaluated on eight real spectral datasets (six based on X-ray Fluorescence--XRF and two based on Gamma-ray Spectrometry) and one synthetic benchmark with known gr

Jose Vinicius Ribeiro, Rafael Figueira Goncalves, Fabio Luiz Melquiades +1

Applied Nuclear Physics Laboratory · State University of Londrina · Dept. of Engineering and Architecture +2

Feature Attribution Explainable Artificial Intelligence Interpretable ML

Ask PDF

Mar 19, 2026

SHAPCA: Consistent and Interpretable Explanations for Machine Learning Models on Spectroscopy Data

In recent years, machine learning models have been increasingly applied to spectroscopic datasets for chemical and biomedical analysis. For their successful adoption, particularly in clinical and safety-critical settings, professionals and researchers must be able to understand and trust the reasoning behind model predictions. However, the inherently high dimensionality and strong collinearity of spectroscopy data pose a fundamental challenge to model explainability. These properties not only complicate model training but also undermine the stability and consistency of explanations, leading to fluctuations in feature importance across repeated training runs. Feature extraction techniques have been used to reduce the input dimensionality; these new features hinder the connection between the prediction and the original signal. This study proposes SHAPCA, an explainable machine learning pipeline that combines Principal Component Analysis (for dimensionality reduction) and Shapely Additive exPlanations (for post hoc explanation) to provide explanations in the original input space, which a practitioner can interpret and link back to the biological components. The proposed framework enables analysis from both global and local perspectives, revealing the spectral bands that drive overall model behaviour as well as the instance-specific features that influence individual predictions. Numerical analysis demonstrated the interpretability of the results and greater consistency across different runs.

Mingxing Zhang, Nicola Rossberg, Simone Innocente +5

School of Computer Science and Information Technology, University College Cork, Ireland · Centre for Research Training in Artificial Intelligence, University College Cork, Ireland · Insight Centre for Data Analytics, University College Cork, Ireland

Explainable Artificial Intelligence Explanation Stability

Ask PDF

May 11, 2026

SpecX: A Large-Scale Benchmark for Multi-Modal Spectroscopy and Cross-Paradigm Evaluation

Existing spectral benchmarks are limited in scale, modality alignment, and evaluation scope, and typically focus on either specialized models or multimodal language models (MLLMs). We introduce SpecX, a large-scale benchmark for multi-modal spectroscopy with cross-paradigm evaluation. SpecX contains 1.7M molecules with diverse spectral modalities, including NMR (1H, 13C, HSQC), IR, MS,UV,Raman and FL, and is organized into three tiers: a large-scale dataset for pretraining, an aligned multi-spectral subset for benchmarking, and a high-quality experimental subset for evaluation. SpecX supports a range of tasks such as molecular elucidation, spectrum simulation, and spectral understanding, and enables unified evaluation across both specialized spectral models and MLLMs. Experiments show that specialized models excel at signal-level modeling, while MLLMs exhibit strengths in high-level reasoning but lack precise spectral grounding. SpecX establishes a unified benchmark for spectral intelligence and highlights the need for spectrum-native foundation models.

Chengrui Xiang, Tengfei Ma, Yujie Chen +3

College of Computer Science and Technology, Hunan University, Changsha, China

Multimodal Model Evaluation

Ask PDF

Abstract

Classification

Explore similar work

Spectral Model eXplainer: a chemically-grounded explainability framework for spectral-based machine learning models

SHAPCA: Consistent and Interpretable Explanations for Machine Learning Models on Spectroscopy Data

SpecX: A Large-Scale Benchmark for Multi-Modal Spectroscopy and Cross-Paradigm Evaluation