cs.LGApr 6, 2026
SaveThe Infinite-Dimensional Nature of Spectroscopy and Why Models Succeed, Fail, and Mislead
Organizations: Department of Computer Science Lucerne University of Applied Sciences and Arts Risch-Rotkreuz, 6343 Switzerland · Institute of Applied Mathematics and Physics ZHAW Zurich University of Applied Sciences Winterthur, 8400 Switzerland
Abstract
Machine learning (ML) models have achieved strikingly high accuracies in spectroscopic classification tasks, often without a clear proof that those models used chemically meaningful features. Existing studies have linked these results to data preprocessing choices, noise sensitivity, and model complexity, but no unifying explanation is available so far. In this work, we show that these phenomena arise naturally from the intrinsic high dimensionality of spectral data. Using a theoretical analysis grounded in the Feldman-Hajek theorem and the concentration of measure, we show that even infinitesimal distributional differences, caused by noise, normalisation, or instrumental artefacts, may become perfectly separable in high-dimensional spaces. Through a series of specific experiments on synthetic and real fluorescence spectra, we illustrate how models can achieve near-perfect accuracy even when chemical distinctions are absent, and why feature-importance maps may highlight spectrally irrelevant regions. We provide a rigorous theoretical framework, confirm the effect experimentally, and conclude with practical recommendations for building and interpreting ML models in spectroscopy.
Figures & tables
Figure 1: Illustration of the concentration of measure for multivariate Gaussian distributions. Shown are the empirical distributions of for samples drawn from (light blue) and (yellow), for increasing dimensionalities , , , and (panels A–D). In low dimensions the two distributions overlap substantially, but as increases the probability mass concentrates sharply around the typical radius , and even small variance differences cause almost complete separation. This illustrates how, in high-dimensional spaces, measures supported by Gaussian (and many non-Gaussian) distributions become effectively disjoint, providing an intuitive geometric basis for the Feldman–Hájek theorem and for the sensitivity of high-dimensional classifiers to minute statistical differences.
| ID | Experiment | Experiment Details |
|---|---|---|
| N1 | Gaussian noise: sweep (QDA) | Classify white noise from two multivariate Gaussians with equal means ( ) and different isotropic and Toeplitz (with parameter ) covariances; we varied the variance gap from 0 to 2 while keeping fixed. We have tested . We have measured classification accuracy. |
| N2 | Gaussian noise: Bayes “oracle” boundary | Classify white noise from two multivariate Gaussians with equal means ( ) and different isotropic covariances; we varied the variance gap from 0 to 2 while keeping fixed. We have tested . We classify white noise by using the ideal threshold on . |
| N3 | Gaussian noise: accuracy vs. dimension | Classify white noise from two multivariate Gaussians with equal means ( ) and different isotropic covariances; we varied the variance gap in while keeping fixed. We tested . We calculate the accuracy of QDA for various values of . |
| N4 | Skew–normal noise: 2D parameter sweeps | Classify two classes of white noise from multivariate skew–normal Azzalini and Valle (1996) . For this experiment, we used the parameters , , , . We then varied from 0 to 2, from 0 to 0.15, and from 0 to 8. |
| S1 | Synthetic spectra: truly identical classes | Classify two classes of spectra each composed of one Lorentzian (both centres chosen according to a normal distribution ) and an FWHM . We generated spectra for each class. We choose . These two sets of spectra are not distinguishable, as there are no differences in the data distributions. |
| S2 | Synthetic spectra: FWHM difference | Classify two classes of spectra each composed of one Lorentzian (both centres chosen accordingly to a normal distribution ) and two different FWHM and . We generated spectra for each class. We choose We study accuracy of classification for various value of . |
Table 1: Overview of experiments and their core ideas.
Figure 2: Ten representative synthetic one-peak spectra per class used in the synthetic-spectra experiments. Each curve is a Lorentzian profile sampled on an -point axis, with peak centre jittered as . Class 0 (blue) and Class 1 (orange) differ only through the FWHM vs. , illustrating that the two classes are visually difficult to distinguish despite being statistically separable in high dimension.
Figure 3: Fluorescence spectra of Spanish olive oil samples classified as Extra Virgin (EVOO), Virgin (VOO), and Lampante (LOO). The region 380–420 nm indicates the Rayleigh scattering peak from the excitation LED. The black line indicates the average spectrum for each class.
| ID | Experiment | Key finding | Reference to Results |
|---|---|---|---|
| N1 | Gaussian noise: sweep (QDA) | Accuracy climbs monotonically with and with dimension ; it reaches almost 1 already with modest gaps for high enough. In high white noise is easily and perfectly classifiable. We have tested and from 0 to 2 for an homogoenous covariance and for a Toeplitz one with . | Fig. 4 ; Methods in Section 3.1 ; Results in Section 4.1 |
| N2 | Gaussian noise: Bayes “oracle” boundary | White noise from the two distributions is almost perfectly classifiable in high enough ; accuracy goes to 1 quickly as grows. We have tested and we have varied from 0 to 1.0. | Fig. 5 ; Methods in Section 3.1 ; Results in Section 4.1 |
| N3 | Gaussian noise: accuracy vs. dimension | Even small becomes highly separable as increases (dimensionality amplifies tiny distributional gaps). We have tested from 0 to 5000 (but visualisaed only until 100, to make the more steeper growing curves more visible) and tested . | Fig. 6 ; Methods in Section 3.1 ; Results in Section 4.1 |
| N4 | Skew–normal noise: 2D parameter sweeps | Tiny shifts in mean, variance or skewness give near-perfect accuracy for most models; random forest saturates the fastest. | Fig. 7 ; Methods in Section 3.2 ; Results in Section 4.2 ; Results in Section 4.3 |
| S1 | Synthetic spectra: truly identical classes | No classifier exceeds chance level accuracy (0.5) confirming that without distributional differences, classification is impossible. | Tab. 3 ; examples in Fig. 2 |
| S2 | Synthetic spectra: width difference | Validation accuracy rises as expected with ; linear and ensemble models approach for large similarity in the spectra. | Fig. 8 ; Results in Section 4.3 |
Table 2: Main findings and references to results.
Figure 4: Results from experiment N1. Classification accuracy of QDA (regularisation parameter equal to 0.4) as a function of the standard-deviation gap between two white-noise classes with equal mean and baseline . Each curve corresponds to a different number of points per array ( ); at each , arrays per class are generated and split 80/20 into train/test. The dashed line at marks perfect accuracy. The results are for the test dataset. Panel (A) has been obtained with a Toeptlitz covariance with , while panel (B) with a homogeneous covariance. It is an interesting observation that adding correlations between neighbouring values, slow down the growth to perfect accuracy, but it does not stop it altogether.
Figure 5: Results from experiment N2. Accuracy of the Bayes classifier for two Gaussian white-noise classes with common mean and variances vs. . The decision uses the sufficient statistic with the LDA threshold .
Figure 6: Results from experiment N3. Test accuracy of QDA (reg_param ) versus the number of points per array for two Gaussian white-noise classes with common mean and variances vs. . Each curve corresponds to a different standard-deviation gap . Datasets contain arrays per class and are split into train/test. The dashed horizontal line at indicates perfect accuracy. Accuracy increases with both the number of points per array and the variance gap ; even small yields near-perfect accuracy as grows—showcasing how high dimensionality amplifies differences in distributions.
Figure 7: Results from experiment N4. Cross-validated classification accuracy for four models (from top to bottom: Random Forest, kNN, Decision Tree, Logistic Regression) on synthetic data drawn from a skew–normal distribution in dimension . Each column sweeps two parameters while holding the third at its baseline: (A) vs. , (B) vs. , (C) vs. , where is the skew (shape) parameter of the skew–normal. Class 0 is fixed at and Class 1 at . For every grid point we generate samples per class and report the mean accuracy over -fold cross validation. Colour encodes accuracy (scale at right, 0.5–1.0); black contour lines mark iso-accuracy levels.
| Model | Accuracy (mean SD) |
|---|---|
| Logistic Regression | |
| K-Neighbors Classifier | |
| Decision Tree Classifier | |
| Random Forest Classifier |
Table 3: Accuracy of multiple models for indistinguishable spectra generated as described in the text. Small deviations from are attributable to finite-sample fluctuations.
Figure 8: Results from experiment S2. Model validation accuracy (mean 1 SD over 5-fold CV) versus spectrum length (log scale) for four classifiers: logistic regression, -NN, decision tree (max depth 5), and random forest (100 trees). Data consist of synthetic one–peak Lorentzian spectra: the two classes differ only in width ( vs. ) while peak centres are jittered ; spectra per class are generated for each . Markers show the mean accuracy with error bars; the legend identifies the models.
Figure 9: Results from experiment S3. Validation accuracy (mean 1 SD over 5-fold CV) versus spectrum length (log scale) for four classifiers on synthetic one–peak spectra with identical signal distributions but class-specific additive noise. Each class contains spectra with Lorentzian FWHM and centres jittered ; noise is i.i.d. Gaussian with mean (class 0) or (class 1) and SD . Accuracy rises with for all methods: random forests reach with tens of intensity points, -NN stabilizes around – , logistic regression climbs steadily toward , while the single decision tree plateaus near .
Figure 10: Results for experiment Ra3/Rb3. LOO-CV classification accuracy as a function of the number of randomly selected pixels ( ) from the spectral noise region (pixels 0–50). Panel (A) shows the results for EVOO vs. LOO, and Panel (B) for EVOO vs. VOO. Each data point represents the mean accuracy over 20 independent random subsets, with error bars indicating the standard deviation. The rapid climb to accuracies above 80%-90% using only a handful of non-contiguous, chemically empty pixels serves as evidence: the model success is driven by high-dimensional statistical separability of instrumental artefacts rather than spectroscopic chemical features.
Figure 11: Empirical covariance matrices of the fluorescence spectra for the two olive oil classes (EXTRA and LAMPANTE). Bright red areas correspond to regions of strong inter-wavelength covariances, notably around the main fluorescence peak and stray-light regions. Such covariance mismatches are sufficient, in high-dimensional space, to enable nearly perfect classification even when chemically meaningful information is absent.
Figure 12: Results for experiment Ra4/Rb4. LOO-CV classification accuracy is mapped across the fluorescence spectrum (gray line) using non-overlapping windows of increasing size . Left Column (Panels A, C, E, G): Experiment EVOO vs. LOO. Right Column (Panels B, D, F, H): Experiment EVOO vs. VOO. The spectral region between 380–420 nm was explicitly removed to eliminate the Rayleigh scattering peak as a trivial discriminant. Note that even at small window sizes (20–50 px), the models achieve 70–80% accuracy in the chemically empty region (pixels 0–400). As dimensionality increases to (Panels G and H), a stable accuracy plateau of 80% emerges across the entire detector, confirming that high-dimensional noise distributions provide sufficient statistical information for class separation, independent of physical spectroscopic features.
Figure 13: Results for experiment Ra5/Rb5. Regional Feature Attribution Map. Mean absolute SHAP values are presented in arbitrary units (a.u.) to facilitate the comparison of relative feature importance across varying window sizes ( ). Left Column (Panels A, C, E, G): Experiment EVOO vs. LOO. Right Column (Panels B, D, F, H): Experiment EVOO vs. VOO. The spectral region between 380–420 nm was explicitly removed to eliminate the Rayleigh scattering peak as a trivial discriminant. The persistence of high attribution in the noise-only region (pixels 0–400) across both experimental tasks indicates a reliance on high-dimensional statistical shortcuts rather than physical chemical signatures.
More in this topic
Classification
Classification maps observations to discrete labels, including familiar, unknown, hierarchical, or multiple categories.
See all 408 papers