SHAPCA: Consistent and Interpretable Explanations for Machine Learning Models on Spectroscopy Data
Authors: Mingxing Zhang, Nicola Rossberg, Simone Innocente, Katarzyna Komolibus, Rekha Gautam, Barry O'Sullivan, Luca Longo, Andrea Visentin
Organizations: School of Computer Science and Information Technology, University College Cork, Ireland · Centre for Research Training in Artificial Intelligence, University College Cork, Ireland · Insight Centre for Data Analytics, University College Cork, Ireland
In recent years, machine learning models have been increasingly applied to spectroscopic datasets for chemical and biomedical analysis. For their successful adoption, particularly in clinical and safety-critical settings, professionals and researchers must be able to understand and trust the reasoning behind model predictions. However, the inherently high dimensionality and strong collinearity of spectroscopy data pose a fundamental challenge to model explainability. These properties not only complicate model training but also undermine the stability and consistency of explanations, leading to fluctuations in feature importance across repeated training runs. Feature extraction techniques have been used to reduce the input dimensionality; these new features hinder the connection between the prediction and the original signal. This study proposes SHAPCA, an explainable machine learning pipeline that combines Principal Component Analysis (for dimensionality reduction) and Shapely Additive exPlanations (for post hoc explanation) to provide explanations in the original input space, which a practitioner can interpret and link back to the biological components. The proposed framework enables analysis from both global and local perspectives, revealing the spectral bands that drive overall model behaviour as well as the instance-specific features that influence individual predictions. Numerical analysis demonstrated the interpretability of the results and greater consistency across different runs.
Figures & tables
Figure 1: Step-by-step workflow of the proposed interpretable spectroscopic pipeline.
Figure 2: A Raman spectroscopy sample. Opacity indicates the most decisive spectral regions for the prediction, while colour encodes intensity relative to the dataset average (red = higher, blue = lower).
Pipeline
Raman spectroscopy
DRS
Accuracy
F1 Score
Accuracy
F1 Score
Sparse PCA + SVC
0.681±0.023
0.663±0.023
0.972±0.002
0.976±0.002
SVC
0.757±0.025
0.735±0.028
0.993±0.001
0.994±0.001
Sparse PCA + RF
0.723±0.024
0.705±0.026
0.936±0.003
0.944±0.002
RF
0.740±0.026
0.725±0.028
0.945±0.003
0.952±0.003
Table 1: Mean test accuracy and F1 score for Raman spectroscopy and DRS data, reported as mean ± 95% confidence interval.
Figure 3: Global explanations for the binary Raman spectroscopy task (classes H and PML). The grey curves show the class-mean spectra.
Figure 4: Local explanation for a binary classification instance correctly predicted as PML. The grey curve represents the Raman spectrum of the instance.
Class
SHAPCA
SHAP
RF+Sparse PCA
SVC+Sparse PCA
RF
SVC
Cosine
Correlation
Cosine
Correlation
Cosine
Correlation
Cosine
Correlation
H
0.820
0.811
0.458
0.406
0.715
0.698
0.370
0.333
PML
0.842
0.835
0.567
0.517
0.759
0.747
0.437
0.403
Table 2: Comparison of global explanation consistency and correlation across classes for Raman spectroscopy data.
SHAPCA
SHAP
RF+Sparse PCA
SVC+Sparse PCA
RF
SVC
Cosine
Correlation
Cosine
Correlation
Cosine
Correlation
Cosine
Correlation
Local
0.622
0.615
0.242
0.239
0.573
0.568
-0.068
-0.060
Table 3: Comparison of local explanation consistency and correlation for Raman spectroscopy data.
Figure 5: Global explanation of DRS multiclass classification across all classes. The grey curve shows the mean spectrum of each class.
Figure 6: Local explanation of multi-classification for an instance correctly predicted as cortBone
Class
SHAPCA
SHAP
RF+Sparse PCA
SVC+Sparse PCA
RF
SVC
Cosine
Corr.
Cosine
Corr.
Cosine
Corr.
Cosine
Corr.
Muscle
0.992
0.984
0.784
0.787
0.694
0.679
0.340
0.280
CortBone
0.988
0.948
0.792
0.706
0.577
0.576
0.751
0.748
TraBone
0.990
0.976
0.842
0.735
0.729
0.694
0.620
0.607
Cartilage
0.981
0.963
0.942
0.950
0.648
0.647
0.387
0.370
Table 4: Comparison of global explanation consistency and correlation across classes for DRS data.
SHAPCA
SHAP
RF+Sparse PCA
SVC+Sparse PCA
RF
SVC
Cosine
Corr.
Cosine
Corr.
Cosine
Corr.
Cosine
Corr.
Local
0.982
0.977
0.459
0.631
0.691
0.685
0.371
0.367
Table 5: Comparison of local explanation consistency and correlation for DRS data.