Evaluating classification and explainability in multimodal petrographic image analysis
Organizations: Department of Earth and Environmental Sciences, Cardiff University, Cardiff, United Kingdom. · Department of Computer Science and Informatics, Cardiff University, Cardiff, United Kingdom. · Department of Computer Engineering, Turkish Air Force Academy, National Defence University, Istanbul, Türkiye.
Abstract
Automated thin-section classifiers increasingly combine plane-polarised and cross-polarised light (PPL and XPL), two complementary views central to petrographic identification. Classification accuracy alone does not reveal whether model errors reflect mineralogical relationships, how predictions depend on each optical view, or whether attribution maps identify regions that influence model decisions. Here, we examine these complementary aspects of the released LITHOS dual-stream vision transformer using outputs from an earlier evaluation of 105,802 expert-annotated grains across 25 categories. Classification errors frequently followed the label taxonomy: 72.6% of misclassified plagioclase, microcline and sanidine grains were assigned the generic Feldspar label, while 87.1% of quartz-family grains received a label within that family, compared with 66.6% assigned their exact label. Replacing the XPL input with its per-channel spatial means produced a greater reduction in true-class probability than equivalent PPL replacement in 20 of 25 categories, but not in the three quartz categories. Five explainable artificial intelligence (XAI) attribution procedures generated visually distinct maps for identical inputs. In a superpixel-deletion analysis involving 30 grains from 29 thin sections, method-minus-random contrasts ranged from to , with all 41 simultaneous 95% confidence intervals including zero. Together, these findings show that error structure, modality sensitivity and attribution behaviour capture distinct aspects of model performance that cannot be established from classification accuracy or visual inspection of attribution maps alone.
Figures & tables
| Evaluation | Grains | Accuracy | Macro recall | Macro precision | Macro-F1 |
| Published baseline Chacón et al. (2025) | |||||
| Two-fold mean s.d. | 105,802 | 0.623 0.019 | 0.415 0.006 | 0.519 0.001 | 0.432 0.002 |
| This study | |||||
| Fold 1 | 52,792 | 0.6361 | 0.4199 | 0.5199 | 0.4339 |
| Fold 2 | 53,010 | 0.6099 | 0.4110 | 0.5187 | 0.4305 |
| Two-fold mean s.d. | 105,802 | 0.6230 0.0185 | 0.4155 0.0063 | 0.5193 0.0009 | 0.4322 0.0024 |
| True category | Predicted category | Count / support | % of true | % reverse | Relationship of labels |
| Sanidine | Feldspar | 121 / 206 | 58.7 | 0.2 | Specific potassium feldspar; generic feldspar label |
| Microcline | Feldspar | 106 / 219 | 48.4 | 0.1 | Specific potassium feldspar; generic feldspar label |
| Plagioclase | Feldspar | 1,437 / 3,488 | 41.2 | 5.8 | Feldspar series; generic feldspar label |
| Hornblende | Amphibole | 91 / 311 | 29.3 | 2.6 | Amphibole-supergroup mineral; group label |
| Muscovite | Mica | 23 / 303 | 7.6 | 0.5 | Mica-group mineral; group label |
| Polycrystalline | Monocrystalline | 4,401 / 17,702 | 24.9 | 10.2 | Quartz textural varieties |
| Procedure and branch | Mean AOC | Mean minus random | Pointwise 95% CI | Simultaneous 95% CI | Grains above random |
| Random order (reference) | |||||
| PPL | 0.145 | — | — | — | — |
| XPL | 0.183 | — | — | — | — |
| Grad-CAM | |||||
| PPL | 0.150 | 0.005 | [ 0.024, 0.040] | [ 0.047, 0.057] | 13/30 |
| XPL | 0.232 | 0.048 | [0.006, 0.092] | [ 0.021, 0.117] | 20/30 |
Multimodal Classification
Multimodal classification combines images, text, audio, signals, metadata, or other inputs to assign labels more accurately and reliably.
See all 212 papers