Evaluating classification and explainability in multimodal petrographic image analysis
Authors: Thomas James Wood, Oktay Karakuş, Can Eyüpoğlu
Organizations: Department of Earth and Environmental Sciences, Cardiff University, Cardiff, United Kingdom. · Department of Computer Science and Informatics, Cardiff University, Cardiff, United Kingdom. · Department of Computer Engineering, Turkish Air Force Academy, National Defence University, Istanbul, Türkiye.
Automated thin-section classifiers increasingly combine plane-polarised and cross-polarised light (PPL and XPL), two complementary views central to petrographic identification. Classification accuracy alone does not reveal whether model errors reflect mineralogical relationships, how predictions depend on each optical view, or whether attribution maps identify regions that influence model decisions. Here, we examine these complementary aspects of the released LITHOS dual-stream vision transformer using outputs from an earlier evaluation of 105,802 expert-annotated grains across 25 categories. Classification errors frequently followed the label taxonomy: 72.6% of misclassified plagioclase, microcline and sanidine grains were assigned the generic Feldspar label, while 87.1% of quartz-family grains received a label within that family, compared with 66.6% assigned their exact label. Replacing the XPL input with its per-channel spatial means produced a greater reduction in true-class probability than equivalent PPL replacement in 20 of 25 categories, but not in the three quartz categories. Five explainable artificial intelligence (XAI) attribution procedures generated visually distinct maps for identical inputs. In a superpixel-deletion analysis involving 30 grains from 29 thin sections, method-minus-random contrasts ranged from −0.043 to 0.057, with all 41 simultaneous 95% confidence intervals including zero. Together, these findings show that error structure, modality sensitivity and attribution behaviour capture distinct aspects of model performance that cannot be established from classification accuracy or visual inspection of attribution maps alone.
Figures & tables
Evaluation
Grains
Accuracy
Macro recall
Macro precision
Macro-F1
Published baseline Chacón et al. (2025)
Two-fold mean ± s.d.
105,802
0.623 ± 0.019
0.415 ± 0.006
0.519 ± 0.001
0.432 ± 0.002
This study
Fold 1
52,792
0.6361
0.4199
0.5199
0.4339
Fold 2
53,010
0.6099
0.4110
0.5187
0.4305
Two-fold mean ± s.d.
105,802
0.6230 ± 0.0185
0.4155 ± 0.0063
0.5193 ± 0.0009
0.4322 ± 0.0024
Table 1: Classification performance of the LITHOS dual-polarisation baseline. 0 0 footnotetext: Macro metrics weight the 25 per-category values equally; s.d. is the sample standard deviation of the two fold values. Each fold was evaluated with the model trained on the other fold.
Figure 1: Classification performance and error structure of the LITHOS dual-polarisation baseline. ( a ) Row-normalised confusion matrix pooled across both section-disjoint folds (105,802 grains). Each row represents the distribution of predicted categories for grains belonging to a given true category. Categories are arranged by petrographic grouping, with red outlines identifying the quartz family, feldspars, micas, amphiboles, carbonates and biogenic categories discussed in the text. The classifier treats all 25 labels as mutually exclusive. ( b ) Per-category F1 scores for fold 1 (52,792 grains), fold 2 (53,010 grains) and the pooled predictions, together with the number of grains in each category ( n ). Confidence intervals are not presented because grain-level thin-section identifiers were unavailable for the complete prediction dataset.
True category
Predicted category
Count / support
% of true
% reverse
Relationship of labels
Sanidine
Feldspar
121 / 206
58.7
0.2
Specific potassium feldspar; generic feldspar label
Microcline
Feldspar
106 / 219
48.4
0.1
Specific potassium feldspar; generic feldspar label
Plagioclase
Feldspar
1,437 / 3,488
41.2
5.8
Feldspar series; generic feldspar label
Hornblende
Amphibole
91 / 311
29.3
2.6
Amphibole-supergroup mineral; group label
Muscovite
Mica
23 / 303
7.6
0.5
Mica-group mineral; group label
Polycrystalline
Monocrystalline
4,401 / 17,702
24.9
10.2
Quartz textural varieties
Table 2: Selected pooled confusions and the relationship between the labels involved. 0 0 footnotetext: Counts are pooled over both folds; support is the number of grains in the true category. The reverse column gives the percentage of grains of the predicted category that were assigned to the true category of the same row. Label relationships follow the LITHOS documentation Chacón et al. (2025) and standard mineral nomenclature Deer et al. (2013) ; Hawthorne and others (2012) ; the classifier itself treats all 25 labels as mutually exclusive.
Figure 2: Category-dependent sensitivity to replacing the PPL or XPL input. ( a ) Mean reduction in true-class probability following replacement of the PPL input (circles) or XPL input (squares) with its per-channel spatial means, while leaving the other input unchanged. Results are based on 706 correctly classified fold-2 grains, with category sample sizes shown in parentheses. ( b ) Difference between the probability reductions following XPL and PPL replacement. Positive values, shown in vermilion, indicate greater sensitivity to XPL replacement and were observed in 20 of 25 categories. Categories follow the ordering in Fig. 1 . Confidence intervals are not shown because only category-level means were retained. Mean replacement preserves average channel values while removing spatial structure and does not isolate the contribution of any individual optical property.
Figure 3: Attribution maps from five procedures for two correctly classified grains. Rows present the PPL and XPL views of ( a ) a monocrystalline grain and ( b ) a polycrystalline grain from fold 2. Both grains were correctly classified, with true-class probabilities of 0.967 and 0.806, respectively. The first column shows the 224×224 pixel model input; external tick marks indicate the patch centre containing the annotated grain. Subsequent columns show attribution overlays from three class-targeted procedures (Grad-CAM, Chefer branch adaptation and Chefer cross adaptation) and two class-agnostic baselines (decoder cross-attention and attention rollout), following the ordering in Table 3 . Warmer colours indicate higher attribution values. Each map was independently min–max normalised, preventing direct comparison of intensities across panels. Grad-CAM overlays also use a different blending procedure. For cross-attention and Chefer cross attribution, maps over one branch are obtained from the class token of the opposite stream. The examples were selected during the original analysis and were not included in the 30-grain deletion sample; consequently, no deletion scores are available for these grains. Panels were reproduced from previously saved figures without resampling. The maps illustrate differences between procedures and branches but do not establish attribution faithfulness or mineralogical reasoning. Underlying images were obtained from the LITHOS dataset Ruiz Puentes (2025) (CC BY-NC-SA 4.0).
Figure 4: Paired deletion contrasts between attribution-ordered and random-ordered superpixel deletion. Results are shown for ( a ) the PPL branch and ( b ) the XPL branch. Each point represents the difference between a grain’s AOC under attribution-guided deletion and its mean AOC under random segment ordering on the same branch (30 grains from 29 thin sections). Black circles indicate section-equal means, thick coloured bars indicate pointwise 95% section-bootstrap confidence intervals, and thin grey bars indicate approximate simultaneous 95% confidence intervals for the family of 41 contrasts, based on 10,000 bootstrap resamples. The dashed line represents equality with random ordering. All simultaneous confidence intervals include zero. Absolute AOC values are reported in Table 3 .
Procedure and branch
Mean AOC
Mean minus random
Pointwise 95% CI
Simultaneous 95% CI
Grains above random
Random order (reference)
PPL
0.145
—
—
—
—
XPL
0.183
—
—
—
—
Grad-CAM
PPL
0.150
0.005
[ − 0.024, 0.040]
[ − 0.047, 0.057]
13/30
XPL
0.232
0.048
[0.006, 0.092]
[ − 0.021, 0.117]
20/30
Table 3: Deletion scores and paired differences from random segment order, with equal weighting of thin sections. 0 0 footnotetext: Values are means over 29 thin sections, each first averaged over its grains (one section contributes two grains). The random reference is each grain’s mean AOC over three shuffled segment orders. Confidence intervals (CIs) are based on 10,000 paired bootstrap resamples of thin sections; simultaneous intervals cover the full family of 41 contrasts (Supplementary Table S2). Differences are computed within grain and branch before averaging and may differ in the third decimal place from the difference of the rounded means. Chefer procedures are architecture-specific adaptations; cross-attention and rollout are class-agnostic structural baselines.
Figure 5: Grain-level variability and branch contrasts in the deletion analysis. ( a ) Method-minus-random AOC contrasts for individual grains, procedures and optical branches, grouped by category (five grains per category). Colours are scaled symmetrically around zero, with no values truncated. GC, Grad-CAM; CB, Chefer branch adaptation; CC, Chefer cross adaptation; CA, cross-attention; RO, rollout. ( b ) Within-grain differences between XPL and PPL method-minus-random contrasts for each procedure. Points represent individual grains, black circles indicate section-equal means, thick bars indicate pointwise 95% confidence intervals, and thin grey bars indicate approximate simultaneous 95% confidence intervals for the family of 41 contrasts.
Few-shot classifiers of rock thin sections are meta-trained on many rock types that a petrographer has identified slide by slide, and this identification is the costly step that new collections lack. We study few-shot recognition of unseen rock types when no training slide is identified, and the only information available is what the microscope records anyway: which slide an image comes from, and whether it was taken in plane-polarized (PPL) or cross-polarized (XPL) light. We first show why generic unsupervised pseudo-tasks fit this setting poorly: the positives they find without labels are either wrong or near-duplicates of the anchor, so they teach the model little. Polarized-light microscopy provides positives that avoid both problems. The PPL and XPL views of one field show the same grains, so they are correct by construction, yet they look very different. We propose cross-polarization pairing (XPP), which treats each slide as a pseudo-class and pairs its PPL and XPL views of the same field location, so that matching must rely on grain shape and texture rather than on colour. On the Nanjing University collection of 108 rock types, with support and query images taken from different slides, XPP is the best label-free method on average over three random splits, and matches a supervised model trained on 78 identified rock types. Controls show that colour augmentation and other acquisition positives do not reproduce the gain. We also show that the image-level protocol of earlier work is saturated, and propose a cross-slide protocol in its place.
Bingchen Yan
School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin, 541004, China
Grain-edge segmentation (GES) and lithology semantic segmentation (LSS) are two pivotal tasks for quantifying rock fabric and composition. However, these two tasks are often treated separately, and the segmentation quality is implausible albeit expensive, time-consuming, and expert-annotated datasets have been used. Recently, foundation models, especially the Segment Anything Model (SAM), have demonstrated impressive robustness for boundary alignment. However, directly adapting SAM to joint GES and LSS is nontrivial due to 1) severe domain gap induced by extinction-dependent color variations and ultra-fine grain boundaries, and 2) lacking novel modules for joint learning on multi-angle petrographic image stacks. In this paper, we propose Petro-SAM, a novel two-stage, multi-task framework that can achieve high-quality joint GES and LSS on petrographic images. Specifically, based on SAM, we introduce a Merge Block to integrate seven polarized views, effectively solving the extinction issue. Moreover, we introduce multi-scale feature fusion and color-entropy priors to refine the detection.
Yili Ren, Shiqi Wen, Li Hou +8
Research Institute of Petroleum Exploration and Development (RIPED) · The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology +1
Automated pollen analysis supports veterinary cytology, but brightfield microscopy is costlier and more complex than lens-less digital in-line holographic microscopy. We evaluate whether reconstructed holograms can narrow this gap and whether model explanations remain reliable under modality change. Six pollen species were imaged by brightfield and holographic microscopy. Raw, single back-propagation and iterative phase retrieval holograms were evaluated with YOLOv26s detection and MobileNetV4 classification after anchor-based annotation transfer. Six attribution methods were assessed for spatial grounding and faithfulness with the Attribution Health Inspection and Repair (AHIR) protocol, which tests model brittleness under weak noise and corrects attribution-map granularity when needed. Brightfield achieved 0.6890 mAP50-95 (0.8865 mAP50) for detection and 0.9687 macro-F1 (0.9705 accuracy) for classification. Reconstructed holograms narrowed the gap with a task-dependent split: p-type was strongest for detection at 0.5324 mAP50-95 (0.8229 mAP50), while r-type was strongest for classification at 0.7695 macro-F1 (0.7866 accuracy), both far above raw-hologram baselines. Activation-based explanations localized strongly on grains, and region-based methods retained ~60 to ~80% of faithfulness under holography. The holographic detector was highly brittle to weak perturbations, saturating deletion-based evaluation while insertion remained informative. Pixel-level gradient explanations approached random floor, yet spatial smoothing restored p-type gradient faithfulness from 0.05 to 0.51. For holographic classification, perturbation-based explanations remained faithful while gradient-based methods fell below random floor. Reconstruction improves low-cost holographic pollen analysis, while AHIR distinguishes genuine attribution failure from artifacts caused by model brittleness and map granularity.
Swarn Warshaneyan, Joial Danyal, Blaž Cugmas +4
Institute of Electronics and Computer Science (EDI), Riga, Latvia · Faculty of Science and Technology, University of Latvia, Riga, Latvia