Uncertainty-Aware Continual Learning for 2D XRD Interpretation Using Simulated and Experimental Data
Authors: Zihan Wang, Hyun Sang Park, Ali Rachidi, David Elbert, Todd C. Hufnagel, Wei Chen
Organizations: Department of Mechanical Engineering, Northwestern University, 2145 Sheridan Road, Evanston, Illinois 60076, USA. · Department of Materials Science and Engineering, Johns Hopkins University, 3400 N. Charles Street, Baltimore, Maryland 21218, USA. · Hopkins Extreme Materials Institute, Johns Hopkins University, 3400 N. Charles Street, Baltimore, Maryland 21218, USA.
Autonomous experimentation systems require characterization models that can interpret heterogeneous measurements and maintain predictive performance as data accumulate. Developing such models involves addressing domain mismatch between data sources, overconfident predictions on unfamiliar inputs, and catastrophic forgetting during sequential updates. In this work, we present a proof-of-concept framework for uncertainty-aware continual interpretation of two-dimensional X-ray diffraction (XRD) data. The framework integrates established components: maximum mean discrepancy-based domain adaptation to reduce simulation-to-experiment discrepancies, a convolutional neural network with a spectral-normalized neural Gaussian process (CNN--SNGP) for predictive uncertainty estimation, and replay with knowledge distillation for sequential model updating. We evaluate the framework using simulated and previously-collected experimental data for crystal structure classification of single-phase body-centered cubic, face-centered cubic, and hexagonal close-packed patterns. Data are organized into predefined sequential blocks to assess adaptation and retention of previously learned knowledge. Within this setting, the results show accurate classification across simulated and experimental data, improved alignment between the two domains, and reduced forgetting compared with naïve fine-tuning. Predictive uncertainty is also evaluated on unfamiliar and noisy inputs. The contribution is the integration and evaluation of these capabilities for sequential XRD interpretation. The framework provides a basis for future integration into autonomous experimentation systems, with experimental planning, instrument control, and closed-loop deployment remaining areas for future development.
Figures & tables
Fig. 1: Conceptual framework for uncertainty-aware continual learning from heterogeneous scientific data. The framework integrates predictive uncertainty estimation, domain adaptation, and sequential model updating. This study evaluates an offline implementation using simulated and experimental 2D XRD data; applications to other characterization tasks and integration with experimental control remain future directions.
Crystal structure
Simulated materials
Experimental materials
FCC
Ag, Al, Cu, Ni, Pb, Pt
Al, Cu, Ni
BCC
Mo, Nb, Fe, Ta, V, Zr
Mo, Nb, Fe ( α -Fe)
HCP
Be, Hf, Mg, Sc, Ti, Zn
Ti, Zn
4H single crystal
–
SiC
4H polycrystalline
–
SiC
Table 1: Simulated and experimental materials used in this study
Fig. 2: Inner level: Uncertainty-aware simulation-to-experiment learning. The inner level employs SNGP for uncertainty-aware classification and aligns simulated and experimental XRD feature distributions using maximum mean discrepancy. Here, Conv. denotes convolution, while BCC, FCC, and HCP denote body-centered cubic, face-centered cubic, and hexagonal close-packed, respectively.
381,024→2,048 ; linear, without spectral normalization
Table 2: Layers and specifications of the CNN–SNGP classifier. CNN denotes convolutional neural network; SNGP denotes spectral-normalized neural Gaussian process; SN–Dense denotes a spectrally normalized dense layer; and ReLU denotes the rectified linear unit activation. The CNN–SNGP–adapt model uses the same architecture with additional domain-adaptation losses.
Fig. 3: Outer-level sequential model updating with replay and distillation. The frozen teacher from the preceding stage and the student process the same replayed inputs. Logit distillation uses Kullback–Leibler divergence to match the student’s temperature-scaled class probabilities to the teacher’s. Feature distillation uses a normalized squared error between their 256-dimensional pre-head features. The student is updated using the combined objective in Eq. (14).
Fig. 4: Sequential organization of simulated and experimental XRD datasets. Each dashed box represents one model-update stage and contains one simulated data block and one experimental data block. The known BCC, FCC, and HCP label space remains fixed across stages, while new material identities and experimental measurements are introduced over time to create sequential distribution shifts. The illustrated sequence represents one block order; three orders are evaluated in the repeated-training comparison.
Model
Train Acc.
Val. Acc.
Sil. Score
Align. Dist.
Sim.
Exp.
Sim.
Exp.
Val.
Val.
CNN
0.9439±0.0483
0.9423±0.0447
0.9431±0.0398
0.9414±0.0480
0.215±0.059
12.696±0.614
CNN–SNGP
0.9464±0.0443
0.9454±0.0473
0.9409±0.0456
0.9474±0.0456
0.533±0.007
6.838±0.641
CNN–SNGP–adapt
0.9472±0.0455
0.9569±0.0385
0.9518±0.0341
0.9569±0.0417
0.664±0.078
3.113±1.203
Table 3: Training and validation performance and latent-space organization on the D(1) – D(3) subsets. Train Acc. and Val. Acc. denote training and validation accuracy, respectively, reported as proportions. Sil. Score and Align. Dist. denote the silhouette score and simulation-to-experiment centroid alignment distance, both calculated from validation pre-head embeddings. Values are mean ± sample standard deviation across three repeats. Higher accuracy and silhouette scores and lower alignment distances indicate better performance. Sim. and Exp. denote simulated and experimental data, respectively.
Fig. 5: t-SNE visualization of latent spaces learned by different models. (a) A conventional CNN produces a weakly organized embedding with class overlap, (b) CNN–SNGP improves class separation but retains domain-specific separation between simulated and experimental samples, and (c) CNN–SNGP-adapt yields compact, class-consistent clusters with improved simulation-to-experiment alignment.
(A) Material-held-out classification
Model
Simulated
Experimental
Acc. (%)
Bal. Acc. (%)
F1
Acc. (%)
Bal. Acc. (%)
F1
CNN
59.1±11.4
56.4±11.3
0.541±0.135
29.1±10.2
28.8±8.7
0.290±0.069
CNN–SNGP
64.9±9.2
63.0±9.4
0.621±0.105
44.3±4.9
42.6±6.5
0.421±0.059
CNN–SNGP–adapt
61.6±11.9
58.4±13.3
0.562±0.170
50.3±6.9
48.2±6.6
0.499±0.067
Table 4: Classification and predictive-probability quality on material-held-out data. Within each repeat, predictions are pooled across the eight material holdouts before calculating metrics. Values are mean ± sample standard deviation across the three repeat-level estimates. Acc., Bal. Acc., and F1 denote accuracy, balanced accuracy, and macro-F1. NLL denotes negative log-likelihood, Brier denotes the multiclass Brier score, and ECE denotes expected calibration error. Higher classification metrics and lower NLL, Brier, and ECE indicate better performance. CNN denotes convolutional neural network, SNGP denotes spectral-normalized neural Gaussian process, and adapt denotes the domain-adapted pipeline.
Input
Model
AUROC
AUPR
FPR95 (%)
OOD false acceptance (%)
Noise
CNN
0.721±0.098
0.512±0.274
37.320±4.221
100.000±0.000
Noise
CNN–SNGP
0.998±0.001
0.989±0.006
0.816±0.019
0.000±0.000
Noise
CNN–SNGP–adapt
1.000±0.000
1.000±0.000
0.000±0.000
0.000±0.000
SiC
CNN
0.749±0.103
0.541±0.202
32.165±6.332
100.000±0.000
SiC
CNN–SNGP
0.997±0.001
0.992±0.003
0.712±0.114
0.000±0.000
SiC
CNN–SNGP–adapt
1.000±0.000
1.000±0.000
0.000±0.000
0.000±0.000
Table 5: Detection performance for noise-only and silicon carbide (SiC) inputs. Out-of-distribution (OOD) inputs are treated as the positive class. AUROC and AUPR denote the areas under the receiver operating characteristic and precision–recall curves. FPR95 is the false-positive rate on known-class reference inputs at an operating point attaining at least 95% OOD detection. OOD false acceptance is the fraction of OOD inputs accepted at the evaluated rejection operating point. Higher AUROC and AUPR and lower error rates indicate better performance. Values are mean ± standard deviation across repeated runs. CNN denotes convolutional neural network, SNGP denotes spectral-normalized neural Gaussian process, and adapt denotes the domain-adapted pipeline.
Fig. 6: Pre-head latent-space evolution during sequential learning. Rows show (a) joint retraining, (b) naïve fine-tuning, (c) replay-only updating, (d) learning-without-forgetting (LwF)-style distillation, and (e) the proposed continual-learning strategy; columns show stages 1–4. At each stage, all methods are visualized using the same ordered set of accumulated training, validation, and test scans. Blue, orange, and green indicate body-centered cubic (BCC), face-centered cubic (FCC), and hexagonal close-packed (HCP) structures, respectively. Larger black-edged circles denote simulated scans, and small dots denote experimental scans. Annotations identify selected material subsets and the earlier blocks in which they appeared. Each t-SNE panel is fitted independently; absolute positions and distances should not be compared across panels.
Fig. 7: Stage-wise test accuracy under five sequential training strategies. Panels show (a) joint retraining, (b) naïve fine-tuning, (c) replay-only updating, (d) learning-without-forgetting (LwF)-style distillation, and (e) the proposed continual-learning strategy. Rows indicate model-update stages, and columns indicate introduced-block positions (P1–P4) within each evaluated sequence. Block identities at these positions depend on the introduction order. Each cell displays the mean test accuracy, with the standard deviation beneath it. Future-block cells are blank. Accuracies are proportions rounded to two decimals, and all panels use a common 0–1 color scale.
Strategy
Sim. Acc. (%)
Exp. Acc. (%)
Sraw↑
Dalign,std↓
Training time (h)
Joint retraining
99.88±0.22
99.44±0.97
0.904±0.020
1.156±0.078
8.49±1.99
Naïve fine-tuning
95.88±3.20
94.23±3.05
0.408±0.116
6.068±1.013
2.32±0.27
Replay-only
99.75±0.43
97.47±2.19
0.616±0.190
4.165±1.754
4.69±0.37
LwF-style distillation
93.51±2.49
89.31±0.24
0.353±0.049
5.787±0.834
1.63±0.24
Proposed CL
99.63±0.37
98.31±1.93
0.761±0.066
2.788±0.594
4.36±0.65
Table 6: Final-stage classification performance, latent-space diagnostics, and cumulative training time for five primary strategies. Results average orders within each seed and report mean ± sample standard deviation across three seeds. Accuracy pools test scans from all introduced blocks. Sim. and Exp. denote simulated and experimental data. Sraw denotes the raw validation silhouette score; Dalign,std denotes the standardized source–target class-centroid gap. Arrows indicate the preferred direction. CL denotes continual learning. Training time includes the complete sequence, validation, GP precision reconstruction, and retries, excluding test exports and separate plotting.
Strategy
AA (%)
Forgetting (pp)
BWT (pp)
FWT (pp)
Old (%)
New (%)
Joint retraining
99.56±0.77
−0.15±1.68
0.89±0.77
29.12±23.63
99.41±1.03
100.00±0.00
Naïve fine-tuning
93.11±3.40
4.00±4.00
−3.41±3.83
26.13±20.95
95.85±4.22
84.89±16.88
Replay-only
98.00±1.73
2.52±2.19
−1.63±3.37
20.97±19.57
97.33±2.31
100.00±0.00
LwF-style distillation
88.63±2.65
−0.63±1.67
3.39±3.34
14.55±23.04
90.33±6.20
83.56±14.26
Proposed CL
98.63±1.52
0.84±2.41
−0.55±2.78
22.38±18.72
98.32±2.08
99.56±0.77
Table 7: Final experimental-domain retention and transfer metrics, using the aggregation in Table 6 . Average accuracy (AA) weights blocks equally. Forgetting, backward transfer (BWT), and forward transfer (FWT) are in percentage points (pp). Old and New denote task-average accuracy on earlier blocks and accuracy on the latest introduced block, respectively. Negative signed forgetting indicates improvement beyond the best earlier post-introduction accuracy. BWT references accuracy at introduction; FWT references the untrained model. CL denotes continual learning.
Strategy
NLL
ECE (%)
Joint retraining
0.017±0.025
0.88±1.12
Naïve fine-tuning
0.228±0.103
12.03±3.97
Replay-only
0.100±0.045
6.82±2.15
LwF-style distillation
0.391±0.035
19.38±5.37
Proposed CL
0.088±0.024
6.58±1.67
Table 8: Final experimental predictive-probability quality for five primary strategies using the same CNN–SNGP–adapt backbone. Results use the aggregation in Table 6 . NLL denotes negative log-likelihood, ECE denotes expected calibration error, and CL denotes continual learning. ECE uses 15 equal-width confidence bins; NLL uses natural logarithms. Lower values indicate better performance. Probabilities use the analytical mean-field calculation without post-hoc temperature scaling.
Fig. A1: Resolution-convergence analysis for XRD image resizing. (a) Representative original and resized-restored XRD images. (b) Mean radial peak-position error as a function of the resized image size. Error bars indicate the standard deviation across images.
Dataset
Domain
Materials
Total
Train
Val.
Test/Eval.
Known-class inventory
Simulation
Ag, Al, Be, Cu, Fe, Hf, Mg, Mo, Nb, Ni, Pb, Pt, Sc, Ta, Ti, V, Zn, Zr
900
–
–
–
Known-class inventory
Experiment
Al, Cu, Fe ( α -iron), Mo, Nb, Ni, Ti, Zn
800
–
–
–
Per-fold development
Simulation
Listed simulated materials excluding m
850
680
85
85
Per-fold outer test
Simulation
m : Al, Cu, Fe, Mo, Nb, Ni, Ti, or Zn
50
0
0
50
Per-fold development
Experiment
Al, Cu, Fe, Mo, Nb, Ni, Ti, Zn, excluding m
700
560
70
70
Per-fold outer test
Experiment
m : Al, Cu, Fe, Mo, Nb, Ni, Ti, or Zn
100
0
0
100
Table A1: Material inventory and per-fold dataset roles in the eight-fold leave-one-material-out evaluation. The experimental material set is Mexp={Al,Cu,Fe,Mo,Nb,Ni,Ti,Zn} . Each fold holds out one material m from this set, reserving all its simulated and experimental scans for outer testing. The remaining data in each domain are partitioned into training, validation, and internal test subsets using an 80%/10%/10% split. Experimental training contains 56 labeled anchors (10% of the training subset) and 504 unlabeled scans per fold. Validation data are used for early stopping and model selection; validation and test scans are excluded from anchor selection. The Test/Eval. column reports internal test counts for development rows and independent evaluation counts for outer-test and OOD rows. SiC and noise-only images belong to a separate uncertainty evaluation.
Partition
Domain
Materials
Total
Train
Val.
Test
Dsim(1)
Simulation
Al, Be, Cu, Fe, Hf, Mg, Mo, Pb, Pt, Sc, Ta, Zn, Zr
247
205
20
22
Dexp(1)
Experiment
Al, Fe, Mo, Ti, Zn
233
186
22
25
Dsim(2)
Simulation
Al, Be, Cu, Hf, Mg, Mo, Nb, Pb, Pt, Sc, Ta, Zn, Zr
252
200
26
26
Dexp(2)
Experiment
Al, Mo, Nb, Ti, Zn
233
192
22
19
Dsim(3)
Simulation
Al, Be, Cu, Hf, Mg, Mo, Ni, Pb, Pt, Sc, Ta, Zn, Zr
251
197
28
26
Dexp(3)
Experiment
Al, Mo, Ni, Ti, Zn
234
184
25
25
Table A2: Dataset partitions used in the main domain-incremental sequential-update evaluation. Block identities remain fixed when their introduction order changes. Image-level training, validation, and test roles are preserved. Experimental blocks D1 – D3 share Al, Mo, Ti, and Zn, with Fe, Nb, and Ni occurring in D1 , D2 , and D3 , respectively; experimental D4 contains FCC Cu only. Every simulated block contains all three crystal classes. Recurring materials use distinct image identities but may include correlated measurements of the same specimen.
Element
ICSD record number
Ag
ICSD 64706
Al
ICSD 64700
Cu
ICSD 136042
Ni
ICSD 37502
Pb
ICSD 96501
Pt
ICSD 64919
Table A3: Record numbers for ICSD CIF files, accessed from FIZ Karlsruhe on September 12, 2025
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Type
Material
Structure
Exp. (s)
CL block: Tr/V/Te
CL anchors / hidden
Fixed / grouped roles
Sim.
Ag
FCC
N/A
D4: 42/4/4
All training labels
L: full material holdout (50) G: development only
Sim.
Al
FCC
N/A
D1: 14/2/1 D2: 13/3/0 D3: 14/2/1
All training labels
L Tr/V/Te: 41/7/2 G: held out / development
Sim.
Be
HCP
N/A
D1: 13/1/2 D2: 13/2/2 D3: 13/2/2
All training labels
L Tr/V/Te: 39/5/6 G: development only
Sim.
Cu
FCC
N/A
D1: 14/1/1 D2: 15/1/1 D3: 14/1/2
All training labels
L Tr/V/Te: 43/3/4 G: held out / development
Sim.
Fe
BCC
N/A
D1: 43/1/6
All training labels
L Tr/V/Te: 43/1/6 G: held out / development
Sim.
Hf
HCP
N/A
D1: 13/3/1 D2: 13/2/2 D3: 12/3/1
All training labels
L Tr/V/Te: 38/8/4 G: development only
Appendix
Table A4: Dataset composition and evaluation assignments. CL denotes continual learning; Tr/V/Te gives training/validation/test image counts. The experimental anchors/hidden entries identify the requested overall training-label budget of 10%; they do not give realized counts for each row or block. The remaining training labels are hidden. The fixed and grouped comparisons also have a requested 10% experimental training-label budget. L denotes the fixed D(1) – D(3) comparison, and G denotes the eight-material held-out evaluation with three repeats. “Held out / development” means that a material is the outer test set in its own holdout fold and development data in other folds. Exp. gives exposure time; N/A denotes not applicable.
Fig. A2: Experimental validation accuracy of CNN–SNGP–adapt versus the requested training-anchor fraction. Markers and error bars show the mean and sample standard deviation across three initializations. All configurations retain the same 77 labeled experimental validation scans.
Replay capacity (images)
0
32
64
128
256
Exp. Acc. (%)
88.19±0.73
94.94±7.70
95.78±7.31
97.05±1.46
97.89±0.73
Appendix
Table A5: Sensitivity to replay-memory capacity. Proposed continual learning is evaluated using the reference order D1 – D2 – D3 – D4 . Values are mean ± sample standard deviation across three random seeds. Exp. Acc. denotes final-stage experimental accuracy pooled across test scans from all four blocks. These reference-order results are reported separately from the order-averaged primary comparison. Zero capacity disables replay supervision and replay-based distillation.
Evaluation set
CNN–DKL
CNN–SNGP -adapt
D(1) – D(3) , Sim. train
100.00
100.00
D(1) – D(3) , Exp. train
100.00
100.00
D(1) – D(3) , Sim. test
100.00
100.00
D(1) – D(3) , Exp. test
100.00
100.00
D(4) , Sim. holdout
98.00
98.00
D(4) , Exp. Cu holdout
43.00
61.00
Appendix
Table A6: Classification accuracy (%) of CNN–DKL and CNN–SNGP-adapt. CNN denotes convolutional neural network, DKL denotes deep kernel learning, and SNGP denotes spectral-normalized neural Gaussian process. The suffix “adapt” indicates domain adaptation. Sim. and Exp. denote simulated and experimental data. D(4) is evaluated separately from D(1) – D(3) .
Stanford University, Stanford, CA, USA · University of Toronto, Toronto, ON, Canada · Vector Institute for Artificial Intelligence, Toronto, ON, Canada +3
Physik-Institut, Universität Zürich, Winterthurerstrasse 190, CH-8057 Zürich, Switzerland · Jeremiah Horrocks Institute for Mathematics, Physics and Astronomy, University of Central Lancashire, Preston PR1 2HE, United Kingdom · Department of Physics, Clarendon Laboratory, University of Oxford, Oxford OX1 RPU, United Kingdom +3