Reliability-Aware Checkpoint Selection for Domain Generalization

Jinshi Liu; Jiahao Li; Pan Liu; Yanfeng Li; Rui Qian; Zhao Tong; Yue Sun; Tao Tan

cs.LGSep 30, 2026

Reliability-Aware Checkpoint Selection for Domain Generalization

Authors: Jinshi Liu, Jiahao Li, Pan Liu, Yanfeng Li, Rui Qian, Zhao Tong, Yue Sun, Tao Tan

Organizations: Shenzhen University · Xiamen University · The Hong Kong University of Science and Technology (Guangzhou) · Macao Polytechnic University · Fudan University · Institute of Information Engineering, Chinese Academy of Sciences

Abstract

Checkpoint selection in domain generalization often relies on source-validation accuracy, yet the selected checkpoint need not provide reliable probabilities on unseen target domains. Source-target distribution shifts can alter accuracy rankings, while accuracy alone does not measure predictive probability quality. We identify an empirical selection opportunity within fixed training trajectories: reselecting among checkpoints with near-optimal source accuracy can improve mean target probability quality with small observed changes in mean target accuracy. We study accuracy-constrained reliability selection (AC), which retains checkpoints within a tolerance of the best source-validation accuracy and ranks them by source reliability. Our reference rule aggregates within-set normalized negative log-likelihood (NLL) and class-wise calibration error (CwECE) using $D_\infty$ . AC uses no target data and requires neither additional training nor weight averaging. We evaluate five domain generalization training algorithms on three benchmarks, using PACS to develop the objectives and a 0.5-percentage-point tolerance. In exploratory aggregation comparisons on 360 OfficeHome and TerraIncognita runs, the reference rule reduces mean target soft-bin squared-gap ECE and CwECE by 0.240% and 0.182%, respectively, and NLL by 0.030 relative to Source-Acc. Mean target accuracy changes by +0.213 percentage points. These results identify opportunities for reliability-aware reselection, while the additional benefit of joint over single-objective ranking remains unresolved.

Figures & tables

Figure 1: Checkpoint selection and the accuracy–reliability trade-off. (a) A representative OfficeHome/VREx trajectory, with the source-accuracy tolerance marked and checkpoints colored by source reliability. (b) Mean target accuracy versus a diagnostic target reliability distance across the pooled 540 trajectories, including PACS development, with the high-accuracy region magnified. Objective-set abbreviations are defined in Section 3.2 . The plotted target distance is not AC’s source-side selection score. Panel (b) is descriptive; target metrics are used only for evaluation.

Figure 2: Overview of the checkpoint selection process. The figure illustrates how checkpoints are evaluated based on source accuracy and reliability, and how the accuracy-constrained selection criterion is applied.

Dataset	Selector	Gap	Acc. $\uparrow$	ECE $\downarrow$	CwECE $\downarrow$	NLL $\downarrow$
OfficeHome	Source-Acc	0.0000	60.87	2.77	3.32	4.2724
	AC-NC	0.1612	61.06	2.48	3.05	4.2362
TerraInc.	Source-Acc	0.0000	42.74	9.10	11.35	2.3438
	AC-NC	0.1091	42.97	8.90	11.25	2.3210
PACS (dev.)	Source-Acc	0.0000	80.73	1.63	2.75	0.7794
	AC-NC	0.1647	80.55	1.59	2.75	0.7728

Table 1: Target results for AC-NC (

D_{\infty}

\delta=0.5

). Top: dataset means, 180 runs each; Gap is the source-accuracy deficit (pp). Bottom: AC-NC minus Source-Acc on the 360 post-development runs, with 95% paired intervals. Accuracy differences are in pp, calibration is scaled by 100, and NLL is unscaled. PACS is development; distance choice is exploratory.

Figure 3: Post-hoc target diagnostics. (a) Errors among predictions with confidence $\geq 90\%$ ; labels show errors/high-confidence predictions. (b,c) OfficeHome Art/ERM: true-label probability and incorrect-prediction confidence on 642 shared errors; dashed lines indicate equality. Samples use the aligned target in split; CwECE in (a) is unscaled.

Comparator	$\Delta$ Acc.	$\Delta$ ECE	$\Delta$ CwECE	$\Delta$ NLL
AC-NLL	$+0.0223$	$-0.0240$	$-0.0424$	$-0.0003$
	$[-0.1061,+0.1458]$	$[-0.0900,+0.0460]$	$[-0.0948,+0.0097]$	$[-0.0084,+0.0083]$
AC-CwECE	$-0.0423$	$+0.0092$	$+0.0084$	$+0.0013$
	$[-0.1853,+0.1029]$	$[-0.0942,+0.1303]$	$[-0.0590,+0.0753]$	$[-0.0118,+0.0164]$

Table 2: Joint versus single-objective ranking on 540 paired runs, including PACS development (descriptive). Entries are AC-NC minus the comparator, with 95% paired intervals. All selectors use the same

\Theta_{0.5}

. Accuracy differences are in pp; calibration is scaled by 100. Full precision and secondary outcomes: Appendix Table E.7 .

Selector	Gap	Acc. $\uparrow$	ECE $\downarrow$	CwECE $\downarrow$	NLL $\downarrow$
Source-Acc	0.0000	80.7253	1.6265	2.7526	0.7794
AC-ECE	0.2167	80.2406	1.5928	2.8638	0.7835
AC-NLL	0.1689	80.3712	1.6402	2.8355	0.7787
AC-CwECE	0.1660	80.5422	1.6172	2.8025	0.7801
AC-NC	0.1647	80.5523	1.5924	2.7501	0.7728
AC-mean	0.1933	80.3927	1.6477	2.8624	0.8020

Table 3: Ranking objectives and feasible-set controls on PACS development only (180 runs,

\delta=0.5

). Gap is the source-accuracy deficit (pp); accuracy and calibration are scaled by 100. These descriptive means do not establish superiority of joint ranking.

Figure 4: Cross-algorithm and cross-dataset evaluation of accuracy-constrained reliability selection. (a) Mean paired changes in target accuracy (stars, percentage points) and NLL (circles, original scale) relative to Source-Acc for five DG training algorithms, pooled over OfficeHome, PACS, and TerraIncognita. Open markers denote AC-NC/ $D_{2}$ and filled markers denote AC-NEC/ $D_{\infty}$ . (b) Mean source–target Kendall $\tau_{b}$ of checkpoint rankings for accuracy, NLL, ECE, and CwECE across the same algorithms and datasets; bars show 95% confidence intervals and shading shows the panel mean. Target NLL decreases for most algorithm–selector combinations, although the effects and rank agreement vary across datasets.

Selector	Candidates	Gap	Acc. $\uparrow$	ECE $\downarrow$	CwECE $\downarrow$	NLL $\downarrow$
Source-Acc	–	0.0000	61.44	4.50	5.81	2.4652
AC-NC ( $\delta=0.1$ )	1.30	0.0063	61.46	4.46	5.78	2.4610
AC-NC ( $\delta=0.5$ )	3.92	0.1450	61.53	4.33	5.69	2.4433
AC-NC ( $\delta=1.0$ )	9.71	0.3516	61.51	4.19	5.62	2.1109

Table 4: Tolerance sensitivity on 540 pooled runs (including PACS development). Candidates is the mean feasible-set size; Gap is the mean source-accuracy deficit (pp). Accuracy and calibration are scaled by 100.

Selector	Acc. (%) $\uparrow$	ECE $\downarrow$	CwECE $\downarrow$	NLL $\downarrow$
Source-Acc	51.94	$5.578\to 5.003$	$7.247\to 6.468$	$2.183\to 1.950$
AC-NC	51.99	$5.481\to 4.993$	$7.155\to 6.497$	$2.190\to 1.969$

Table 5: Separate source-fitted TS study on OfficeHome/TerraIncognita. Entries equally average 10 dataset–algorithm cells (36 matched runs each); arrows show raw

\to

TS values. Calibration is scaled by 100.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: Target-domain examples comparing Source-Acc and reliability-selected checkpoints. The left columns show a source-domain class exemplar and a target image. Each heat map masks one $16\times 16$ patch: $\Delta$ confidence is the decrease in top-1 probability (purple: support for the prediction; blue: counter-evidence), whereas $\Delta$ true-label NLL is the increase in true-label loss (green: support for the true class; magenta: conflicting evidence). Colors are comparable within a metric, not between metrics. Right-hand bars show the three largest softmax probabilities.

Dataset	Algorithm	Gap (pp)	$\Delta$ target acc.	$\Delta$ ECE	$\Delta$ CwECE	$\Delta$ NLL	$\Delta$ WHD
OfficeHome	CORAL	0.2159	$+0.13$	$-0.39$	$-0.33$	$-0.0468$	$-0.09$
	ERM	0.1798	$+0.35$	$-0.29$	$-0.32$	$-0.0438$	$-0.05$
	GroupDRO	0.2264	$+0.20$	$-0.45$	$-0.39$	$-0.0527$	$+0.00$
	IRM	0.0449	$+0.09$	$-0.07$	$-0.09$	$-0.0074$	$+0.00$
	VREx	0.1391	$+0.19$	$-0.23$	$-0.22$	$-0.0302$	$-0.12$
PACS	CORAL	0.1677	$-0.31$	$-0.06$	$+0.09$	$+0.0000$	$+0.97$

Appendix

Table C.1: Default AC-NC minus Source-Acc for each dataset–algorithm group (36 matched runs per row). Gap is the Source-Acc minus AC-NC source-accuracy difference. Accuracy and worst held-out-domain (WHD) deltas are in percentage points; calibration deltas are multiplied by 100, and NLL retains its original scale.

Selector	Acc. $\uparrow$	ECE $\downarrow$	CwECE $\downarrow$	NLL $\downarrow$
Source-Acc	60.85	2.61	3.26	2.1211
AC-NC	60.97	2.36	3.05	2.1156
checkpoint-SWAD + BN recal.	62.62	1.95	2.91	2.0026
checkpoint-SWAD, no BN recal.	48.36	11.99	3.35	9567.3991

Appendix

Table D.1: Design-matched checkpoint-SWAD results on OfficeHome (180 runs). Accuracy is in percent; Gaussian soft-bin squared-gap ECE and CwECE are multiplied by 100; NLL is unscaled. The two checkpoint-SWAD variants use identical averaged parameters and differ only in BatchNorm handling.

Algorithm	Dataset	Selector	Acc. $\uparrow$	NLL $\downarrow$	ECE $\downarrow$	CwECE $\downarrow$
ERM	OfficeHome	Source-Acc	65.786	1.501	11.661	0.716
		AC	65.636	1.507	11.603	0.719
		LODO-Acc	65.754	1.447	9.349	0.699
		AC-LODO	65.720	1.443	8.621	0.703
	TerraIncognita	Source-Acc	46.199	2.487	32.301	13.131
		AC	45.809	2.446	32.070	13.002

Appendix

Table D.2: Fixed-hyperparameter LODO results (36 runs per row). Accuracy is in percent; hard-bin ECE and class-frequency-weighted CwECE are multiplied by 100; NLL is unscaled.

Scope	Metric	AC $-$ LODO-Acc	AC-LODO $-$ LODO-Acc
Fixed	Accuracy	$+0.196\;[-0.755,+1.291]$	$-0.062\;[-0.441,+0.367]$
	NLL	$+0.122\;[+0.061,+0.183]$	$-0.020\;[-0.055,+0.005]$
	ECE	$+2.753\;[+1.837,+3.681]$	$-0.562\;[-1.160,-0.099]$
	CwECE	$+0.198\;[-0.064,+0.468]$	$-0.017\;[-0.130,+0.077]$
Selected	Accuracy	$-0.485\;[-1.206,+0.473]$	$+0.175\;[-0.074,+0.486]$
	NLL	$+0.286\;[+0.188,+0.388]$	$-0.032\;[-0.064,-0.004]$

Appendix

Table D.3: Paired LODO contrasts, averaged equally over the two algorithms and two datasets. Each entry is the mean difference followed by a 95% bootstrap interval. Positive accuracy and negative error differences favor the first selector. Hyperparameter-selection intervals are descriptive because only three trial blocks are available.

Dataset	Selector	Acc. (%) $\uparrow$	ECE ( $\times 100$ ) $\downarrow$	CwECE ( $\times 100$ ) $\downarrow$	NLL $\downarrow$	WHD (%) $\uparrow$	WC (%) $\uparrow$
OfficeHome	Source-Acc	57.44	2.77	2.92	1.9327	45.61	12.08
	PAIR iid-last10	50.03	2.11	3.38	2.2105	38.58	10.92
	PAIR val-filter	57.44	2.94	3.12	1.9435	45.95	11.93
	AC-NC	57.63	2.54	2.70	1.9025	45.49	12.00
PACS	Source-Acc	79.79	2.05	3.17	0.7219	70.53	56.69
	PAIR iid-last10	70.12	1.88	2.26	0.9669	62.25	49.57

Appendix

Table D.4: VREx-only comparison over 36 runs per dataset. Acc., WHD, and WC are percentages; ECE and CwECE are multiplied by 100, while NLL retains its original scale. WHD and WC denote worst held-out-domain and worst-class accuracy, respectively.

NC distance	$\Delta$ Acc.	$\Delta$ ECE	$\Delta$ CwECE	$\Delta$ NLL
$D_{1}$	$+0.2209$	$-0.2337$	$-0.1611$	$-0.0289$
	$[-0.0464,+0.5072]$	$[-0.3975,-0.0853]$	$[-0.2640,-0.0579]$	$[-0.0473,-0.0115]$
$D_{2}$	$+0.2234$	$-0.2428$	$-0.1754$	$-0.0296$
	$[-0.0448,+0.5093]$	$[-0.4062,-0.0930]$	$[-0.2783,-0.0728]$	$[-0.0480,-0.0120]$
$D_{\infty}$	$+0.2133$	$-0.2395$	$-0.1821$	$-0.0295$
	$[-0.0592,+0.5017]$	$[-0.4038,-0.0903]$	$[-0.2841,-0.0803]$	$[-0.0483,-0.0119]$

Appendix

Table E.1: Paired changes from Source-Acc on OfficeHome and TerraIncognita (360 runs). Accuracy differences are in percentage points; calibration differences are multiplied by 100. The second line gives 95% percentile bootstrap intervals.

Target metric	Mean difference	95% paired bootstrap CI
Accuracy	$-0.0095$	$[-0.0561,+0.0270]$
ECE	$-0.0065$	$[-0.0270,+0.0122]$
CwECE	$-0.0200$	$[-0.0462,+0.0024]$
NLL	$-0.0006$	$[-0.0035,+0.0019]$

Appendix

Table E.2: NC

D_{\infty}

minus NC

D_{1}

on all 540 runs at

\delta=0.5

. Accuracy differences are in percentage points; calibration differences are multiplied by 100.

Objectives	Distance	$\Delta$ src. acc.	$\Delta$ tgt. acc.	$\Delta$ ECE	$\Delta$ CwECE	$\Delta$ NLL	$\Delta$ WHD
NC	$D_{1}$	$-0.1455$	$+0.0940$	$-0.1645$	$-0.0995$	$-0.0213$	$+0.1788$
NC	$D_{2}$	$-0.1442$	$+0.0848$	$-0.1723$	$-0.1109$	$-0.0218$	$+0.1745$
NC	$D_{\infty}$	$-0.1450$	$+0.0845$	$-0.1710$	$-0.1195$	$-0.0219$	$+0.1965$
NE	$D_{1}$	$-0.1472$	$+0.0159$	$-0.1489$	$-0.0844$	$-0.1583$	$+0.1483$
NE	$D_{2}$	$-0.1469$	$+0.0007$	$-0.1438$	$-0.0792$	$-0.1574$	$+0.1583$
NE	$D_{\infty}$	$-0.1457$	$-0.0001$	$-0.1345$	$-0.0762$	$-0.1568$	$+0.1677$

Appendix

Table E.3: All nine objective-set and aggregation combinations at

\delta=0.5

, as paired mean changes from Source-Acc over 540 runs. Accuracy and WHD differences are in percentage points; calibration differences are multiplied by 100.

Target metric	Mean difference	95% paired bootstrap CI
Accuracy	$+0.0044$	$[-0.0827,+0.0974]$
ECE	$-0.0210$	$[-0.0653,+0.0186]$
CwECE	$-0.0187$	$[-0.0560,+0.0165]$
NLL	$-0.1398$	$[-0.4150,+0.0002]$

Appendix

Table E.4: NEC minus NC at fixed

D_{\infty}

and

\delta=0.5

on 540 paired runs. Accuracy is in percentage points, calibration differences are multiplied by 100, and NLL is unscaled.

Selector	Gap	Src. acc. $\uparrow$	Src. NLL $\downarrow$	Src. CwECE $\downarrow$	WHD $\uparrow$	WC $\uparrow$
AC-NLL	0.138017	84.4905	1.269184	2.2609	51.5809	24.5767
AC-CwECE	0.155944	84.4725	1.292866	2.1405	51.5854	24.5787
AC-NC	0.145010	84.4835	1.271047	2.1888	51.6157	24.6432

Appendix

Table E.5: Source selection behavior and secondary target outcomes for the component ablation. Each rule uses the same feasible sets, with 3.920370 candidates on average. Gap is in percentage points; accuracy, CwECE, WHD, and worst-class accuracy (WC) are multiplied by 100. WHD averages 135 configuration-level minima.

Dataset	Selector	Gap	Acc. $\uparrow$	ECE $\downarrow$	CwECE $\downarrow$	NLL $\downarrow$	WHD $\uparrow$	WC $\uparrow$
OfficeHome	AC-NLL	0.1432	61.0523	2.5276	3.0798	4.240746	48.2747	12.8679
	AC-CwECE	0.1686	61.0611	2.4858	3.0615	4.236254	48.2056	13.0297
	AC-NC	0.1612	61.0613	2.4828	3.0488	4.236244	48.2139	12.9993
PACS (dev.)	AC-NLL	0.1689	80.3712	1.6402	2.8355	0.778687	71.8626	60.1070
	AC-CwECE	0.1660	80.5422	1.6172	2.8025	0.780122	72.1708	59.8490
	AC-NC	0.1647	80.5523	1.5924	2.7501	0.772756	72.1615	60.0456

Appendix

Table E.6: Component ablation by dataset (180 runs each,

\delta=0.5

). Gap is in percentage points; target accuracy, ECE, CwECE, WHD, and WC are multiplied by 100. NLL retains its original scale.

	AC-NC minus AC-NLL		AC-NC minus AC-CwECE
Metric	Mean $\Delta$	95% CI	Mean $\Delta$	95% CI
Source accuracy	$-0.006993$	$[-0.018585,+0.004787]$	$+0.010934$	$[+0.000081,+0.022039]$
Target accuracy	$+0.022297$	$[-0.106108,+0.145756]$	$-0.042315$	$[-0.185333,+0.102861]$
ECE	$-0.0240$	$[-0.0900,+0.0460]$	$+0.0092$	$[-0.0942,+0.1303]$
CwECE	$-0.0424$	$[-0.0948,+0.0097]$	$+0.0084$	$[-0.0590,+0.0753]$
NLL	$-0.000260$	$[-0.008409,+0.008276]$	$+0.001312$	$[-0.011792,+0.016416]$

Appendix

Table E.7: Paired component differences (AC-NC minus each comparator), with 95% percentile bootstrap intervals. Accuracy differences are in percentage points; calibration differences are multiplied by 100; NLL retains its original scale.

$\delta$ (pp)	Candidates	Gap (pp)	Acc. $\uparrow$	ECE $\downarrow$	CwECE $\downarrow$	NLL $\downarrow$
0.1	1.30	0.0063	61.46	4.46	5.78	2.4610
0.5	3.92	0.1450	61.53	4.33	5.69	2.4433
1.0	9.71	0.3516	61.51	4.19	5.62	2.1109

Appendix

Table E.8: NC

D_{\infty}

tolerance ablation on 540 runs. Gap is the mean source-accuracy deficit in percentage points. Accuracy and calibration values are multiplied by 100.

Dataset	$\delta$	Candidates	Gap	Acc.	ECE	CwECE	NLL
OfficeHome	0.1	1.28	0.0075	60.90	2.69	3.26	4.2633
	0.5	3.80	0.1612	61.06	2.48	3.05	4.2362
	1	9.44	0.4164	60.99	2.46	2.98	3.2860
PACS (dev.)	0.1	1.40	0.0072	80.77	1.59	2.75	0.7741
	0.5	5.20	0.1647	80.55	1.59	2.75	0.7728
	1	14.23	0.3686	80.30	1.59	2.82	0.7766

Appendix

Table E.9: NC

D_{\infty}

tolerance ablation by dataset (180 runs per dataset). Gap is in percentage points; accuracy and calibration values are multiplied by 100.

Tolerance	Metric	Mean difference	95% CI
0.1	Source accuracy	$+0.1387$	$[+0.1245,+0.1530]$
	Target accuracy	$-0.0649$	$[-0.2724,+0.1364]$
	ECE	$+0.1314$	$[+0.0317,+0.2375]$
	CwECE	$+0.0879$	$[+0.0175,+0.1588]$
	NLL	$+0.017667$	$[+0.0047,+0.0311]$
1.0	Source accuracy	$-0.2066$	$[-0.2328,-0.1798]$

Appendix

Table E.10: NC

D_{\infty}

differences relative to

\delta=0.5

on 540 run pairs. Accuracy differences are in percentage points; calibration differences are multiplied by 100. Intervals are 95% paired percentile bootstrap intervals from 10,000 resamples (seed 20260924).

Dataset	Selector	Src. acc.	Tgt. acc.	ECE	CwECE	NLL $\downarrow$	Step
OfficeHome	Source-Acc	76.14	60.87	2.77	3.32	4.2724	2553
	pure ECE	2.96	2.55	0.04	0.04	4.1890	164 (94.4%)
	pure CwECE	2.07	1.91	0.03	0.01	4.2168	173 (95.6%)
	pure NLL	74.54	60.40	1.92	2.82	1.7333	1723
PACS	Source-Acc	93.68	80.73	1.63	2.75	0.7794	2656
	pure ECE	85.10	73.58	0.94	2.54	0.7927	1264 (7.8%)

Appendix

Table E.11: Pure calibration-only selection can select degenerate low-accuracy checkpoints. Source and target accuracy are percentages; ECE and CwECE are multiplied by 100, while NLL retains its original scale. Step is the mean selected training step; parenthetical values report the percentage of runs selecting step 0.

Dataset	Algorithm	Ckpts	Src. acc. range	Src. NLL range	Src. CwECE range	Tgt. NLL range	Tgt. CwECE range
OfficeHome	CORAL	4.33	0.3639	0.0689	0.0063	0.1498	0.0091
	ERM	4.03	0.3283	0.0441	0.0051	0.1196	0.0086
	GroupDRO	5.08	0.3753	0.0642	0.0064	0.1691	0.0101
	IRM	2.11	0.0979	2.1954	0.0012	11.3938	0.0024
	VREx	3.44	0.2772	0.0388	0.0039	0.0852	0.0062
PACS	CORAL	5.06	0.3727	0.0336	0.0032	0.2091	0.0138

Appendix

Table E.12: Reliability variation within

\Theta_{0.5}

(36 runs per row). Source-accuracy ranges are in percentage points; CwECE ranges use the

[0,1]

scale; NLL ranges remain on their original scale.

Dataset	Source proxy	Spearman $\rho$ $\uparrow$	Kendall $\tau$ $\uparrow$	Checkpoints	Valid/total
OfficeHome	source-out accuracy	$-0.045$	$-0.059$	684	104/180
	AC utility	$+0.513$	$+0.438$	684	104/180
PACS	source-out accuracy	$+0.138$	$+0.122$	936	128/180
	AC utility	$+0.092$	$+0.074$	936	128/180
TerraInc.	source-out accuracy	$+0.065$	$+0.068$	497	74/180
	AC utility	$+0.114$	$+0.113$	497	74/180

Appendix

Table E.13: Mean within-

\Theta_{0.5}

rank correlation. Source accuracy is paired with target accuracy; source NC utility is paired with target NC utility. Valid runs have at least three feasible checkpoints and nonconstant utilities; ties receive average ranks for Spearman

\rho

Item	Setting
Backbone	Code default: ImageNet-1K-pretrained ResNet-50 ( resnet50.ram_in1k ).
Source split	Code default: 20% validation per source domain; target domain excluded from selection.
Trajectory	5001 updates; logged steps $0,100,\ldots,5000$ (51 per run).
Seeds	Hyperparameter seed 0: defaults; 1–2: registry draws. Trial seeds 0–2: splits and, jointly with hyperparameter seeds, draws. Training seed hashes all run keys.
Shared search	Learning rate $10^{-5}$ – $10^{-3.5}$ ; weight decay $10^{-6}$ – $10^{-2}$ ; batch size 8–45; dropout $\{0,0.1,0.5\}$ .
Algorithm search	CORAL penalty weight $\gamma\in[10^{-1},10^{1}]$ ; GroupDRO $\eta\in[10^{-3},10^{-1}]$ ; IRM/VREx penalty weight $10^{-1}$ – $10^{5}$ and annealing step 1–9999.

Appendix

Table G.1: Training-code defaults and logged trajectory steps for the main sweep.

Explore similar work

Jul 29, 2026stat.ML

Chaos Is a LADDER: Domain Generalization Beyond Invariance via Reweighting

Domain generalization (DG) aims to learn from multiple source domains and generalize to unseen target domains. Most DG methods pursue invariance: they seek a causal representation whose prediction rule is invariant across domains. This principle is effective when the causal mechanism is stable, but becomes restrictive when the domain itself modulates how causal content maps to the response. In this case, directly feeding domain style into the predictor can create misleading shortcuts, since style does not by itself cause the response. Yet the apparent chaos of multiple styles can become a ladder: style can locate the unseen target domain among source domains and guide which domain-dependent prediction rules should be trusted. We propose \emph{Latent Adaptive Domain Disentanglement and Environment Reweighting} (LADDER), a fixed-model DG pipeline that learns causal/style representations, freezes the encoders, fits source-specific classifiers, and uses an unlabeled target-domain covariate set only at inference to compute weights over these fixed classifiers, with no target labels or model-state updates. We establish theoretical guarantees for source reweighting and validate LADDER on simulations, FMoW, and a location-grouped iWildCam protocol, with gains in overall and group-averaged accuracy.

Yuhang Jiang, Fengchuan Zhang, Sanguo Zhang +1

School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing, China · Key Laboratory of Big Data Mining and Knowledge Management, Chinese Academy of Sciences, Beijing, China

Multimodal Domain Generalization Reweighting Chaos

Ask PDF

Aug 10, 2026cs.LG

ReliableNet: A Chance-Constrained Approach to Trustworthy Classification in Deep Learning

A prediction that is both confident and wrong is a critical reliability failure because it can bypass abstention and human review precisely when the model is mistaken. Empirical risk minimization (ERM) controls average loss but not this failure directly, while calibration, uncertainty estimation, conformal risk control, and selective prediction methods target related reliability properties rather than bounding the joint failure event during training. We propose ReliableNet, which constrains the Joint Confident-Wrong (JCW) probability, the probability that a prediction is simultaneously confident and incorrect, below a user-specified risk budget

α\in(0,1)

. We formulate this as a chance-constrained ERM problem, use a conservative smooth inner approximation whose population feasibility implies the original JCW constraint. Across four tabular and two image datasets, ReliableNet is the only method certified within the JCW budget for every dataset and seed in distribution, when compared against baselines spanning ERM, post-hoc calibration, conformal risk control, and selective prediction. Under demographic, ambiguity, spurious-correlation, novel-class, and covariate shifts, it achieves the lowest empirical JCW among the compared methods while remaining very competitive in accuracy, coverage, calibration, and selective prediction. Risk-coverage results further indicate that ReliableNet achieves better selective ranking than the benchmark methods on most datasets. Overall, ReliableNet provides a principled approach to trustworthy classification.

Ange-Clément Akazan, Ineza Remy Mugenga, Abebe Geletu +2

School of Agriculture and Science, University of KwaZulu-Natal, Pietermaritzburg, KwaZulu-Natal, South Africa · 2AIMS Research and Innovation Centre, African Institute for Mathematical Sciences, Kigali, Rwanda · 3African Institute for Mathematical Sciences, Kigali, Rwanda

Empirical Risk Minimization Deep Learning

Ask PDF

May 9, 2026cs.LG

TopoGeoScore: A Self-Supervised Source-Only Geometric Framework for OOD Checkpoint Selection

Out-of-distribution (OOD) robustness is difficult to diagnose when target-domain labels are unavailable. We consider a more restrictive source-only variant of unsupervised accuracy estimation: selecting robust checkpoints using only source-domain representations, with no target samples or target labels. We propose \textbf{TopoGeoScore}, a source-only geometric scorer for label-free OOD checkpoint selection. Given a trained checkpoint, we construct class-conditional mutual

k

-nearest-neighbour graphs from source embeddings and extract three interpretable signals: a torsion-inspired reduced Laplacian log-determinant for global class-manifold complexity, Ollivier--Ricci curvature for local neighbourhood regularity, and higher-order topological summaries for fragmented connectivity, loops, and global--local inconsistency. Instead of fixing their weights by hand, TopoGeoScore learns a non-negative linear score through a self-supervised objective that enforces invariance under approximately geometry-preserving embedding views and separation from structure-breaking views. The score remains interpretable and uses no target-domain samples or labels. Results across CIFAR-based corruption and distribution-shift benchmarks, ImageNet-C, MNLI

\to

HANS transfer, and OGBN-Arxiv suggest that source representations contain measurable global--local--topological evidence of robustness, supporting practical checkpoint selection before deployment under distribution shift.

Farid Hazratian, Ali Zia, Hien Duy Nguyen

University of Tehran · Tehran, Iran · La Trobe University +3

Out-Of-Distribution Domain Invariance Geogs-Slam

Ask PDF