What Remains Without Test Labels? Re-evaluating a Graph-Level Anomaly Detection Benchmark

Yujie Lu; Da Yan

cs.LGOct 16, 2026

What Remains Without Test Labels? Re-evaluating a Graph-Level Anomaly Detection Benchmark

Authors: Yujie Lu, Da Yan

Organizations: Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington, Bloomington, Indiana, USA.

Abstract

Graph-level anomaly detectors are trained only on normal graphs, yet benchmarks may use test labels for model selection. We examine whether UB-GOLD results reused by later papers support label-free detection. We audit UB-GOLD and 14 GLAD code bases, re-run 21 detectors on UB-GOLD's folds under selection rules with and without test labels, and evaluate sensitivity to known structural changes. Test labels select checkpoints in UB-GOLD's shared training loop and checkpoints or hyperparameters in 8 of the 14 code bases. UB-GOLD also evaluates its kernel baselines against misaligned labels. Across nine datasets, selecting checkpoints by test AUROC adds an average of 5.12 AUROC points [3.97, 6.24] for the 12 UB-GOLD detectors relative to the final checkpoint and 4.13 [3.32, 4.79] relative to held-out normal graphs. In a post hoc permutation analysis with no truly better checkpoint, picking the best of noisy test evaluations still yields a gain more than half as large as the one observed against the final checkpoint. Configurations tuned on test labels score a further 1.53 points [0.64, 2.42] above repository defaults, a gap that mixes the value of test labels with that of tuning itself. With tuned hyperparameters and label-free checkpoint selection, the two strongest detectors retain their lead, while comparisons below them are less stable. Six of eight detectors evaluated out of sample miss size-preserving composition changes that graphlet counts detect. We recommend reporting results selected without test labels as the main comparisons and release per-run outputs that allow verification without retraining.

Figures & tables

Figure 1: Almost every detector scores lower under label-free checkpoint selection, and the order below the two leaders changes. Mean UB9 AUROC from the same runs with tuned hyperparameters. Hollow markers use test labels under each method’s own rule where available and UB-GOLD’s rule otherwise. Filled markers use Val-Normal , which makes only checkpoint selection label-free. LGKDE’s own rule scores below Val-Normal . ProtoGLAD has no checkpoints. Recent methods are descriptive (Appendix B.2 ).

Figure 2: Test labels can influence reported AUROC through tuning and other design choices, checkpoint selection, and final evaluation. Our target setting permits their use only for final evaluation. The lower panel locates the RQ2 protocols along the hyperparameter and checkpoint axes. Arrows indicate the two estimands.

			Test labels select
Code base	Venue	Commit	Checkpoint	Hyperparam.	Other	Selection rule stated in the paper
Benchmark
UB-GOLD (12 detectors)	ICLR’25	c940457	$\bullet$	$\bullet$	$\circ$	HP search stated as an upper bound, checkpoint rule not described
Recent methods re-evaluated in this paper
MUSE	NeurIPS’24	07da392	L	L	$\circ$	labelled validation split
LGKDE †	ICML’26	1bba5ef	$\bullet$	$\circ$	$\circ$	early stopping on a validation set

Table 1: Test-label selection appears in UB-GOLD’s shared training loop and in 8 of the 14 GLAD code bases from 2023 to 2026. Rows describe the released code at the listed commit, which may differ from the code behind a paper’s numbers. Checkpoint selection includes early stopping. Appendix C gives source locations and the paper version consulted for each entry.

Detector	Oracle	Last	Val-Normal	HP-Free
SIGNET	74.7 (2)	70.7	71.6 (1)	69.3 (1)
CVTGAD	76.1 (1)	71.0	71.1 (2)	66.5 (4)
OCGTL	72.1 (7)	68.7	68.8 (3)	60.1 (15)
GOOD-D	72.3 (6)	67.7	67.6 (4)	66.8 (2)
GLASS †	72.8 (4)	63.8	64.9 (5)	64.5 (5)
OCGIN	70.2 (8)	64.5	64.8 (6)	63.9 (10)

Table 2: With tuned hyperparameters, SIGNET and CVTGAD lead under label-free checkpoint selection, while the order below them changes. Training-free references score within the deep detectors’ range. Mean UB9 AUROC with ranks among 21 detectors, sorted by Val-Normal . Oracle uses each method’s own rule where available. Table 23 uses UB-GOLD’s rule. HP-Free uses defaults with Val-Normal . ProtoGLAD has no checkpoints. † denotes recent methods. Italics denote references.

Figure 3: Test-label checkpoint selection raises every detector’s score, with a larger mean gain for recent methods. UB9 checkpoint leakage is Oracle under UB-GOLD’s rule minus Val-Normal , with the same tuned hyperparameters and 95% dataset-bootstrap intervals. The dotted line marks the pre-specified 2-point threshold. Circles denote UB-GOLD detectors and squares denote recent methods.

	GX (synthetic)			MS-v2 (NCI1)
Detector	T1a	T3a	T3b	MS1	MS2	MS3
GLADC	51.1	50.5	54.6	53.4	56.0	52.6
CVTGAD	69.9	53.2	56.3	68.9	82.0	76.9
GOOD-D	92.7	57.0	60.7	69.3	75.8	69.9
GraphDE	49.9	48.6	50.0	51.7	44.5	50.0
GraphCL-IF	50.3	50.1	49.0	52.8	59.1	52.6

Table 3: Six of the eight confirmatory detectors are blind to size-preserving composition changes that graphlet counts detect. Values are mean Val-Normal AUROC over 5 seeds. Bold scores exceed 60 and therefore do not meet the blindness criterion. The second block reports the 4 pilot detectors descriptively. Intervals and results for the 9 recent methods appear in Appendix D.6 .

Item	What to report	Finding
Training	Training classes and the anomaly class	§ 5.1
Tuning	Data used for tuning, results with defaults	§ 5.2
Checkpoint	Selection rule and the data it uses	§ 5.2
Direction	Score direction, fixed in advance	App. C
Baselines	Source, commit, rule and anomaly class	§ 5.1
References	Size-only and graphlet-count scores	§ 5.3

Table 4: A checklist for GLAD evaluations.

Appendix figures & tables54 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Written	Rule	Information available
A0	10-12 21:37	Records that the pre-registration was written at 20:51 by file time, not at the time in its header. The content is unchanged.	Timing note only.
A1	10-12 21:17	HP-Free sets every option to UB-GOLD’s mymain.py default except the batch sizes. An option whose default crashes takes its modal per-dataset value.	No metric of the added detectors and no HP-Free run. Only the pilot runs had metrics.
A2	10-12 21:17	GraphDE’s REDDIT-B script line cannot be parsed, so its dangling flag is dropped.	As A1.
A3	10-12 21:17	Val-Normal for the added detectors picks the evaluation point with the lowest mean anomaly score of the held-out normals, using each detector’s own test-time scorer.	As A1.
A4	10-12 21:17	CVTGAD is evaluated on its raw score. Oracle replays UB-GOLD’s selection on the normalised score.	As A1.
A5	10-12 21:17	GraphDE’s main score is the negated in-distribution probability, as in its paper. UB-GOLD’s orientation is kept as a reproduction column.	As A1.

Appendix

Table 5: Amendments A0–A16. Dates are month-day in 2026 and times are EDT. The last column states which test metrics existed or had been seen when the amendment was written, or, for a ratification, when the port note it ratifies was written. “Added detectors” are the eight UB-GOLD detectors run after the pilot.

ID	Written	Rule	Information available
A17	10-13 11:01	Primary analyses stay at UB-GOLD commit c940457. A pre-specified secondary analysis re-runs GLocalKD and GLADC at commit a96137b, before PR #3. Defects introduced by PR #3 are attributed to it. UB-GOLD’s Type IV datasets are excluded.	Not stated in the amendment. No a96137b run existed.
A18	10-13 12:42	LGKDE keeps the pre-registered HP-Free definition, its tuned rows are labelled “MUTAG config for all”, and every device uses the original’s CUDA copy semantics. The reproduction target is not met and the method is kept.	LGKDE grid and reproduction results. Its definitions were fixed at 01:31 before any LGKDE metric.
A19	10-13 13:53	SUL/GUL caps its anchor count at the training-set size where the released code would crash. HC-GLAD gets a memory-feasibility rule.	Both rules fixed in port notes before any run of these methods.
A20	10-13 13:55	ProtoGLAD’s tuned setting replays its test-AUROC grid search, reseeded per cell and fold. Detectors without checkpoints are excluded from pooled checkpoint inflation.	ProtoGLAD grid complete, with definitions fixed at 12:49 before its first metric. No pooled statistic for recent methods.
A21	10-13 13:55	A method whose tuned and default settings are identical has no hyperparameter axis. A descriptive flag marks runs whose scores collapse into ties.	SUL/GUL grid complete. No pooled statistic for recent methods.
A22	10-13 14:39	The SUL/GUL reproduction check is not evaluable, because the paper reports only fully contaminated training. A post hoc port check is added, and the row label states 500 epochs against the paper’s 1000.	Paper read after the SUL/GUL grid was complete.

Appendix

Table 6: Amendments A17–A29, continuing Table 5 . Dates are month-day in 2026 and times are EDT.

ID	Written	Rule	Information available
A30	10-14 15:09	Implements A17 on the a96137b tree with two edits that only make it parse. Only the two training loops are swapped, and each run uses the device of its paired c940457 run.	Smoke and sanity checks of the a96137b runner only.
A31	10-14 15:56	The official evaluation runs as soon as all pre-specified experiments are complete, instead of on the planning date. No criterion changes.	Provisional output of the frozen statistics code.
A32	10-14 20:55	Specifies five post hoc robustness analyses on stored outputs, namely seven label-free selectors, a permutation null, dataset robustness with a crossed bootstrap, a size-matched evaluation and reproduction-fidelity tiers (Appendix G ). All are post hoc, and no criterion changes.	Provisional output of the frozen statistics code and the first paper draft. None of these analyses computed.
A33	10-14 23:47	Specifies two descriptive follow-ups. The 12 UB-GOLD detectors are re-run with the class roles swapped on BZR, COX2 and PROTEINS_full, and CVTGAD and GOOD-D are re-run with their precomputed structural encodings replaced (Appendices F.11 and F.12 ).	All main results and the post hoc analyses. No run of either follow-up.
A34	10-15 00:22	Replaces the all-zero input of the ablation by all ones, because a zero input makes both losses divide zero by zero under UB-GOLD’s zero-bias initialisation. Ratifies the build of both follow-ups.	Automatic checks of smoke runs only. No follow-up metric.
A35	10-15 14:22	Fixes the inputs, intervals and handling of missing values for the follow-up analyses.	Run counts and training logs only. No follow-up metric.

Appendix

Table 7: Amendments A30–A39, continuing Table 6 . A36 and A37 specify experiments for a later version and are not listed. Dates are month-day in 2026 and times are EDT.

Entry	Label	Created	Files	State recorded
1	M0	10-13 18:22	4871	Pre-registration and A0–A21. Main grid of the 12 UB-GOLD detectors running. No pooled statistic for recent methods, and no G1 aggregate for the added detectors inspected.
2	M1	10-13 23:57	5314	Statistics code and first molecular generator frozen, A24–A26 written. Before any pooled statistic of the frozen code and before any typed run.
3	M1b	10-14 02:52	5400	Typed-anomaly infrastructure built. Control results on GX and the first generator written before any deep typed run.
4	M2	10-14 04:00	5442	MS-v2 frozen and the first generator retired. Before any deep typed run on GX or MS-v2.
5	(none)	10-14 08:46	5699	Typed grid of the 12 UB-GOLD detectors collected. No pooled statistic computed.
6	M3	10-14 17:53	5819	All main grids complete. Before the first run of the frozen statistics code on real data.

Appendix

Table 8: Entries of the provenance ledger. Dates are month-day in 2026 and times are UTC, four hours ahead of the EDT times in Tables 5 to 7 . “Files” is the number of hashed files. Entries M0, M1, M2 and M7 were also registered with an external registry under embargo.

Outcome	Candidates	Reason
Included, first group (A10)	MUSE (NeurIPS 2024), LGKDE (ICML 2026), GLADPro (KDD 2025), SEGO (AAAI 2025), GLADMamba (ECML-PKDD 2025)	First tier, plus GLADMamba from the second tier, whose code already runs on UB-GOLD’s folds.
Included, second group (A11)	GLASS (preprint 2026), ProtoGLAD (preprint 2026), HC-GLAD (Neural Networks 2026), SUL/GUL (WWW 2026)	Second tier. SUL/GUL runs in its mode without contamination, because UB-GOLD’s training folds contain no anomalies.
Not ported, code read	CLINIC (ICML 2026)	Second tier, but custom CUDA extensions made it costly to build.
Optional tier, not added	HimNet (ECML-PKDD 2023)	Published before 2024. Its code evaluates the final epoch only.
	GRAM (Neural Networks 2024)	Only a demonstration on a dataset outside UB-GOLD is released. Its code selects by training loss.
	FANFOLD (preprint 2024)	Released code does not run as published.

Appendix

Table 9: Inclusion flow for the recent methods. The survey named 46 candidates, and the table gives each with its venue and the outcome. Code availability refers to 12 and 13 October 2026. “Tier” refers to the survey’s priority tiers described in the text. Appendix C.3 gives the code findings for the candidates whose code we read.

Method	Commit	Licence	Tuned setting	Own rule	Reproduction check	Tier
MUSE	07da392	none	Released per-dataset values, which were chosen with labelled anomalies. Modal values for COX2, ENZYMES, MUTAG.	None on test. Labelled validation split, not available in one-class folds.	Met with its own code and splits, 4 of 5. On UB-GOLD folds without labelled selection, 2 of 7.	Incomplete configuration
LGKDE	1bba5ef	MIT	The only released configuration, made for MUTAG, on every dataset.	Early stopping on mean of test AUROC and AUPRC, then the best epoch.	Not met, 1 of 4 (MUTAG), also under its own data pipeline.	Reproduction not met
GLADPro	1f811b9	MIT	README lines for MUTAG, PROTEINS, IMDB-B. Modal values, equal to defaults with 500 epochs, elsewhere.	Epoch with the highest test AUROC averaged over 5 folds.	Contested. Against a third-party re-run, 0 of 3. Two further matches, on datasets with a likely different class, are not counted.	Incomplete configuration, or not met under the second reading
GLADMamba	2b2462c	none	Released scripts. Modal values for PROTEINS and MUTAG.	Epoch with the highest test AUROC averaged over 5 trials.	Met, 6 of 8. The two misses match under its released protocol.	Incomplete configuration
SEGO	45bdda7	MIT	No detection config released. Modal values of its 10 dataset-pair lines.	Last epoch. Oracle uses UB-GOLD’s rule.	Met, 2 of 5. On its own pair task with released lines, 4 of 5.	Incomplete configuration
GLASS	3bbba39	MIT	One released configuration, equal to the defaults.	Test-selected checkpoint, then the best of 24 scoring rules on test.	Met, 5 of 7.	Native

Appendix

Table 10: Ports of the nine recent methods. “Own rule” is the selection rule of the released code, replayed as Oracle -own. “Reproduction check” gives the pre-registered check, own rule and tuned setting against the paper, or for GLADPro against a third-party re-run, as the number of datasets within

\pm 5

AUROC points. Datasets on which the reference uses a different anomaly class are not counted. “Tier” is the reproduction-fidelity tier of Appendix B.2 . Default settings are the repository defaults.

Code base	Repository	Commit
UB-GOLD	UB-GOLD/UB-GOLD	c940457 (a96137b before PR #3)
MUSE	kswoo97/GLAD_MUSE	07da392
LGKDE	MathAdventurer/LGKDE	1bba5ef
GLADPro	ZhenyuYangMQ/GLADPro	1f811b9
GLADMamba	Yali-Fu/GLADMamba (now unavailable, same commit on a public mirror)	2b2462c
SEGO	name-is-what/SEGO	45bdda7

Appendix

Table 11: Repositories and commits of the audited code bases.

Type (v1)	Size	Graphlet count + LW	Size only	Edge-label pair + LW	Valid	Min. anomalies per seed
MS1	preserving	53.5	46.7	49.6	no	302
MS1	changing	57.2	54.0	80.6	no	222
MS2	preserving	77.8	47.3	60.8	no	145
MS2	changing	80.4	59.0	84.6	no	117
MS3	preserving	42.4	44.0	56.4	no	137
MS3	changing	50.9	51.1	82.3	no	101

Appendix

Table 12: MolSwap v1, retired. Control AUROC in points, mean over five seeds. The last column is the smallest number of anomalies in any seed. No size-preserving type reaches 85 on graphlet count + LW, so no type is a valid test and no full deep run was made on v1.

It.	Pool	Fam.	$k$	Min. yield	Edits	Graphlet count + LW	Size	Edge-label	Null	Size-changing		Pass
				m / sc	(mean)	mean [min–max]	only	pair + LW	edit	graphlet	size
1	P0	A	1	295 / 294	1.0	54.1 [53.1–54.6]	49.2	51.0	47.8	65.5	68.3	no
1	P0	A	2	295 / 294	1.8	70.2 [68.1–71.7]	49.2	50.9	47.8	73.9	68.3	no
1	P0	A	all	295 / 294	2.9	78.6 [76.0–80.3]	49.2	50.9	47.8	78.4	68.4	no
1	P0	B	1	106 / 106	1.0	47.5 [44.6–50.4]	49.9	56.6	55.3	63.4	68.9	no
1	P0	B	2	106 / 106	1.3	49.6 [47.1–52.5]	49.9	57.8	55.3	63.6	68.9	no

Appendix

Table 13: All 26 MS-v2 design candidates and the control scores on which the pre-written selection rule acted. AUROC in points, mean over five seeds, with the range over seeds for graphlet count + LW. Min. yield is the smallest number of anomalies in any seed for the size-preserving (m) and size-changing (sc) variants, Edits is the mean number of swaps per anomaly, and Null edit scores the unedited parents with graphlet count + LW. “No (yield)” marks candidates with fewer than 100 anomalies in some seed. Family A turns a six-ring into a five-ring, B a five-ring into a six-ring, C a six-ring into a four-ring and F a six-ring into a three-ring. D relocates a ring, and E turns a fused junction into a spiro junction.

Generator	Type	Size	Graphlet	Size	Edge-label	Null edit	Valid	In	Anomalies per seed
			count + LW	only	pair + LW	(graphlet)		decision	(seeds 0–4)
GX	T1a	preserving	100.0	51.0	51.2	–	yes	yes	250 each
GX	T3a	preserving	90.1	49.1	50.2	–	yes	yes	250 each
GX	T3b	preserving	91.2	51.0	50.7	–	yes	yes	250 each
MS-v2	MS1	preserving	95.1	48.8	52.3	46.2	yes	yes	237, 246, 262, 255, 252
MS-v2	MS1	changing	95.5	67.4	76.6	–	no	no	237, 245, 261, 255, 252

Appendix

Table 14: Control results that decide which typed anomalies are valid tests. AUROC in points, mean over five seeds. A type is valid if graphlet count + LW is at least 85 and size only is at most 60, and confounded if edge-label pair + LW is at least 85, which never happens. Null edit scores the unedited parent molecules with graphlet count + LW and shows that the choice of parents alone is not detected. The size-changing variants fail the size-only condition by construction and serve only the scope test. Each seed also has 327 test normals (MS-v2) or 250 (GX).

Confirmatory detectors (enter the decision)
	GX, size-preserving			MS-v2, size-preserving			MS-v2, size-changing
Detector	T1a	T3a	T3b	MS1	MS2	MS3	MS1	MS2	MS3
GLADC	51.1	50.5	54.6	53.4	56.0	52.6	85.7	87.0	85.7
CVTGAD	69.9	53.2	56.3	68.9	82.0	76.9	85.0	88.5	87.4
GOOD-D	92.7	57.0	60.7	69.3	75.8	69.9	85.3	86.6	84.8
GraphDE	49.9	48.6	50.0	51.7	44.5	50.0	48.0	43.5	46.1

Appendix

Table 15: Typed anomalies for every detector. Val-Normal AUROC in points, mean over five seeds. The size-preserving columns enter the decision for the eight confirmatory detectors, and the size-changing columns serve only the scope test. Bold values exceed 60 and therefore do not meet the blindness criterion. The rows of the nine recent methods are descriptive. The last rows are the training-free controls that decide validity.

Confirmatory detectors (enter the decision)
	GX, size-preserving			MS-v2, size-preserving			MS-v2, size-changing
Detector	T1a	T3a	T3b	MS1	MS2	MS3	MS1	MS2	MS3
GLADC	47.0–55.2	48.6–52.3	51.7–56.9	52.1–54.7	54.1–57.9	50.9–54.2	84.6–86.7	85.6–88.3	85.0–86.4
CVTGAD	66.6–73.5	49.4–56.7	52.7–59.4	66.9–70.9	80.5–83.6	75.8–78.0	83.8–86.3	87.7–89.0	87.3–87.6
GOOD-D	91.3–94.0	54.0–59.3	59.0–62.0	66.1–72.5	72.7–80.4	68.3–71.5	84.2–86.8	85.1–88.4	83.4–86.1
GraphDE	48.2–51.7	46.5–51.2	48.5–51.5	49.8–53.6	42.2–46.5	48.0–51.6	46.3–50.0	40.2–46.4	44.5–47.7

Appendix

Table 16: Seed-bootstrap 95% intervals for the means in Table 15 . With five seeds per cell these intervals are coarse and serve only to show which cells lie close to the blindness threshold of 60.

Group	Detectors	Pairs	GX and MS-v2	MS-v2	most MS-v2 types	Verdict
			Share of pairs blind		Detectors blind on
Confirmatory, GX features F2 (primary)	8	48	0.81	0.75	6	confirmed
Confirmatory, GX features F1 (sensitivity)	8	48	0.77	0.75	6	confirmed
Pilot (descriptive)	4	24	0.79	0.58	3	partial

Appendix

Table 17: Decision summary. A pair is one detector on one valid size-preserving type. Blind fractions are taken over the pairs on GX and MS-v2 together and over the MS-v2 pairs alone. The verdict applies the pre-specified rule to each group, but only the first row is the confirmatory decision.

Blindness threshold	Blind pairs (of 48)	Blind MS-v2 pairs (of 24)	Detectors blind on most MS-v2 types	All three conditions
50	27.1%	12.5%	1	not met
52.5	54.2%	37.5%	4	not met
55	66.7%	54.2%	6	not met
57.5	77.1%	66.7%	6	met
60 (fixed in advance)	81.2%	75.0%	6	met
62.5	83.3%	75.0%	6	met

Appendix

Table 18: The typed-anomaly decision under other blindness thresholds (post hoc). Shares of the eight confirmatory detectors’ size-preserving pairs whose mean Val-Normal AUROC is at or below the threshold. The pre-specified conditions are at least 75% of all pairs, at least 60% of MS-v2 pairs and at least five detectors blind on most MS-v2 types.

Detector	Fit	HP	AIDS	BZR	COX2	DHFR	ENZ.	NCI1	PROT.	IMDB-B	RDT-B	Mean
WL-IF	train	default	87.9	34.8	54.3	39.8	60.3	48.9	56.7	55.3	73.1	56.8
WL-IF	train	tuned	91.8	37.6	55.2	41.2	61.9	50.1	60.2	56.4	73.6	58.6
WL-IF	test	default	72.0	45.7	58.7	43.1	58.8	45.1	32.5	57.2	74.6	54.2
WL-IF	test	tuned	78.8	46.3	60.6	44.2	61.0	46.3	34.7	58.9	75.2	56.2
WL-OCSVM	train	default	4.2	51.1	47.0	32.7	57.6	66.3	19.0	47.0	52.9	42.0
WL-OCSVM	train	tuned	8.4	51.7	48.0	34.5	58.8	67.2	19.5	47.5	53.3	43.2

Appendix

Table 19: Kernel baselines on UB9. AUROC in points, mean over five folds, and the mean over the nine datasets. Train fit is the protocol of every other detector and the only one ranked. Test fit is UB-GOLD’s code path, with labels paired correctly. Default uses UB-GOLD’s code defaults, and tuned picks the configuration with the best mean test AUROC in UB-GOLD’s search space, an upper bound. § marks a constant kernel matrix, where every graph gets the same score and AUROC is 50 by construction.

Detector	Fit	HP	DD	COLLAB	p53	MMP	Mean
WL-IF	train	default	63.2	54.3	51.8	48.6	54.5
WL-IF	train	tuned	65.0	57.1	55.1	53.3	57.6
WL-IF	test	default	31.6	49.5	45.9	44.0	42.8
WL-IF	test	tuned	32.9	52.2	46.8	44.6	44.1
WL-OCSVM	train	default	18.2	35.3	49.7	47.8	37.8
WL-OCSVM	train	tuned	18.6	35.8	54.9	55.5	41.2

Appendix

Table 20: Kernel baselines on the secondary datasets, as in Table 19 . MUTAG was not run.

Dataset	Folds	Released pairing	Aligned pairing	Labels out of place
AIDS	5	50.63	71.50	0.33
BZR	5	46.31	45.68	0.33
COX2	5	54.64	53.75	0.36
DHFR	5	49.69	43.64	0.43
ENZ.	5	53.15	58.19	0.26
NCI1	5	50.41	45.46	0.50

Appendix

Table 21: UB-GOLD’s unchanged WL-IF pipeline. AUROC in points of the same scores, paired with labels as in the released code and as they should be, mean over five folds. The last column is the mean fraction of test positions whose label differs between the two passes over the shuffled test loader. The last row averages all 65 folds.

Dataset	Chance s.d.	WL-IF	WL-OCSVM	PK-IF	PK-OCSVM
AIDS	1.62	61.42 (7.1)	52.56 (1.6)	52.01 (1.2)	50.93 (0.6)
BZR	3.53	51.71 (0.5)	51.16 (0.3)	59.08 (2.6)	46.67 ( $-$ 0.9)
COX2	3.25	49.56 ( $-$ 0.1)	53.34 (1.0)	52.48 (0.8)	52.15 (0.7)
DHFR	2.16	51.64 (0.8)	50.30 (0.1)	52.79 (1.3)	48.31 ( $-$ 0.8)
ENZ.	3.18	51.03 (0.3)	53.75 (1.2)	49.82 ( $-$ 0.1)	52.45 (0.8)
NCI1	0.90	50.41 (0.5)	54.18 (4.6)	50.22 (0.2)	51.39 (1.5)

Appendix

Table 22: UB-GOLD’s published kernel AUROCs, in points, with the z-score against random pairing in parentheses. The chance standard deviation is that of a five-fold mean under random pairing, computed from the fold sizes. Bold marks values more than three standard deviations from 50.

		Oracle	Oracle	Last	Val-Normal	Val-Normal	HP-Free
Detector	Group	own rule	UB-GOLD rule			95% CI
SIGNET	UB-GOLD	74.7 (2)	74.7 (2)	70.7 (2)	71.6 (1)	[65.4, 78.4]	69.3 (1)
CVTGAD	UB-GOLD	76.1 (1)	76.1 (1)	71.0 (1)	71.1 (2)	[64.5, 78.3]	66.5 (4)
OCGTL	UB-GOLD	72.1 (7)	72.1 (6)	68.7 (4)	68.8 (3)	[59.4, 79.6]	60.1 (17)
GOOD-D	UB-GOLD	72.3 (6)	72.3 (5)	67.7 (6)	67.6 (4)	[60.1, 76.7]	66.8 (2)
OCGIN	UB-GOLD	70.2 (8)	70.2 (7)	64.5 (7)	64.8 (6)	[55.7, 75.6]	63.9 (10)

Appendix

Table 23: UB9 leaderboard with all 30 rows. Mean AUROC over nine datasets and five folds, with the rank among the 30 rows in parentheses. The 95% interval of Val-Normal is a dataset bootstrap. HP-Free uses repository defaults selected by Val-Normal . For kernel baselines, the first four columns use test-selected hyperparameters.

Detector	AIDS	BZR	COX2	DHFR	ENZ.	NCI1	PROT.	IMDB-B	RDT-B	Mean
SIGNET	89.3	81.4	70.0	68.5	62.7	69.9	73.3	70.8	86.7	74.7
CVTGAD	99.2	78.2	65.1	65.9	68.5	74.6	74.6	69.9	88.7	76.1
OCGTL	99.5	63.3	55.8	59.4	65.6	77.6	74.2	63.5	89.5	72.1
GOOD-D	96.5	66.2	62.3	64.2	65.1	63.3	76.2	66.2	90.4	72.3
OCGIN	99.8	68.2	57.8	59.6	64.0	52.0	76.1	60.9	93.0	70.2
InfoGraph-OCSVM	99.7	62.7	57.8	60.3	64.5	44.0	75.0	63.6	81.4	67.7

Appendix

Table 24: UB9 per dataset under Oracle with each method’s own rule. AUROC in points, mean over five folds. Mean is the UB9 mean of Table 23 . § marks a constant kernel matrix (AUROC 50 by construction).

Detector	AIDS	BZR	COX2	DHFR	ENZ.	NCI1	PROT.	IMDB-B	RDT-B	Mean
SIGNET	91.7	72.2	69.7	70.3	61.8	69.9	63.6	57.5	79.9	70.7
CVTGAD	91.0	72.3	55.3	59.3	66.1	74.0	65.0	70.4	85.2	71.0
OCGTL	99.3	54.1	54.3	51.8	63.4	76.8	69.4	60.5	88.4	68.7
GOOD-D	91.6	66.3	54.4	59.1	58.7	58.1	67.9	64.2	89.3	67.7
OCGIN	92.7	64.8	50.1	52.4	62.8	46.8	63.6	56.2	90.7	64.5
InfoGraph-OCSVM	86.7	62.0	55.8	48.7	63.3	43.4	68.8	43.7	79.9	61.3

Appendix

Table 25: UB9 per dataset under Last , as in Table 24 .

Detector	AIDS	BZR	COX2	DHFR	ENZ.	NCI1	PROT.	IMDB-B	RDT-B	Mean
SIGNET	91.9	77.6	70.3	72.3	60.4	70.1	64.1	57.4	80.2	71.6
CVTGAD	91.0	72.9	55.6	60.0	66.4	74.0	64.3	69.9	85.3	71.1
OCGTL	99.3	54.1	54.3	51.5	63.4	76.9	69.3	62.2	88.6	68.8
GOOD-D	91.7	66.3	54.4	59.6	59.2	57.8	66.1	63.9	89.6	67.6
OCGIN	92.7	64.8	50.1	52.5	62.7	49.2	63.7	56.4	91.2	64.8
InfoGraph-OCSVM	99.6	62.0	56.7	60.0	60.0	33.8	74.2	56.9	79.3	64.7

Appendix

Table 26: UB9 per dataset under Val-Normal , as in Table 24 .

Detector	AIDS	BZR	COX2	DHFR	ENZ.	NCI1	PROT.	IMDB-B	RDT-B	Mean
SIGNET	92.7	72.5	63.9	62.1	52.5	73.5	67.1	58.6	80.6	69.3
CVTGAD	93.8	66.6	58.6	56.2	56.3	51.4	70.5	60.2	85.1	66.5
OCGTL	99.3	48.0	46.1	44.9	50.6	63.1	50.4	55.3	83.2	60.1
GOOD-D	92.4	61.7	54.7	59.5	60.5	55.3	67.2	64.5	85.7	66.8
OCGIN	93.6	59.7	50.8	49.5	56.5	48.1	71.3	56.0	89.2	63.9
InfoGraph-OCSVM	99.6	62.0	56.7	55.6	60.2	40.2	74.1	49.9	79.1	64.1

Appendix

Table 27: UB9 per dataset under HP-Free (repository defaults, selected by Val-Normal ), as in Table 24 .

	Oracle (own rule)					Val-Normal
Detector	DD	COLLAB	p53	MMP	MUTAG	DD	COLLAB	p53	MMP	MUTAG
SIGNET	72.0	71.6	67.4	76.3	88.2	69.5	67.0	55.6	64.7	58.6
CVTGAD	75.1	66.1	68.9	69.7	–	67.3	60.4	64.2	68.6	–
OCGTL	80.5	51.6	67.1	57.5	86.5	73.8	47.9	62.7	59.4	79.9
GOOD-D	80.0	64.0	68.6	71.5	–	73.9	57.4	66.2	67.6	–
OCGIN	80.5	60.3	70.2	68.3	65.5	77.9	56.9	59.0	65.9	66.1

Appendix

Table 28: Secondary datasets under Oracle (own rule) and Val-Normal . AUROC in points, mean over five folds. “–” marks a dataset that was not run for that row. ‡ marks GLADC on COLLAB, where three of five folds diverged and the value is a two-fold mean. § marks a constant kernel matrix.

	Last					HP-Free
Detector	DD	COLLAB	p53	MMP	MUTAG	DD	COLLAB	p53	MMP	MUTAG
SIGNET	69.7	66.4	54.8	66.5	57.7	62.0	59.6	61.2	66.5	–
CVTGAD	67.9	59.7	64.9	68.8	–	71.1	62.0	64.2	68.5	–
OCGTL	74.2	46.3	49.7	59.4	79.7	67.6	50.3	51.9	56.0	–
GOOD-D	74.0	56.7	65.6	68.3	–	66.9	57.9	66.8	69.0	–
OCGIN	77.9	55.9	64.8	67.3	66.1	76.4	51.9	59.8	57.0	–

Appendix

Table 29: Secondary datasets under Last and HP-Free , as in Table 28 . The four UB-GOLD detectors with MUTAG runs have no HP-Free run on MUTAG.

Figure 4: Every checkpointed detector scores higher when the checkpoint is chosen with test labels. Checkpoint leakage (inflation) per detector on UB9, Oracle with UB-GOLD’s rule minus Val-Normal (left) or Last (right), all with the tuned hyperparameters, with 95% dataset-bootstrap intervals. The dotted line marks the 2-point threshold fixed in advance. Circles are UB-GOLD detectors and squares recent methods.

	Oracle $-$ Val-Normal		Oracle $-$ Last		Own rule
Detector	Mean [95% CI]	$\geq 2$	Mean [95% CI]	$\geq 2$	$-$ Val-Normal
SIGNET	+3.15 [ $-$ 0.19, 6.70]	5/9	+4.01 [0.52, 7.52]	4/9	–
CVTGAD	+5.02 [2.79, 7.41]	7/9	+5.12 [2.85, 7.46]	7/9	–
OCGTL	+3.21 [1.29, 5.34]	4/9	+3.41 [1.56, 5.41]	5/9	–
GOOD-D	+4.65 [2.69, 6.74]	7/9	+4.55 [2.74, 6.38]	6/9	–
OCGIN	+5.34 [3.25, 7.54]	7/9	+5.69 [3.66, 7.86]	8/9	–

Appendix

Table 30: Inflation per detector on UB9, in AUROC points, with dataset-bootstrap 95% intervals. “

\geq 2

” counts the datasets (or, in pooled rows, the detectors) with at least 2 points of inflation. All rows use UB-GOLD’s selection rule, and the last column gives Oracle

-

Val-Normal under each recent method’s own rule.

Figure 5: The gain from test-tuned hyperparameters is concentrated in a few detectors and is negative for others. Each bar is one detector’s Val-Normal AUROC on UB9 with UB-GOLD’s or the released per-dataset hyperparameters minus its Val-Normal AUROC with repository defaults, in points. Lines are 95% dataset-bootstrap intervals over the nine UB9 datasets. GLASS and SUL/GUL release a single configuration and are not shown.

Selection rule	SIGNET	OCGTL	OCGIN	GLocalKD ∗	Mean of 4	$-$ Val-Normal
Oracle (test labels, early stop)	74.6	72.1	70.4	64.3	70.4	+4.26
Oracle without early stop	76.8	72.9	70.6	64.3	71.1	+5.04
Val-Normal (reference)	71.3	69.1	65.1	59.0	66.1	0
Last	70.3	68.8	64.7	59.0	65.7	$-$ 0.39
Pseudo-anomaly AUROC	70.2	68.4	65.6	59.9	66.0	$-$ 0.10
Pseudo-anomaly top-3 ensemble	70.2	68.4	65.8	60.1	66.1	$-$ 0.00

Appendix

Table 31: Label-free checkpoint selectors on the four pilot detectors. Mean UB9 AUROC in points. The last column is the mean over the four detectors minus that of Val-Normal . ∗ GLocalKD has no REDDIT-B runs in this experiment and is averaged over eight datasets. These runs predate the main grid but use the same folds and UB-GOLD hyperparameters, so their values differ slightly from Table 23 .

Rows ranked	Protocols compared	$\tau_{b}$	Fold-bootstrap CI	Dataset-bootstrap CI	Top 3 changed
UB-GOLD (12)	Oracle vs Val-Normal	0.76	[0.61, 0.85]	[0.36, 0.85]	yes
	Oracle vs Last	0.70	[0.58, 0.79]	[0.42, 0.85]	yes
	Val-Normal vs Last	0.70	[0.58, 0.85]	[0.42, 0.97]	no
	Oracle ${}_{\text{UB-GOLD}}$ vs Val-Normal	0.76	[0.61, 0.85]	[0.36, 0.85]	yes
	Oracle vs HP-Free	0.45	[0.33, 0.58]	[0.27, 0.67]	no
	Val-Normal vs HP-Free	0.52	[0.39, 0.70]	[0.39, 0.76]	yes

Appendix

Table 32: Kendall’s

\tau_{b}

between UB9 rankings under two protocols, with fold- and dataset-bootstrap 95% intervals. “Top 3 changed” says whether the set of the three best rows differs. Oracle uses each method’s own rule, Oracle

{}_{\text{UB-GOLD}}

the UB-GOLD rule, and Oracle

{}_{\text{defaults}}

the UB-GOLD rule with repository defaults.

		AUPRC, higher is better				FPR95, lower is better
Detector	Group	Oracle	Last	Val-Normal	HP-Free	Oracle	Last	Val-Normal	HP-Free
SIGNET	UB-GOLD	70.4	67.2	67.9	66.3	71.2	71.2	71.2	74.8
CVTGAD	UB-GOLD	74.4	67.3	67.3	64.6	62.7	72.9	71.8	74.6
OCGTL	UB-GOLD	71.4	69.3	69.4	65.0	65.8	70.6	70.2	79.8
GOOD-D	UB-GOLD	70.9	65.2	65.5	64.5	70.4	71.5	72.5	73.3
OCGIN	UB-GOLD	70.1	65.6	65.9	65.1	66.9	79.6	79.3	76.0

Appendix

Table 33: Secondary metrics on UB9, mean over nine datasets and five folds, in points. Oracle uses UB-GOLD’s rule for every method, and HP-Free is Val-Normal with repository defaults. Kernel baselines and training-free references have no checkpoints, so their first three columns of each metric coincide.

Detectors	Protocol	$\chi^{2}_{F}$	$p$	Iman–Davenport $p$	Nemenyi CD	Holm-significant pairs
12 UB-GOLD	Oracle	34.54	$3.0\times 10^{-4}$	$4.2\times 10^{-5}$	5.55	0 of 66
12 UB-GOLD	Last	26.57	0.0053	0.0024	5.55	0 of 66
12 UB-GOLD	Val-Normal	21.65	0.027	0.019	5.55	0 of 66
12 UB-GOLD	HP-Free	18.64	0.068	0.056	5.55	0 of 66
21	Oracle	69.09	$2.6\times 10^{-7}$	$2.1\times 10^{-9}$	10.44	0 of 210
21	Last	75.23	$2.5\times 10^{-8}$	$4.6\times 10^{-11}$	10.44	0 of 210

Appendix

Table 34: Friedman tests on UB9 per protocol, with the Nemenyi critical difference in units of average rank and the number of Holm-significant pairs among all pairs of detectors. Oracle uses UB-GOLD’s rule for every method.

		Non-finite at	Tie-	Diverged
Detector	Runs	selected point	collapsed	(excluded)
OCGTL	135	0	5	0
GLADC	119	0	0	6
GraphCL-OCSVM	130	0	3	0
GraphDE	130	0	13	0
HC-GLAD	100	2	2	0

Appendix

Table 35: Detectors with flagged or diverged runs. Runs count every dataset, fold and hyperparameter setting. Tie flags need saved per-graph scores, which are missing for the kernel baselines, the training-free references and part of the runs of SIGNET, OCGTL, OCGIN and GLocalKD, so a zero there does not show the absence of ties.

Quantity (12 UB-GOLD detectors)	All runs	Drop non-finite	Drop tie-collapsed	Drop either
		(12 runs)	(71 runs)	(71 runs)
Inflation, Oracle $-$ Val-Normal	+4.13 [3.32, 4.79]	+4.13 [3.32, 4.79]	+4.13 [3.32, 4.79]	+4.13 [3.32, 4.79]
detectors with $\geq 2$ points	12/12	12/12	12/12	12/12
Inflation, Oracle $-$ Last	+5.12 [3.97, 6.24]	+5.12 [3.97, 6.24]	+5.11 [3.97, 6.23]	+5.11 [3.97, 6.23]
detectors with $\geq 2$ points	12/12	12/12	12/12	12/12
Hyperparameter gap	+1.53 [0.64, 2.42]	+1.53 [0.64, 2.42]	+1.54 [0.65, 2.43]	+1.54 [0.65, 2.43]

Appendix

Table 36: Pre-specified quantities for the 12 UB-GOLD detectors with all runs and with flagged runs dropped. Mean over detectors in AUROC points, with dataset-bootstrap 95% intervals.

Method	Pairs	Bit-identical	Differ	Median	90th pct.	Max	Median	90th pct.	Max
				$\|\Delta\|$ at Last			$\|\Delta\|$ at Val-Normal
GLASS	50	18	32	0.45	2.45	5.20	0.00	4.04	8.33
SUL/GUL	50	50	0	–	–	–	–	–	–

Appendix

Table 37: Pairs of runs with identical configuration, seed and split.

|\Delta|

is the absolute AUROC difference in points between the two runs of a pair, over the pairs that differ.

Detector	Quantity	c940457	a96137b	Paired difference	95% CI (dataset bootstrap)
GLocalKD	Oracle	66.36	66.41	+0.05	[ $-$ 0.01, 0.14]
GLocalKD	Last	62.51	62.29	$-$ 0.22	[ $-$ 0.62, 0.10]
GLocalKD	Val-Normal	62.52	62.30	$-$ 0.22	[ $-$ 0.62, 0.10]
GLocalKD	Oracle $-$ Val-Normal	3.84	4.11	+0.27	–
GLocalKD	Oracle $-$ Last	3.85	4.12	+0.27	–
GLADC	Oracle	68.22	68.11	$-$ 0.12	[ $-$ 0.26, 0.01]

Appendix

Table 38: GLocalKD and GLADC at the commit before the pull request (a96137b) and at the commit used throughout (c940457), UB9 means in AUROC points and paired differences over 45 folds per detector. The last two rows per detector give the checkpoint leakage.

	UB-GOLD’s anomaly class (class 0)					Flipped (class 1)
Detector	Oracle	Val-Normal	Last	$\Delta_{\mathrm{VN}}$	$\Delta_{\mathrm{L}}$	Oracle	Val-Normal	Last	$\Delta_{\mathrm{VN}}$	$\Delta_{\mathrm{L}}$
SIGNET	74.90	70.65	68.50	4.25	6.40	68.08	64.34	64.96	3.74	3.12
OCGTL	64.47	59.21	59.24	5.26	5.24	76.53	69.57	69.88	6.96	6.65
OCGIN	67.35	59.55	59.51	7.80	7.84	63.43	62.89	62.93	0.54	0.50
GLocalKD	61.49	57.61	57.57	3.88	3.92	52.27	51.51	51.51	0.76	0.76
GLADC	67.53	61.34	61.52	6.19	6.00	46.42	43.03	42.06	3.39	4.36

Appendix

Table 39: Checkpoint leakage with UB-GOLD’s anomaly class and with the class roles swapped, on BZR, COX2 and PROTEINS_full. Means over 15 (dataset, fold) cells in AUROC points.

\Delta_{\mathrm{VN}}

is Oracle

-

Val-Normal and

\Delta_{\mathrm{L}}

is Oracle

-

Last .

	UB-GOLD’s anomaly class			Flipped
Reference	BZR	COX2	PROTEINS_full	BZR	COX2	PROTEINS_full
Size only	42.5	50.9	57.8	63.9	50.2	65.5
Graphlet count + LW	44.8	57.5	55.0	67.6	43.6	65.8
GRAIL-0	54.4	58.6	56.3	64.3	51.2	64.5

Appendix

Table 40: Training-free references in both orientations, AUROC in points, mean over five folds.

		GX, size kept			MS-v2, size kept			MS-v2, size changed
Detector	Input	T1a	T3a	T3b	MS1	MS2	MS3	MS1	MS2	MS3
CVTGAD	encodings	69.9	53.2	56.3	68.9	82.0	76.9	85.0	88.5	87.4
	$x_{s}=1$	56.6	55.9	52.1	48.7	59.2	49.8	78.8	83.4	79.5
GOOD-D	encodings	92.7	57.0	60.7	69.3	75.8	69.9	85.3	86.6	84.8
	$x_{s}=1$	50.4	55.1	50.3	52.7	54.7	49.1	82.1	84.0	81.0

Appendix

Table 41: Typed anomalies with and without the precomputed structural encodings. Val-Normal AUROC, mean over 5 seeds. Bold scores exceed 60 and therefore do not meet the blindness criterion.

Detector	Quantity	Unablated	$x_{s}=1$	Paired difference	95% CI (dataset bootstrap)
CVTGAD	Oracle	76.08	69.22	$-$ 6.86	[ $-$ 12.27, $-$ 2.41]
CVTGAD	Last	70.96	65.67	$-$ 5.29	[ $-$ 13.11, 1.12]
CVTGAD	Val-Normal	71.05	65.62	$-$ 5.43	[ $-$ 13.09, 1.03]
CVTGAD	Oracle $-$ Val-Normal	5.02	3.60	$-$ 1.43	–
CVTGAD	Oracle $-$ Last	5.12	3.55	$-$ 1.57	–
GOOD-D	Oracle	72.27	70.58	$-$ 1.68	[ $-$ 4.28, 0.66]

Appendix

Table 42: UB9 with and without the precomputed structural encodings, means over 45 (dataset, fold) cells in AUROC points. The last two rows per detector give the checkpoint leakage.

Analysis	Question	Quantity	Detectors
Selector robustness ( G.2 )	Is the gap to Oracle specific to Val-Normal or Last ?	Oracle minus each of seven label-free selectors	12 UB-GOLD, 8 recent
Selection on noise ( G.3 )	How large a gap does picking the best test evaluation produce when no checkpoint is better?	Permutation null of the maximum minus Last , and correlations across runs	8 UB-GOLD, 8 recent (correlations 12 UB-GOLD, 8 recent)
Dataset robustness ( G.4 )	Does one dataset drive the inflation, and what is the inferential target?	Leave-one-dataset-out means, per-dataset inflation, crossed bootstrap	12 UB-GOLD
Size matching ( G.5 )	How much benchmark AUROC remains when anomalies and normals have similar size?	AUROC on size-matched test subsets	8 UB-GOLD, 9 recent
Reproduction fidelity ( G.6 )	Do pooled recent-method results rest on uncertain ports?	Pooled inflation without those methods	8 recent

Appendix

Table 43: The five post hoc analyses specified in A32. All use stored outputs and no new training. “Recent” are the eight recent methods with checkpoints.

Detectors	Per-graph scores	S1, S2, S4–S6	S3	S7	Permutation null	Size matching
SIGNET, OCGTL, OCGIN, GLocalKD	no	45	–	–	–	–
GLADC, GOOD-D, GraphDE, GraphCL-IF, GraphCL-OCSVM, InfoGraph-IF, InfoGraph-OCSVM	yes	45	45	–	45	43
CVTGAD	yes	45	45	45	45	43
MUSE, LGKDE, GLADPro, HC-GLAD, SUL/GUL	yes	45	45	–	45	43
GLADMamba, SEGO, GLASS	yes	45	45	45	45	43
ProtoGLAD (no checkpoints)	yes	–	–	–	–	43

Appendix

Table 44: Runs used by each analysis, on UB9 with UB-GOLD’s or the released hyperparameters (45 runs per detector). S1 to S7 are the selectors of Appendix G.2 . Size matching uses 43 of 45 runs because two AIDS folds have no matched pair. “–” means not available.

		UB-GOLD detectors			Recent methods
	Selector	$n$	Oracle $-$ selector	Same 8 as S3	$n$	Oracle $-$ selector
S1	Last	12	5.12 [3.97, 6.24]	5.56 [4.22, 6.90]	8	6.69 [5.75, 7.61]
S2	Val-Normal (lowest mean held-out score)	12	4.13 [3.32, 4.79]	4.25 [3.31, 5.05]	8	7.47 [6.15, 8.83]
S3	Lowest median held-out score	8	4.15 [3.22, 4.98]	4.15 [3.22, 4.98]	8	7.42 [6.30, 8.65]
S4	Lowest mean training loss	12	5.10 [3.88, 6.25]	5.58 [4.20, 6.97]	8	6.75 [5.70, 7.71]
S5	Point closest to half the epoch budget	12	5.21 [4.00, 6.30]	5.53 [4.20, 6.77]	8	7.67 [6.06, 9.33]

Appendix

Table 45: Gap between Oracle and seven label-free selectors on UB9, pooled over detectors, in AUROC points with 95% dataset-bootstrap intervals.

n

is the number of detectors with complete coverage. The middle column restricts the UB-GOLD pool to the eight detectors that S3 covers, so that all selectors are compared on the same detectors. For GLADMamba, SEGO and GLASS, S7 equals S2.

Detector	S1	S2	S3	S4	S5	S6	S7
SIGNET	4.01	3.15 †	–	4.22	5.11	6.33	–
OCGTL	3.41	3.21	–	3.13	4.65	5.35	–
OCGIN	5.69	5.34	–	5.33	5.10	5.10	–
GLocalKD	3.85	3.84	–	3.86	3.44	3.40	–
GLADC	5.07	4.82	4.68	4.99	4.47	4.16	–
CVTGAD	5.12	5.02	5.28	5.02	6.17	6.56	10.81

Appendix

Table 46: Gap between Oracle and each label-free selector for single detectors on UB9, in AUROC points (mean over nine datasets of five-fold means). † marks a 95% dataset-bootstrap interval that includes zero. No interval lies entirely below zero. “–” means that the selector is not available for the detector.

Figure 6: The gap to Oracle stays positive under every pre-specified label-free selector. Large markers are the pooled gap Oracle

-

selector on UB9 in AUROC points, with 95% dataset-bootstrap intervals, for the UB-GOLD detectors (circles) and the recent methods (squares). Small markers are single detectors. Open diamonds are the permutation null of Appendix G.3 for the eight detectors of each group with per-graph scores, shown as the maximum minus Last at S1 and the maximum minus the trajectory mean at S6. The row below the axis gives the number of detectors in each group. S7 covers one UB-GOLD detector and three recent methods.

		Against Last			Against the trajectory mean (S6)
Detector	Points	Observed	Null	Ratio	Observed	Null	Ratio
GLADC	25.0	5.07	2.28	0.45	4.16	2.27	0.55
CVTGAD	147.2	5.12	5.47	1.07	6.56	5.47	0.83
GOOD-D	91.1	4.55	5.35	1.18	6.23	5.36	0.86
GraphDE	63.1	3.38	2.15	0.64	3.01	2.14	0.71
GraphCL-IF	8.0	3.99	2.53	0.63	3.93	2.55	0.65

Appendix

Table 47: Permutation null of checkpoint selection on UB9, in AUROC points. “Observed” is the gap of Oracle under UB-GOLD’s rule. “Null” is the mean over 200 label permutations of the maximum over all evaluation points minus the AUROC at Last , or minus the trajectory mean, aggregated like the observed gap. “Ratio” is null divided by observed. “Points” is the mean number of evaluation points per run. Pooled rows give 95% dataset-bootstrap intervals for the null. The four pilot detectors have no per-graph scores.

Detectors	Property of the run	Runs	Detectors	$\rho$	$p$
UB-GOLD	Number of evaluation points	540	12	$-0.05$	0.29
UB-GOLD	Spread of test AUROC	540	12	0.42	$<0.001$
Recent	Number of evaluation points	360	8	0.17	$<0.001$
Recent	Spread of test AUROC	360	8	0.28	$<0.001$
Both	Number of evaluation points	900	20	0.06	0.09
Both	Spread of test AUROC	900	20	0.36	$<0.001$

Appendix

Table 48: Spearman correlation across runs between the observed gap Oracle

-

Last of a run and two properties of its trajectory on UB9. The spread is the standard deviation of test AUROC over the evaluation points of the run. These correlations are descriptive.

	On this dataset				Without this dataset
Dataset	Oracle $-$ Val-Normal	$>0$	Oracle $-$ Last	$>0$	$-$ Val-Normal	$-$ Last
AIDS	2.50 [1.96, 3.08]	11	5.24 [4.25, 6.23]	11	4.33	5.10
BZR	4.40 [3.01, 5.85]	11	5.10 [3.29, 7.05]	11	4.10	5.12
COX2	4.61 [3.41, 5.69]	11	5.30 [4.23, 6.36]	12	4.07	5.10
DHFR	5.00 [4.58, 5.71]	11	7.30 [6.64, 7.84]	11	4.02	4.85
ENZYMES	4.44 [3.17, 6.03]	12	3.79 [2.90, 4.60]	12	4.09	5.29

Appendix

Table 49: Inflation of the 12 UB-GOLD detectors per UB9 dataset and after leaving out each dataset, in AUROC points. “On this dataset” gives the mean over the 12 detectors with a 95% fold-bootstrap interval and the number of detectors with a positive gap. “Without this dataset” gives the pooled mean over the other eight datasets. The last row gives the pooled means over all nine datasets with their dataset-bootstrap and crossed-bootstrap intervals.

Figure 7: The inflation is positive on every UB9 dataset and does not depend on any single dataset. (a) Mean over the 12 UB-GOLD detectors of Oracle

-

Val-Normal (circles) and Oracle

-

Last (triangles) on each dataset, in AUROC points, with 95% fold-bootstrap intervals that keep the detectors fixed. Tick marks give the smallest and largest single-detector value. (b) The two pooled means over all nine datasets, with dataset-bootstrap (thick) and crossed detector-by-dataset bootstrap (thin) 95% intervals, and the means after leaving out each dataset. The dashed line marks the pre-specified threshold of 3.0 points.

Dataset	Anomalies	Pairs	Min	Mean	Reported	Full	Matched
			Matched share (%)			Size only
AIDS	400	5	0	1	no	98.7	50.0
BZR	319	64	16	20	no	42.5	42.3
COX2	365	88	22	24	no	50.9	50.0
DHFR	295	259	80	88	yes	41.9	47.4
ENZYMES	100	89	80	89	yes	43.7	51.1

Appendix

Table 50: Size matching per UB9 dataset. Anomalies and matched pairs are summed over five test folds. The matched share is the percentage of anomalies with a partner, as the minimum and the mean over folds. A dataset is reported if the minimum is at least 30%. The size-only reference gives AUROC in points on the full test sets and on the matched subsets, as a mean over folds with matched pairs.

	Val-Normal		Oracle		Oracle $-$ Val-Normal
Detector	Full	Matched	Full	Matched	Full	Matched
GLADC	59.9	50.5	65.0	54.6	5.1	4.2
CVTGAD	70.0	62.1	73.7	63.9	3.7	1.8
GOOD-D	66.0	59.2	70.9	61.4	4.9	2.2
GraphDE	52.2	49.7	53.4	50.2	1.2	0.6
GraphCL-IF	62.2	53.6	65.0	54.9	2.9	1.4

Appendix

Table 51: AUROC on the full test sets and on the size-matched subsets, mean over the six reported UB9 datasets, in points. Each detector is scored at the checkpoints that Val-Normal and Oracle (UB-GOLD’s rule) chose on the full test set. ProtoGLAD has no checkpoints, so both columns coincide and it is not pooled. The four pilot detectors have no per-graph scores.

Pool	Dropped	Methods	Oracle $-$ Val-Normal	Oracle $-$ Last
All recent methods with checkpoints	none	8	7.47 [6.15, 8.83]	6.69 [5.75, 7.61]
Without “reproduction not met”	LGKDE	7	8.16 [6.71, 9.58]	7.32 [5.73, 8.93]
Also without contested or not evaluable checks	LGKDE, GLADPro, SUL/GUL	5	6.38 [5.39, 7.43]	5.05 [3.65, 6.69]

Appendix

Table 52: Pooled inflation of the recent methods with checkpoints on UB9, with and without methods whose reproduction is uncertain. AUROC points with 95% dataset-bootstrap intervals. Oracle uses UB-GOLD’s rule. The first row equals the recent-method pool of the main text.

Group	Device	Recorded cost	Kind
12 UB-GOLD detectors, main grid	CPU, some GPU	1601 core-hours on CPU nodes, 1263 core-hours on GPU nodes used for their memory, 4.5 GPU-run-hours, and 1121 core-hours for the default setting on the secondary datasets	est.
Four-detector pilot	GPU	Not recorded
MUSE	CPU	14.6 core-hours	meas.
LGKDE	CPU and GPU	About 20 core-hours for eight datasets. NCI1 and REDDIT-B on GPU, about 0.7 GPU-hours per REDDIT-B run alone. The whole grid ran in four short jobs on one GPU node.	est., meas.
GLADPro	CPU	About 495 core-hours, plus about 20 for diagnostics on its own splits	est.
GLADMamba	CPU	About 307 core-hours for 55 runs and 10 diagnostics. The other 45 runs took one short job.	est.

Appendix

Table 53: Compute per group of runs. “Est.” marks estimates from timing runs made before the grid, and “meas.” marks values measured on finished runs. GPU-run-hours count the hours of each run on a GPU that up to three other runs could share.

Item	Description
Purpose	To let others evaluate graph-level anomaly detectors on UB-GOLD’s folds without test labels, with the same folds, held-out normals, protocols and statistics as this paper.
Contents	(i) Fold index lists for every dataset in the study, matched graph by graph to UB-GOLD’s loader, with the held-out normal graphs of each fold. (ii) The typed-anomaly suite, made of the synthetic GX generator and the MS-v2 generator with size-preserving and size-changing variants and five seeds, frozen by sha256 and shipped with its control scores. The retired first molecular generator and its control scores are included as a record. (iii) The instrumented harness for the 12 UB-GOLD detectors and the adapters for the nine recent methods. (iv) One record per run with seeds, device, library versions, configuration, code hash and pinned third-party commits, the metrics at every evaluation point, and the per-graph scores, except for the 180 UB9 pilot runs, which have trajectories only. (v) The frozen statistics code and the leaderboard tables, with one column per protocol and published numbers marked as upper bounds. The provenance record of Appendix A is available on request.
Source data	Public TU graph datasets ( Morris et al., 2020 ) and Tox21 tasks ( Huang et al., 2016 ) , loaded with UB-GOLD’s code. MS-v2 edits real NCI1 molecules of the normal class. No new raw data were collected.
Labels and features	UB-GOLD’s node features and anomaly class are kept for comparability, including the cases where the anomaly class is the majority class (Section 5.1 ). Variants with the minority class as the anomaly and with corrected Tox21 HSE and PPAR-gamma tasks will be added in a later release.
People	IMDB-B, REDDIT-B and COLLAB describe social or collaboration networks. They contain graph structure only, without node attributes, and are already public. No data about people were added.
Intended use	Label-free model selection and evaluation of graph-level anomaly detectors, and tests of whether a detector responds to specific structural changes. Oracle columns are diagnostic upper bounds only.

Appendix

Table 54: Datasheet for the released resources.

Graph-Based Anomaly Detection

See all 34 papers

The Selection Rule Decides the Winner: A Pre-Registered Audit of Open-Set Graph Anomaly Detection
Sep 27, 2026Farhan Shahriyar Hossain, Taufikur Rahman Fuad, Md Abrar Jahin +1Few-Shot Anomaly Detection Benchmark Auditing
Islamic University of Technology · University of Southern California · Qatar Computing Research Institute (QCRI)
Feature-Context Consistency: Unsupervised Adversarial Detection with Drift Stability on Attributed Graphs
Oct 12, 2026Firas Bayram, Maxime CordyGraph-Based Anomaly Detection Anomaly Detection
SnT, University of Luxembourg Luxembourg
GAD in the Wild: Benchmarking Graph Anomaly Detection under Realistic Deployment Challenges
May 8, 2026Jingjing Zhou, Shiyu Huang, Qing Qing +7Graph Neural Networks Graph Representation Learning
Zhejiang Gongshang University · Jilin University · Adelaide University +2

Sep 27, 2026J/K move · Enter open · S save

The Selection Rule Decides the Winner: A Pre-Registered Audit of Open-Set Graph Anomaly Detection

Farhan Shahriyar Hossain, Taufikur Rahman Fuad, Md Abrar Jahin, Md Rizwan Parvez

Islamic University of Technology · University of Southern California · Qatar Computing Research Institute (QCRI)

Few-Shot Anomaly Detection Benchmark Auditing Anomaly Detection Graph-Based Anomaly Detection

Open paper AskPDF arXiv

Open-set graph anomaly detection trains on a few labeled anomalies from one class and must also find anomaly classes that were never labeled. Published results share three conventions: the test score is read at the best epoch on the test set, baseline numbers are copied from earlier papers, and most anomalies are minority classes relabeled as anomalous. We ask how much of the reported ranking these conventions decide. We re-run two recent methods, DEMO and NSReg, together with OUTPOST, a small first-order detector built for this study. All three use one protocol with identical seeds and splits on eight graphs (seven for the baselines, which cannot run on ogbn-mag), ten seeds each, and every run is scored under both the best-epoch rule and a deployable validation rule. Before the runs that test them, we registered 40 predictions. Three findings hold. First, the rule changes the leader: under the best-epoch rule, OUTPOST and NSReg each lead three of seven graphs, while under the validation rule, NSReg leads five. Second, the best-epoch bonus depends on how the benchmark was built: 0.045--0.080 AUC-ROC on the three small relabeled-class graphs and 0.002--0.014 on the three real fraud graphs. Third, pseudo-labeling in OUTPOST is worth 0.038--0.065 AUC-ROC on the same three graphs but gives no benefit on any real fraud graph. We also show that a 0.002 tie band for hyperparameter selection lies below the paired standard error on all six graphs tested, even at ten seeds. Twelve of our 40 predictions were falsified, and we report them. We close with a short reporting checklist.

Abstract

Graph-Based Anomaly Detection

Explore similar work

The Selection Rule Decides the Winner: A Pre-Registered Audit of Open-Set Graph Anomaly Detection

Feature-Context Consistency: Unsupervised Adversarial Detection with Drift Stability on Attributed Graphs

GAD in the Wild: Benchmarking Graph Anomaly Detection under Realistic Deployment Challenges