AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis

Natapong Nitarach; Phume Ngampornsukswadi; Pittawat Taveekitworachai; Surapon Nonesung; Warit Sirichotedumrong; Kunat Pipatanakul

cs.CLOct 9, 2026

AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis

Authors: Natapong Nitarach, Phume Ngampornsukswadi, Pittawat Taveekitworachai, Surapon Nonesung, Warit Sirichotedumrong, Kunat Pipatanakul

Organizations: SCB DataX, SCBX Group · Independent Researcher

Abstract

Evaluating large language models (LLMs) on AMLworld's smallest test splits is costly because each of over a million targets requires a separate graph prompt. Sampling reduces this cost, but small balanced subsets change unweighted precision and F1. We introduce AMLworld-Compact, an evaluation set that retains every illicit test transaction and samples benign transactions from supervised-score strata. Its HT-Coreset protocol reduces evaluation targets by 271x and 611x on HI-Small and LI-Small. Horvitz-Thompson weights give unbiased full-test confusion-count estimates for fixed predictions; recall is exact, while precision and F1 are ratio estimates. In a held-out study using Graph Feature Preprocessor (GFP) features, one boosted-tree family defines strata for evaluating the other over 500 draws per configuration. At the released sizes, F1 RMSE is 0.00-2.58 percentage points versus 4.44-7.41 for uniform sampling with HT weights; conservative intervals remain broad. Under the evaluated graph prompts, seven open-weight LLMs have weighted precision near the predict-all-illicit baseline. In an outcome-stratified audit of 1,000 HI-Small few-shot outputs from one run per model, lexical checks pass for 84.5% of correct and 56.2% of incorrect binary verdicts, so these markers do not establish correctness. We release the evaluation sets, construction code, and audit tools at https://huggingface.co/datasets/typhoon-ai/AMLworldCompactEval and https://github.com/nat-nischw/AMLworld-Compact.

Figures & tables

Figure 1: HT-Coreset (a) retains illicit targets (diamonds) and samples benign targets (circles). High-score benign targets form one stratum, defined by the construction ensemble; all fit at the released budgets. The other benign targets form three difficulty terciles (shaded rows). In (b), predictions on the targets selected in (a) are weighted by

1/\pi_{e}

for full-test estimates or counted equally for subset diagnostics;

\pi_{e}

is the inclusion probability. Markers represent transaction targets; marker counts are illustrative.

	HI-Small	LI-Small
Full test edges	$1{,}015{,}669$	$1{,}384{,}810$
Retained targets	$3{,}753$	$2{,}268$
Illicit edges (all retained)	$1{,}251$	$756$
Full-test illicit rate	$0.12\%$	$0.05\%$
Subset illicit rate	$33.3\%$	$33.3\%$
Target-count reduction	$271\times$	$611\times$

Table 1: Full-test and released-set sizes. Each retained edge is one evaluation target with a neighborhood from the full graph. Token estimates concern only the released graph strings, using four characters per token; instructions, demonstrations, and outputs are excluded. HT weights account for the altered class proportions.

Figure 2: Full-split agreement for the construction ensemble. (a) HT-Coreset has

|\Delta F1|{<}10^{-13}

pp at

12

budgets; stars mark the maximum tested reductions:

271\times

on HI-Small and

611\times

on LI-Small. Gray curves and labels show subset illicit percentages (right axes). (b)

|\Delta F1|

at the released budgets (

50

-draw means). B3/B4 meet the

5

pp F1 tolerance but exceed it on precision (Table 9 ). Exact agreement follows from retaining this ensemble’s TP/FP/FN contributors at weight one.

Sampler	F1 RMSE (pp)	F1 CI width (pp)	Coverage (%)
HI-Small: LGB $\to$ XGB
Uniform + HT	6.76	40.3	100.0
Terciles + HT	6.74	58.7	99.6
HT-Coreset	0.00	55.0	100.0
HI-Small: XGB $\to$ LGB
Uniform + HT	7.41	40.4	99.8

Table 2: Sampling with a held-out predictor family at the released target counts. LGB=LightGBM; XGB=XGBoost. The arrow runs from the stratum constructor to the evaluated family. All samplers retain every illicit edge and use HT weights. HT-Coreset combines hard-negative retention with difficulty terciles. RMSE and mean

95\%

confidence-interval (CI) width concern F1 and are in percentage points; coverage is a percentage over

500

draws. Each draw estimates the mean of five fixed, validation-thresholded seed metrics. Bold marks the lowest F1 RMSE within each dataset–direction block. Intervals are conservative and can remain broad despite small point errors.

Model	M	F1	Lift	F1	P	R	TF1
		HT-weighted (full test)		Unweighted (subset)
GPT-OSS-20B	FS	0.236	0.96	46.4	32.8	79.2	10.5
GPT-OSS-20B	ZS	0.246	1.00	49.9	33.3	99.5	5.6
Qwen3.5-27B	FS	0.240	0.98	40.7	32.5	54.5	10.1
Qwen3.5-27B	ZS	0.230	0.94	45.8	31.6	82.9	6.3
Nem-Nano-30B	FS	0.245	1.00	49.5	33.4	95.4	7.5

Table 3: LLM results on HI-Small, means over

5

runs. All metrics are percentages except Lift; M is prompting method (FS=ICL-FS, ZS=ICL-ZS). Both summaries use the same HT-Coreset targets and predictions. Qwen3.5-35B denotes the 35B-A3B model. R and TF1 are unchanged by weighting because their illicit edges have weight one. Full results for both splits and unweighted run SD are in Table 14 ; HT run SD and sampling uncertainty are in Table 15 .

Detection outcome	Illicit	Benign	$\bar{s}_{\mathrm{ML}}$
Both correct	652	311	0.69
ML only correct	59	2,064	0.08
LLM only correct	410	23	0.13
Both wrong	130	104	0.40

Table 4: Detection-error decomposition on HI-Small (GPT-OSS-120B, few-shot;

3{,}753

edges). Illicit and Benign are ground-truth edge counts. Groups use binary detection correctness. Each edge’s ML score averages the two boosters’ illicit probabilities over five training seeds.

\bar{s}_{\mathrm{ML}}

averages these scores within each group; ML decisions use threshold

0.80

Figure 3: Predicted labels on ground-truth-benign cases in HI-Small (HI) and LI-Small (LI). Each row shows percentages of all benign cases pooled over five runs. Legitimate is the benign label;

\langle\mathrm{none}\rangle

denotes a missing typology label. Typology concentrations vary with model and prompting; dominant-label shares appear in Section F.5 .

Binary verdict	$n$	Full trace	First 8k chars
Correct	470	397 (84.5%)	388 (82.6%)
Incorrect	530	298 (56.2%)	295 (55.7%)

Table 5: Revised lexical checks by binary correctness in the

1{,}000

-trace audit. Counts require Parse, Recall, and at least two structural-marker categories for every verdict, removing the original benign-verdict Match bypass. Percentages use the row denominator. The 8k-character window matches the API judges’ text limit. These are outcome-stratified descriptive rates, not population estimates.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Typology	HI-Small		LI-Small
	$n$	%	$n$	%
gather-scatter	$259$	$32.7$	$27$	$15.5$
scatter-gather	$142$	$18.0$	$30$	$17.2$
fan-in	$\phantom{0}90$	$11.4$	$11$	$\phantom{0}6.3$
fan-out	$\phantom{0}85$	$10.7$	$29$	$16.7$
cycle	$\phantom{0}74$	$\phantom{0}9.4$	$30$	$17.2$

Appendix

Table 6: Ground-truth typology distribution in HT-Coreset subsets (typology-labeled illicit edges; % within labeled-illicit). Of the illicit edges,

791

1{,}251

on HI-Small and

174

756

on LI-Small carry explicit typology labels in AMLworld; the remainder are illicit without a programmatic typology assignment. stack is the rarest labeled typology (

4.0\%

HI-Small), gather-scatter the most common (

32.7\%

HI-Small).

Source	$k$	Cap	Purpose
Altman et al. (2023)	$2$	$100$	GNN training
Pirmorad (2025)	— †	—	LLM ICL
Ours	$\mathbf{2}$	$\mathbf{50}$	LLM ICL

Appendix

Table 7: Subgraph extraction parameters vs. prior work. Cap

=

neighbors per expanded account per BFS hop. † Pipeline described as “extract a localized

k

-hop subgraph” (their Section II-A) with a

2

-hop illustration (their Figure

2

); neither

k

nor a per-hop neighbor cap is stated explicitly.

Model	Arch	Active	Context	TP	temp	top_p	top_k
GPT-OSS-20B	MoE	3.6B	128K	1	1.0	1.0	—
Nemotron-3-Nano-30B	MoE	3B	256K	2	1.0	0.95	—
Qwen3.5-35B-A3B	MoE	3B	256K	2	0.6	0.95	20
GPT-OSS-120B	MoE	5.1B	128K	4	1.0	1.0	—
Nemotron-3-Super-120B	MoE	12B	256K	4	1.0	0.95	—
Qwen3.5-397B-A17B	MoE	17B	256K	8	0.6	0.95	20

Appendix

Table 8: Open-weight model configurations and decoding parameters. Context = deployed max_model_len ; TP = tensor-parallel size. All models use max_tokens

=32{,}768

. † Qwen3.5-27B has 256K native context but we deployed at 128K under

4

-way tensor parallelism (VRAM-bound); this cap drives the

38.7\%

no-response rate analyzed in Section 5.2 .

	$\|\Delta$ F1 $\|$	$\|\Delta$ P $\|$	$\|\Delta$ R $\|$	$\sigma$
HI-Small (Full: P=93.1, R=57.2, F1=70.9)
B1 Random	18.1	15.6	22.3	24.7
B2 Stratified	16.0	11.1	18.0	19.9
B3 No IW	1.9	6.9	0.0	0.0
B4 No Hard-Neg	2.6	8.7	0.0	3.3
B5 Fogliato	6.3	3.7	8.7	7.9

Appendix

Table 9: Construction ablation for the original three-scorer ensemble (

k{=}50

draws per method). On HI-Small,

n{=}3{,}753

and the threshold is

0.7881

; on LI-Small,

n{=}2{,}268

and it is

0.4877

. All

\Delta

values and

\sigma

(std of detection F1) are in pp. HT-Coreset retains every edge contributing to this ensemble’s TP, FP, and FN, giving exact P/R and numerical agreement on F1 (

|\Delta

|<10^{-13}

pp).

HI-Small: LGB $\to$ XGB, benign budget $4\times$ illicit count
Sampler	P RMSE	F1 bias	F1 RMSE	F1 width	Cov.
Uniform + HT	14.46	+0.48	5.04	29.6	100.0
Terciles + HT	13.84	+0.63	4.74	49.6	100.0
HT-Coreset	0.00	+0.00	0.00	44.9	100.0
HI-Small: LGB $\to$ XGB, benign budget $8\times$ illicit count
Uniform + HT	10.63	+0.38	3.64	21.0	100.0

Appendix

Table 10: Larger budgets in the held-out-family study (

500

draws per configuration). Bias is mean estimated F1 minus exact full-test F1. RMSE, bias, and mean

95\%

interval width are in percentage points; coverage is a percentage. Every quantity concerns the mean of the five fixed seed metrics. The primary

1{:}2

budgets appear in Table 2 ; all budgets and per-seed results are released as CSVs.

HI-Small
Method	F1	P	R
LightGBM+GFP	65.7 $\pm$ 1.2	77.9 $\pm$ 4.4	56.9 $\pm$ 0.7
XGBoost+GFP	67.7 $\pm$ 0.3	81.7 $\pm$ 2.1	57.8 $\pm$ 0.6
Two boosters	67.8 $\pm$ 0.2	84.3 $\pm$ 0.5	56.7 $\pm$ 0.2
LI-Small
LightGBM+GFP	27.5 $\pm$ 0.5	50.0 $\pm$ 4.9	19.0 $\pm$ 0.4

Appendix

Table 11: Supervised detection baselines on the full file-order test partition (mean

\pm

std, five training seeds). Means are percentages; stds are in pp.

Method	LightGBM	XGBoost	GCPAL
HI-Small
B1 Random	1.1	2.5	7.5
B2 Stratified	15.8	19.8	9.1
B3 No IW	3.0	2.3	5.3
B4 No Hard-Neg	4.5	2.3	5.3
B5 Fogliato	7.7	8.2	1.4

Appendix

Table 12: Per-model

|\Delta F1|

(pp), averaged over five training seeds on one canonical draw per sampling method. Each cell compares with that individual scorer’s full-set F1; Table 9 instead uses the construction ensemble. All models use GFP features. HT-Coreset’s rounded zeros include small nonzero deviations; only its XGBoost result on HI-Small matches to numerical precision (

4{\times}10^{-15}

pp).

	$\|\Delta F1\|$	$\|\Delta$ P $\|$	$\|\Delta$ R $\|$	$\sigma$	$p_{W}$
B1 Random	$\phantom{0}1.57$	$\phantom{0}1.42$	$\phantom{0}2.22$	$1.85$	$<10^{-4}$
B2 Stratified	$\phantom{0}1.62$	$\phantom{0}1.77$	$\phantom{0}2.13$	$2.02$	$<10^{-4}$
B3 No IW	$\phantom{0}0.00$	$\phantom{0}0.00$	$\phantom{0}0.00$	$0.00$	—
B4 No Hard-Neg	$\phantom{0}3.79$	$10.01$	$\phantom{0}0.00$	$0.00$	$<10^{-4}$
B5 Fogliato	$\phantom{0}0.93$	$\phantom{0}0.64$	$\phantom{0}1.48$	$1.14$	$<10^{-4}$
B6 Leskovec	$\phantom{0}1.01$	$\phantom{0}2.56$	$\phantom{0}0.00$	$0.90$	$<10^{-4}$

Appendix

Table 13: Elliptic real-data ablation (

n{=}3{,}249

k{=}50

draws). All

\Delta

and

\sigma

are in pp;

\sigma

= std of subset

F1

;

p_{W}

= one-sided Wilcoxon vs HT-Coreset (

p

undefined for B3 because every draw has the same metrics as HT-Coreset on this dataset). B3 follows the Elliptic HT selection rule with unit weights; its AMLworld counterpart instead samples benign edges uniformly. B4 excludes the hard-negative stratum and cannot estimate its contribution to full-test counts. The

0.00

entries in the

|\Delta F1|

column (HT-Coreset and B3) are likewise

<10^{-13}

pp; the

|\Delta

|

and

|\Delta

|

zeros are exact.

		HT-weighted (full test)			Unweighted (subset)		Not reweighted
Model	M	$F1_{\text{HT}}$	$P_{\text{HT}}$	Lift	F1 $\pm$ SD	P	R	TF1	V%
HI-Small : population illicit rate $0.1232\%$ , predict-all floor $F1_{\text{HT}}{=}0.2460\%$
GPT-OSS-20B	FS	0.2360	0.1182	0.959	$46.4\pm 0.40$	32.8	79.2	10.5	88.3
GPT-OSS-20B	ZS	0.2457	0.1230	0.999	$49.9\pm 0.08$	33.3	99.5	5.6	99.8
Qwen3.5-27B	FS	0.2404	0.1205	0.978	$40.7\pm 0.08$	32.5	54.5	10.1	61.5
Qwen3.5-27B	ZS	0.2302	0.1153	0.936	$45.8\pm 0.05$	31.6	82.9	6.3	88.1

Appendix

Table 14: Full LLM results on both test partitions, means over five stochastic runs. Both summaries use the same HT-Coreset targets and predictions. Metrics are percentages except Lift (

P_{\rm HT}

divided by full-test prevalence). Unweighted F1 includes its run SD in percentage points; HT-F1 SD and sampling intervals appear in Table 15 . R and TF1 are unchanged by weighting because all illicit targets are retained. V% is the fraction of requests with positive recorded token usage, including some without a final answer.

Model	M	$F1_{\rm HT}$	$\mathrm{SD}_{\rm run}$	$\mathrm{SE}_{\rm samp}$	95% interval	Lift	95% interval
		(%)	( $10^{-3}$ pp)	( $10^{-3}$ pp)	(%)
HI-Small
GPT-OSS-20B	FS	0.2360	2.34	2.02	$[0.2254,0.2493]$	0.959	$[0.9165,1.0136]$
GPT-OSS-20B	ZS	0.2457	0.76	0.21	$[0.2448,0.2490]$	0.999	$[0.9950,1.0117]$
Qwen3.5-27B	FS	0.2404	0.90	4.52	$[0.2191,0.2667]$	0.978	$[0.8915,1.0851]$
Qwen3.5-27B	ZS	0.2302	0.43	1.75	$[0.2224,0.2406]$	0.936	$[0.9041,0.9778]$

Appendix

Table 15: Sampling uncertainty for the 28 five-run mean LLM results. F1 and its interval endpoints are percentages; run SD and linearized sampling SE are in

10^{-3}

percentage points. Lift is precision divided by full-test illicit prevalence. Intervals are conservative, cellwise 95% confidence envelopes obtained from simultaneous hypergeometric bounds for 15 run–stratum counts ( Section D.2 ). They condition on the frozen design, realized allocation, and five fixed prediction vectors; they do not include uncertainty over future inference runs. Coverage is not simultaneous across the 28 cells. Endpoints are rounded outward.

Split	Model	$\Delta F1_{\rm HT}$	$\mathrm{SE}_{\rm paired}$	95% interval	$\Delta$ Lift	95% interval
		(pp)	( $10^{-3}$ pp)	(pp)
HI-Small	GPT-OSS-20B	-0.0098	2.01	$[-0.0244,0.0056]$	-0.039	$[-0.100,0.023]$
HI-Small	Qwen3.5-27B	0.0101	4.14	$[-0.0235,0.0468]$	0.042	$[-0.095,0.192]$
HI-Small	Nem-Nano-30B	-0.0006	0.92	$[-0.0082,0.0073]$	-0.002	$[-0.034,0.030]$
HI-Small	Qwen3.5-35B-A3B	0.0063	1.61	$[-0.0118,0.0252]$	0.026	$[-0.048,0.103]$
HI-Small	GPT-OSS-120B	0.0037	0.90	$[-0.0041,0.0119]$	0.015	$[-0.017,0.049]$

Appendix

Table 16: Few-shot minus zero-shot differences in five-run mean metrics. F1 differences and endpoints are percentage points; paired linearized SE is in

10^{-3}

percentage points. The SE uses same-edge covariance across both prompts and all runs. Conservative 95% envelopes use simultaneous bounds on 30 run–stratum counts and do not exploit pairing. Coverage is per comparison, not simultaneous across this table. Endpoints are rounded outward.

LLM	Group	n	ill.	$\bar{s}$	$\sigma_{s}$
HI-Small
GPT-OSS-20B	Both correct	1,142	566	.53	.47
	ML only correct	1,944	145	.11	.29
	LLM only correct	407	366	.17	.28
	Both wrong	260	174	.31	.41
Nem-Nano	Both correct	802	700	.86	.30

Appendix

Table 17: Detection-error groups on HI-Small (few-shot, run

42

n

and ill. are unweighted total and illicit counts;

\bar{s}

and

\sigma_{s}

are the mean and std of the two-booster ML score (threshold

0.80

). The main task results average five runs.

Figure 4: Per-class typology

F1

on HI-Small (few-shot, run

42

; ML threshold

0.80

), conditional on correct illicit detection. The ensemble’s macro-

F1

72.7\%

versus

6.6

–

15.8\%

for LLMs. Unlike the TF1 in Table 3 , this comparison uses each model’s true positives. Class supports appear in Table 6 .

Typology	n	ML Ens.	LLM
fan-in	90	58.9	98.9
bipartite	43	60.5	97.7
random	66	84.8	97.0
fan-out	85	94.1	88.2
stack	32	96.9	100.0
cycle	74	100.0	98.6

Appendix

Table 18: Per-typology recall (%) on HI-Small illicit edges with programmatic labels (

n{=}791

total). ML Ens. = soft-vote of LightGBM and XGBoost at

\tau{=}0.80

; LLM = GPT-OSS-120B FS (run

42

). Bold = ML’s weakest typologies, where LLMs recover the missed illicit edges.

Model	Cor.	Over	Und.	Typo	NR
GPT-OSS-20B	17.5	50.2	1.8	13.5	17.0
GPT-OSS-120B	15.5	57.8	0.2	16.2	10.4
Nem-Nano-30B	18.9	63.6	0.6	14.9	2.0
Nem-Super-120B	24.9	54.5	4.7	13.9	2.0
Qwen3.5-35B-A3B	28.4	54.4	4.6	11.7	0.9
Qwen3.5-397B	22.3	63.3	0.1	13.4	0.9

Appendix

Table 19: Failure taxonomy on HI-Small (ICL-FS), in percent of cases. Cor. =correct binary verdict and, where a reference exists, correct typology; Over =over-prediction (FP), Und. =under-prediction (FN), Typo =wrong typology given correct detection, NR =no response, meaning the request returned no output to score rather than output we failed to parse. For Qwen3.5-27B these are API-layer rejections from a context-budget mismatch, not a reasoning outcome ( Section H.5 ). All columns are unweighted percentages of the

3{,}753

run-

42

cases. NR counts requests with no returned response or reasoning text; an empty final response with a nonempty reasoning trace is not counted as NR.

Method	F1	P	R	TF1	%C
HI-Small (ML threshold = 0.80)
ML Ensemble	68.1	84.8	56.8	72.7	0
Score-gate (14-cell mean)	65.3	73.8	58.5	71.2	5
Same-route always illicit	87.3	77.5	100.0	—	0
DT + GPT-OSS-20B	87.3	77.6	99.7	57.0	21
DT + Qwen3.5-27B	82.0	75.9	89.1	59.7	21

Appendix

Table 20: DT results on AMLworld-Compact (

1{:}2

subset ratio). All metrics are percentages. DT rows use ICL-ZS and report means over

5

runs (

F1

std

<0.3

pp; TF1 std up to

1.0

pp on LI-Small GPT-OSS-20B). Detection

F1

, P, and R use HT-weighted counts; TF1 is typology macro-

F1

on correctly detected illicit edges with a valid typology output. %C is the percentage of compact-set edges whose rule uses an LLM prediction. The always-illicit control uses DT’s same eligible route (

21\%

of HI-Small edges and

35\%

of LI-Small edges) but requires no LLM calls and supplies no typology on routed edges, so its TF1 is unreported. DT uses strata constructed from ground-truth labels. Score-gate rows are means over

14

separately grid-searched model–prompting cells (

7

models, FS and ZS, run

42

) for the score-only recall gate, using DT’s typology fusion. Their per-cell

F1

ranges are

64.7

–

66.4\%

on HI-Small and

33.4

–

37.2\%

on LI-Small ( Section G.2 ).

Model	Compact pass		DT estimate
	HI	LI	HI	LI
GPT-OSS-20B	$107$	$\phantom{0}67$	$22$	$23$
GPT-OSS-120B	$106$	$\phantom{0}66$	$22$	$23$
Qwen3.5-27B	$\phantom{0}98$	$\phantom{0}51$	$20$	$18$
Nem-Nano-30B	$154$	$\phantom{0}95$	$32$	$33$
Nem-Super-120B	$156$	$\phantom{0}96$	$32$	$33$

Appendix

Table 21: Model-token counts per compact-set pass (millions, run 42, ICL-ZS). A compact-pass entry is (mean input

+

mean output tokens)

\times

the coreset size. DT estimates scale the unrounded full-pass cost by

776/3{,}753

on HI-Small and

789/2{,}268

on LI-Small (

21\%

and

35\%

consultation, respectively), assuming consulted edges have the same mean token cost as the full set. This gives estimated savings of

79\%

on HI-Small and

65\%

on LI-Small; the relative spread across models is unchanged.

	Pass rate (%)				Agreement ( $\kappa$ )
Step	Regex	DeepSeek	Opus	Gemini	Fleiss	Mean pair	Min pair	Max pair
Parse	80.4	83.5	73.9	81.3	0.80	0.81	0.71	0.91
Recall	82.8	80.7	76.6	81.8	0.90	0.90	0.81	0.97
Match	75.7	80.1	70.3	73.7	0.64	0.64	0.51	0.77
Conclude	19.2	24.0	27.1	20.9	0.73	0.74	0.66	0.87

Appendix

Table 22: Original step-rubric results on the same

1{,}000

HI-Small traces (few-shot, run

42

), including empty traces. Rates pool all seven models; agreement uses item-level labels from four annotators and six pairs. The regex heuristic uses full traces and API judges use the first

8{,}000

characters.

Figure 5: Regex step-rubric pass rates by model on the $1{,}000$ -trace HI-Small audit slice (ICL-FS, run $42$ ). Pooled annotator rates and agreement appear in Table 22 .

Checks	References	Full	First 8k chars
Original regex	All	538/1000	531/1000
Original regex	Available	421/760	417/760
No bypass	All	533/1000	525/1000
No bypass	Available	417/760	412/760

Appendix

Table 23: Traces passing the specified lexical checks but failing the stated answer checks. Rows marked All retain archived Conclude failures, including missing-reference cases; Available-reference rows exclude all

240

illicit cases without a reference typology; benign cases remain evaluable. No-bypass rows require two structural-marker categories even for benign verdicts. Original labels and scores remain unchanged; each row applies a stated sensitivity rule to the same archived outputs.

Step	NLP	Jr.	Sr.	Mean	Fleiss’ $\kappa$
Parse	100.0	100.0	96.4	98.8	$-0.01$
Recall	100.0	100.0	100.0	100.0	undefined †
Match	53.6	96.4	67.9	72.6	0.10
Conclude	21.4	21.4	21.4	21.4	1.00

Appendix

Table 24: Human pass rates (%) and agreement on

28

traces. Parse has

27/28

unanimous items but expected agreement of

0.976

, yielding

\kappa=-0.01

. † Recall is uniformly positive. Conclude agreement is imposed by the shared parser score.

Model	Rate	Mean	Median	Rate	Mean	Median	Correct	Wrong	$\Delta$
	Few-shot			Zero-shot			Few-shot by correctness
GPT-OSS-20B	83.0	4,147	3,438	99.8	2,179	2,013	4,176	4,129	$-47$
Qwen3.5-27B	61.3	14,764	14,678	88.1	12,647	12,125	14,280	15,041	$+761$
Nem-Nano-30B	98.0	2,810	2,343	100.0	1,485	1,441	2,853	2,788	$-66$
Qwen3.5-35B-A3B	99.1	22,208	19,204	100.0	15,566	14,585	23,980	21,003	$-2{,}977$
GPT-OSS-120B	89.6	1,998	1,899	99.8	1,147	1,117	1,965	2,015	$+51$

Appendix

Table 25: Trace statistics on HI-Small (run

42

). Rate is the percentage of cases with nonempty reasoning_content ; all lengths are character counts of these traces. Correct and Wrong give the mean lengths for correct and incorrect binary verdicts under ICL-FS;

\Delta

reports wrong minus correct. Displayed lengths and differences are rounded.

Model	D	F1	P	R	Over	Und
GPT-OSS-120B	HI	$-0.94$	$+0.12$	$-6.4$	$-4.6$	$+2.1$
GPT-OSS-120B	LI	$+0.79$	$+1.41$	$-4.1$	$-6.6$	$+1.4$
Nem-Super-120B	HI	$-3.00$	$+2.16$	$-22.9$	$-18.6$	$+7.6$
Nem-Super-120B	LI	$-2.23$	$+2.75$	$-25.1$	$-21.3$	$+8.4$
Pooled mean	–	$-1.35$	$+1.61$	$-14.6$	$-12.8$	$+4.9$

Appendix

Table 26: Verification prompt effect (ICL-V

-

ICL-FS), in percentage points of unweighted compact-set metrics (run

42

). D=dataset (HI or LI); Over/Und use FP/FN counts divided by all cases in the compact set. Precision rises and recall falls in all four cells;

F1

changes have mixed signs.

Model	Variant	Det-F1	Det-P	Det-R	Typ-F1	Typ-Acc
DeepSeek-V4-Pro	ZS-Graph	$89.3$	$88.8$	$89.8$	$20.9$	$26.3$
	ZS-Base	$89.0$	$88.7$	$89.2$	$17.3$	$21.7$
	FS-Base	$86.4$	$90.1$	$83.0$	$26.5$	$28.3$
	ZS-Abstain	$51.2$	$90.0$	$35.8$	$16.1$	$21.7$
	FS-CoT-Elim	$82.3$	$88.8$	$76.7$	$21.3$	$23.7$
	FS-TypFirst	$90.2$	$89.0$	$91.5$	$\mathbf{30.3}$	$\mathbf{32.3}$

Appendix

Table 27: Frontier API probe on

198

cases:

22

per class across

8

illicit typologies and benign (

8{:}1

illicit-to-benign ratio). All metrics are percentages. Typology metrics use this nine-class pool; detection uses the parsed typology, not the stated binary verdict. Missing typologies (

249

3{,}564

responses) count as benign in all metrics. The

18

scored cells have no API errors. Bold marks column maxima across those cells. Instructions and exact template stems appear in Section I.3.1 .

Stratified Sampling

See all 10 papers

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation
Jul 30, 2026Philipp D. Siedler, Jordan SassoonLLM Evaluation Benchmark Auditing
Aleph Alpha Research, Heidelberg, Germany.
Explainable AML Triage with LLMs: Evidence Retrieval and Counterfactual Checks
Mar 22, 2026Dorothy Torres, Wei Cheng, Ke HuAnti-Money Laundering Anti-Money Laundering Detection
School of Science, Technology, Engineering and Mathematics 263 Academy Street, Room S204 Jersey City, NJ 07306 · School of Electrical Engineering and Computer Science, The Pennsylvania State University, 207 Electrical Engineering West, University Park, PA 16802, USA
Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
Jan 30, 2026Zhuochun Li, Yong Zhang, Ming Li +8LLM Evaluation LLM-as-a-Judge
Ping An Technology (Shenzhen) Co., Ltd. · University of Pittsburgh · University of Maryland, College Park +1

Jul 30, 2026J/K move · Enter open · S save

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

Philipp D. Siedler, Jordan Sassoon

Aleph Alpha Research, Heidelberg, Germany.

LLM Evaluation Benchmark Auditing Benchmark Construction LLM Auditing

Open paper AskPDF arXiv

Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples. We introduce a dataset-centric meta-evaluation framework that audits benchmark datasets at the sample level along five latent dimensions: 1. Cognitive and Knowledge Demands, 2. Language and Content Quality, 3. Task Properties, 4. Context, and 5. Ethics, Safety, and Fairness. Applying this framework, we annotate five influential benchmarks -- MMLU, ARC, WinoGrande, HellaSwag, and TruthfulQA -- revealing pronounced internal heterogeneity that is not captured by aggregate accuracy scores. We show how these annotations enable criterion-driven orchestration of composite benchmark subsets across datasets, supporting targeted evaluation of model capabilities such as Reasoning Depth or Ethical Sensitivity. This approach reframes benchmark evaluation as dataset introspection, providing a principled methodology for analyzing and re-composing existing benchmarks to better reflect diverse evaluation needs.

Abstract

Stratified Sampling

Explore similar work

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

Explainable AML Triage with LLMs: Evidence Retrieval and Counterfactual Checks

Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry