PairAudit: Guiding Human Review with Graph Tokens under Distribution Shift

Jiran Tao; Binyan Jiang

cs.LGOct 7, 2026

PairAudit: Guiding Human Review with Graph Tokens under Distribution Shift

Authors: Jiran Tao, Binyan Jiang

Organizations: The Hong Kong Polytechnic University

Abstract

Intrusion detectors can confidently misclassify attacks that were not seen during training. Human review can correct these errors, but only a limited number of cases can be checked. Uncertainty-based review may overlook confident errors, while anomaly scores alone do not show whether changing the review plan will correct more errors. We introduce PairAudit to find overlooked errors and improve review under a fixed budget. Its graph tokens capture prediction patterns across connected nodes. Rather than building another predictor through feature aggregation, PairAudit uses unusual relational patterns to uncover potential errors in existing predictions. Human feedback then helps decide whether these findings justify changing review priorities. Experiments across security tasks show that PairAudit corrects more errors on average than uncertainty-based review, including more errors on unseen attacks. These gains account for all review costs and do not require retraining the detector.

Figures & tables

Symbol	Meaning
$\mathcal{U},N,K$	Target node pool, its size, and number of known review classes.
$p_{i},p_{ik},\hat{y}_{i},y_{i}$	Class probabilities, class- $k$ probability, predicted label and human label.
$H,s_{R}(i),R,R_{H}$	Reviewable node limit, Raw uncertainty, full Raw ordering, and its first $H$ nodes.
$S,u_{i},U(S)$	Reviewed set ( $\|S\|=H$ ), correction indicator, and correction count.
$v_{i},G,G_{\mathrm{OOD}}$	Unseen-subtype flag; overall and OOD net corrections relative to Raw.
$\bar{p}_{i},x_{i},z_{i}$	Mean neighbor posterior, input features and standardized graph token.

Table 1: Notation.

R,J

rank all

N

nodes;

C

is the

L

-node candidate set, and

S

is the

H

-node reviewed set.

Figure 1: Raw (upper row) and candidate replacements (lower row). Check marks indicate retention; crossed nodes overlap Raw’s first

H

. The die denotes uniform pair sampling; p/l gives candidate gain

+1

and Raw gain

0

. Feedback updates

p_{m}

(Equation ( 3 )); authorization also requires three non-ties. Both paid endpoints count within

H

(inset; Equation ( 4 )). Dashed lines connect queue segments. Node counts are illustrative.

Scenario	Fits	Raw	PairAudit	$G$	$G_{\mathrm{OOD}}$	Auth. (%)
UNSW / Fuzzers	20	224.95	327.90	+102.95	+105.72	100.000
CICIDS / Hulk	10	1097.60	1887.39	+789.79	+830.80	100.000
X-IIoTID / RDOS	3	440.33	780.63	+340.30	+461.00	100.000
TON / XSS	3	383.33	463.35	+80.01	+102.02	100.000

Table 2: Equal-budget corrections and net gains over Raw.

G_{\mathrm{OOD}}

counts only unseen-subtype errors.

Scenario	$N$	$H$	$H/N$ (%)	Labels	$2m/H$ (%)	On auth. (%)
UNSW / Fuzzers	25,305	2,531	10.00	144.4	5.71	5.71
CICIDS / Hulk	27,114	2,712	10.00	136.0	5.01	5.01
X-IIoTID / RDOS	14,187	1,419	10.00	72.0	5.07	5.07
TON / XSS	11,407	1,141	10.00	78.9	6.92	6.92

Table 3: Target population and review budget used before deciding.

Figure 2: (a) New authorizations at each nominal look, as percentages of all runs per task; the inset magnifies rare late decisions. (b) Early rejection on Coauthor Physics, with

H=1{,}282

reviews. Adopting the same candidate exchange

B\cup C

without audit corrects 863.33 errors; PairAudit corrects 1,097.37, including audit costs. Raw corrects 1,120.33. Trust Score is a separate ranking reference.

Method	UNSW / Fuzzers	CICIDS / Hulk	X-IIoTID / RDOS	TON / XSS
Raw	0.00 / 0.00	0.00 / 0.00	0.00 / 0.00	0.00 / 0.00
Trust Score	-23.90 / -29.90	+886.60 / +817.20	-41.00 / +0.67	+166.67 / +135.67
Energy	-37.15 / -36.05	-270.40 / -263.80	-12.67 / +0.00	+26.33 / +29.33
GNNSafe	-61.40 / -50.60	-546.60 / -549.70	-98.67 / +0.00	-13.33 / +26.67
GRASP	-11.20 / +3.90	+224.10 / +425.70	-155.67 / +0.00	-138.33 / -87.00
PairAudit	+102.95 / +105.72	+789.79 / +830.80	+340.30 / +461.00	+80.01 / +102.02

Table 4: Independent review strategies at equal budget. Entries are

G/G_{\mathrm{OOD}}

Task / pool	$N$	$H$	OOD (%)	$\rho_{\mathrm{err}}$ (%)	$G$	$G_{\mathrm{OOD}}$	Auth. (%)	Cost (%)
IoT23 / ARP (1%)	13,693	1,370	1.00	0.514	+58.59	+20.03	99.96	9.83
IoMT / ARP (full)	6,554	656	20.90	12.084	+44.19	+44.51	100.00	8.61
IoMT / ARP (95%)	1,442	145	95.01	54.924	+44.33	+44.33	100.00	5.53
BoT / DDoS-TCP	7,482	749	66.65	65.143	+37.81	+101.60	99.50	13.05

Table 5: Stress tests and multiclass review, relative to the equal-budget Raw baseline for each pool.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Task	$H$	OOD (%)	$G$	$G_{\mathrm{OOD}}$	Auth. (%)	Cost (%)
X-IIoTID / CoAP	1,419	35.24	+282.47	+447.33	100.00	5.07
WUSTL-IIoT-2021 / UDP	710	70.51	+118.64	+124.03	98.39	9.12

Appendix

Table 6: Predicted-attack priority on unseen normal activity, relative to equal-budget Raw.

Task	Benign priority + Raw	PairAudit
X-IIoTID / RDOS	$-102.67/0.00$	$+340.30/+461.00$
UNSW / Fuzzers	$+91.45/+104.85$	$+102.95/+105.72$

Appendix

Table 7: Benign-priority comparison. Entries are

G/G_{\mathrm{OOD}}

, relative to the original Raw ranking.

Task	Fits	Train acc.	Train F1	Val. acc.	Val. F1
UNSW / Fuzzers	20	0.996–0.997	0.993–0.994	0.995–0.997	0.992–0.995
CICIDS / Hulk	10	0.979–0.981	0.962–0.965	0.979–0.981	0.962–0.966
X-IIoTID / RDOS	3	0.952–0.953	0.951–0.952	0.939–0.945	0.937–0.944
TON / XSS	3	0.985–0.986	0.982–0.984	0.986–0.986	0.982–0.984
IoT23 / ARP	3	0.959–0.960	0.866–0.868	0.958–0.959	0.860–0.865
IoMT / ARP	3	0.968–0.974	0.951–0.961	0.964–0.971	0.945–0.956

Appendix

Table 8: Known-source qualification. Ranges span the fitted configurations, not independent events.

Scenario	Condition	$G$	$G_{\mathrm{OOD}}$	Auth. (%)	$2m/H$ (%)	Harm (%)
UNSW / Fuzzers	PairAudit	+102.95	+105.72	100.000	5.71	0.000
UNSW / Fuzzers	Fixed weights	+95.35	+98.28	100.000	5.75	0.000
UNSW / Fuzzers	Node marginal	+97.95	+100.35	100.000	5.51	0.000
UNSW / Fuzzers	Matched shuffle	+95.32	+97.81	100.000	5.72	0.000
UNSW / Fuzzers	Global shuffle	+57.98	+60.60	98.833	7.88	0.017
CICIDS / Hulk	PairAudit	+789.79	+830.80	100.000	5.01	0.000

Appendix

Table 9: Complete component and relational control metrics. All settings retain the same budget and ordinary correction utility.

Scenario	Reported	$M=1$	$2$	$3$	$4$	$8$	$16$
UNSW / Fuzzers	102.95	103.99	72.32	81.27	102.95	103.83	103.62
CICIDS / Hulk	789.79	720.16	792.47	792.94	789.60	800.41	789.80
X-IIoTID / RDOS	340.30	334.34	335.71	337.40	340.30	344.53	347.21
TON / XSS	80.01	-22.07	65.20	73.25	80.01	72.63	67.73
IoMT / ARP, full	44.19	-6.47	30.03	46.11	44.38	39.16	48.35
IoMT / ARP, 95% OOD	44.33	41.37	42.65	43.66	44.22	42.87	43.32

Appendix

Table 10: Component-count sensitivity: net corrections

G

at the same total review budget.

Figure 3: Exchange-fraction sensitivity at fixed

H

. (a) Net gain relative to mean Raw corrections. (b) Decision labels as a fraction of

H

. Points average the same primary fits and 300 audit runs per fit; the dotted line marks

\tau=1/3

. Table 11 includes exact gains, authorization and harm rates.

Scenario	Setting	$G$	$G_{\mathrm{OOD}}$	Auth. (%)	Cost (%)
UNSW / Fuzzers	0.1	+88.52	+88.80	100.000	5.06
UNSW / Fuzzers	0.2	+99.78	+101.16	100.000	5.15
UNSW / Fuzzers	$1/3$	+102.95	+105.74	100.000	5.65
UNSW / Fuzzers	0.4	+102.30	+105.66	100.000	6.04
UNSW / Fuzzers	0.5	+99.16	+104.02	100.000	6.97
CICIDS / Hulk	0.1	+213.15	+223.90	100.000	5.01

Appendix

Table 11: Sensitivity to the exchange fraction

\tau

. Each setting has 300 audit runs per fit.

Scenario	Setting	$G$	$G_{\mathrm{OOD}}$	Auth. (%)	Cost (%)
UNSW / Fuzzers	Ordered	+102.95	+105.74	100.000	5.65
UNSW / Fuzzers	Shuffle C	+102.96	+105.74	100.000	5.64
UNSW / Fuzzers	Shuffle D	+102.97	+105.74	100.000	5.62
UNSW / Fuzzers	Shuffle both	+102.95	+105.72	100.000	5.69
CICIDS / Hulk	Ordered	+789.81	+830.80	100.000	5.01
CICIDS / Hulk	Shuffle C	+789.81	+830.80	100.000	5.01

Appendix

Table 12: Sensitivity to pair formation. Each setting has 300 audit runs per fit.

Task	Fit ID	Mix/cohort	$N$	$H$	$G$	$G_{\mathrm{OOD}}$	Auth. (%)	Cost (%)
UNSW / Fuzzers	21	Full	25,305	2,531	+104.96	+104.97	100.00	5.11
UNSW / Fuzzers	22	Full	25,305	2,531	+166.39	+171.00	100.00	5.07
UNSW / Fuzzers	23	Full	25,305	2,531	+153.07	+153.00	100.00	5.06
UNSW / Fuzzers	24	Full	25,305	2,531	+44.11	+49.32	100.00	9.27
UNSW / Fuzzers	25	Full	25,305	2,531	+96.33	+98.95	100.00	5.32
UNSW / Fuzzers	26	Full	25,305	2,531	+96.35	+101.84	100.00	5.48

Appendix

Table 13: Main-text PairAudit configurations and workloads. Each row contains 600 audit runs; fit IDs identify frozen configurations in the supplement.

Explore similar work

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors
Feb 26, 2026Abhay Sheshadri, Aidan Ewart, Elias Kempf +8Model Auditing Artificial Intelligence Alignment
Anthropic Fellows Program · Anthropic
Benchmarking Agentic Review Systems
Jun 18, 2026Dang Nguyen, Wanqing Hao, Yanai Elazar +1Peer Review Precision Recall
University of Chicago · Bar-Ilan University
Send a SCOUT First: Pre-hoc Reasoning for Adaptive Detector Allocation in Prompt-Injection Defense
May 29, 2026Shuhao Zhang, Jiarui Li, Qi Cao +2Prompt-Injection Detectors LLM Defense Mechanisms
UC San Diego · University of Illinois Urbana-Champaign

Feb 26, 2026, cs.CLJ/K move · Enter open · S save

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

Abhay Sheshadri, Aidan Ewart, Elias Kempf, Kai Fronsdal+7

Anthropic Fellows Program · Anthropic

Model Auditing Artificial Intelligence Alignment Adversarial Evaluation Adversarial Training

Open paper AskPDF arXiv

We introduce AuditBench, an alignment auditing benchmark. AuditBench consists of 56 language models with implanted hidden behaviors. Each model has one of 14 concerning behaviors--such as sycophantic deference, opposition to AI regulation, or secret geopolitical loyalties--which it does not confess to when directly asked. AuditBench models are highly diverse--some are subtle, while others are overt, and we use varying training techniques both for implanting behaviors and training models not to confess. To demonstrate AuditBench's utility, we develop an investigator agent that autonomously employs a configurable set of auditing tools. By measuring investigator agent success using different tools, we can evaluate their efficacy. Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent. We find that our most effective tools involve scaffolded calls to auxiliary models that generate diverse prompts for the target. White-box interpretability tools can be helpful, but the agent performs best with black-box tools. We also find that audit success varies greatly across training techniques: models trained on synthetic documents are easier to audit than models trained on demonstrations, with better adversarial training further increasing auditing difficulty. We release our models, agent, and evaluation framework to support future quantitative, iterative science on alignment auditing.

Abstract

Explore similar work

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

Benchmarking Agentic Review Systems

Send a SCOUT First: Pre-hoc Reasoning for Adaptive Detector Allocation in Prompt-Injection Defense