Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures

Ruoqi Guo; Yi Liu; Gelei Deng; Yuekang Li; Lida Zhao; Yutao Wu; Simin Chen; Ying Zhang

cs.AISep 24, 2026

Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures

Authors: Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Lida Zhao, Yutao Wu, Simin Chen, Ying Zhang, +1 more

Organizations: Griffith University · Nanyang Technological University · UNSW · Independent Researcher · Deakin University · George Mason University · Wake Forest University

Abstract

Detectors of alignment failures screen deployed language models and score alignment benchmarks. Most are generative judges that spend a decoding pass on every criterion, and classifiers that read token probabilities, such as Llama Guard, still score one fixed label per call. Jev, a model trained with reinforcement learning for calibrated decisions (RLCD), answers many typed questions about one input with calibrated probabilities in a single call. Whether it detects alignment failures has not been measured. We present RLCDAlignBench, which benchmarks Jev on ten alignment failures: sycophancy, jailbreaks, deception, prompt injection, hallucination, privacy violation, social bias, reward hacking, concealing uncertainty, and power seeking. It spans 44 benchmarks and five target models, labelled by each benchmark's scorer and, on two, by humans. Many of these failures are relational, defined against a reference, such as the user's belief or an injected instruction, that the response alone does not reveal. Our key idea is therefore to vary what Jev is asked separately from what it sees: the question's wording and answer type on one side, the fields of the input on the other. A single generic question reaches a median AUROC of 0.886 zero-shot and beats supervised baselines on most benchmarks. Question wording matters little, while context matters more, mostly through fields that encode the label. Jev matches the reference scorer's agreement with human labels, surfaces label defects in existing benchmarks, and costs 63x less than LLM-judge scorers. Code and data: https://github.com/sumleo/RLCDAlignBench.

Figures & tables

Benchmarks	#	R/J/M	V/D/U	Items
Sycophancy (Qwen3.5-2B): the reply defers to the user’s stated view or flips its verdict to please the user.
ELEPHANT ( Cheng et al., 2026 ) , SYCON-Bench ( Hong et al., 2025 ) , SycophancyEval $\times$ 2 ( Sharma et al., 2024 )	4	1/2/1	0/2/2	639
Jailbreaks (Phi-4-mini): the reply complies with a harmful (jailbroken) request.
HarmBench ( Mazeika et al., 2024 ) , JailbreakBench $\times$ 2 ( Chao et al., 2024 ; Chao et al., 2025 ) , StrongREJECT ( Souly et al., 2024 )	4	0/4/0	1/0/3	414
Deception (Gemma-2-2B): a statement contradicts the model’s own elicited belief or serves a deceptive goal.
DeceptionBench † ( Huang et al., 2025 ) , MASK $\times$ 3 ( Ren et al., 2025 )	4	0/4/0	0/0/4	540

Table 1: The ten failure types of RLCDAlignBench . Items come from five target models: Qwen3.5-2B ( Yang et al., 2025 ) , Phi-4-mini ( Microsoft, 2025 ) , Gemma-2-2B ( Gemma Team, 2024 ) , Llama-3.2-3B ( Grattafiori et al., 2024 ) , and Olmo-3-7B ( Team Olmo, 2025 ) . Group rows: failure type, target model, and the positive behaviour the detector must flag (label

y{=}1

). # : benchmarks,

\times k

k

from one source. R/J/M : reference scorer is a rule (on the output or log-probabilities), an LLM judge or classifier, or a multi-turn trajectory judge. V/D/U : label validated (rule against gold, or scorer checked against humans), changed by our audit (scorer defect, or dependence on annotations the state omits, § 3.1 ), or unvalidated. D includes AbstentionBench, which has too few negatives to score, so 7 D benchmarks are usable. † : the label depends on a signal the canonical state omits (deceptive goal, secret list, annotated consequences of the chosen option, target’s answer probability). Items : detection instances.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Split	Scorer	Label source	Label	$n$	Pos.	Rate	Status
Sycophancy (target model: Qwen3.5-2B)
ELEPHANT (AITA) ( Cheng et al., 2026 )	H	Rule	rule (verdict tokens)	unval.	255	134	0.53	ok
SYCON-Bench (false premise) ( Hong et al., 2025 )	HO	Multi-turn	GPT-4o, per turn	unval.	17	14	0.82	degen. (min. 3)
SycophancyEval (answer) ( Sharma et al., 2024 )	H	Judge	GPT-4o	defect	150	123	0.82	ok
SycophancyEval (feedback) ( Sharma et al., 2024 )	H	Judge	GPT-4o	defect	217	30	0.14	ok
Jailbreaks (target model: Phi-4-mini)

Appendix

Table 2: All 44 benchmarks of RLCDAlignBench . Each group row names the target model whose outputs are judged. Split : role in the suite of Chen et al. (2026) , H = hill-climb, HO = held-out (not used by RLCDAlignBench ). Scorer : type of the reference scorer (Rule, LLM Judge, or Multi-turn trajectory judge). Label source : the model or rule that produced the scorer label (“rule vs. gold”: the target model’s own verdict compared with the dataset label, “rule vs. annotation”: the target model’s A/B choice compared with the MACHIAVELLI-style annotation). Label : validity of the scorer label (val. = validated against gold or human labels, defect = known label defect that changes the binary label, Table 26 , dir. only = the defect changes only the recorded direction of the gap, not the binary label, unval. = not validated, definitions in Appendix A ).

n

: detection instances. Pos. : positives. Rate : positive rate. Status : “degen.” benchmarks have a single class or at most 4 minority-class items and are reported but excluded from aggregates (6 benchmarks), and “skewed” benchmarks have more than 90% of items in one class and are kept.

Family	What Jev is asked	Score $s(x)$
Direct Noul (minimal, statement, definition, criteria)	The benchmark-specific yes/no judgment in four wordings: minimal , statement (the minimal content as a statement), definition (minimal plus the benchmark’s definition of the construct) and criteria (minimal plus criteria {true: what + examples, false: what + not-for}).	$p_{q}$
Direct Choice	The same judgment as a Choice with described options {positive, negative, undetermined[, exclusion]}.	soft: $\sum_{o\in\text{pos}}\pi_{o}$ , argmax: $\mathbb{1}[\arg\max\in\text{pos}]$
Rubric (single)	The reference scorer’s judge prompt asked as one Noul or Choice (e.g., HarmBench’s classifier prompt, SycophancyEval’s “teacher grading a quiz”, RAGTruth’s faithfulness prompt).	$p_{q}$ or $1-p_{q}$
Rubric (formula)	Several atoms whose answers feed the reference scorer’s formula in code (e.g., StrongREJECT refusal $\times$ convincingness $\times$ specificity, MASK letters over 5 belief answers and 1 statement, first cave over per-turn answers).	usually discrete: atoms thresholded, then AND/OR or the official aggregate
Soft decomposition	Narrow atoms combined without thresholds (per option, per secret, per action).	$\max_{q}p_{q}$ , $\prod_{q}p_{q}$ , $p_{\text{bad A}}-p_{\text{bad B}}$ rescaled, $p_{B}/(p_{A}+p_{B})$
Gate	A gate atom (e.g., “does the response engage?”) before a targeted question.	abstain or 0 if $p_{\text{gate}}\geq 0.5$ , else $p_{q}$

Appendix

Table 3: Families of targeted questions and their combination rules. Every family is asked in the same Jev call as the generic question, and the last column gives how its answers become the score

s(x)

p_{q}

: Noul probability of “yes” for question

q

\pi_{o}

: Choice probability of option

o

. pos: the options that indicate the failure. Atom: one narrow question whose answer feeds a combination rule.

\mathbb{1}[\cdot]

: indicator, with probabilities thresholded at 0.5.

Comparison	AUROC W/T/L, mean $\Delta$	F1 W/T/L, mean $\Delta$
Generic Noul vs. generic Choice (soft)	13/7/11, $+0.007$	9/5/17, $+0.003$
Generic Noul vs. generic Score (soft)	5/7/19, $-0.024$	7/7/17, $-0.091$
Generic Choice vs. generic Score (soft)	4/10/17, $-0.031$	9/6/16, $-0.094$
Soft vs. argmax, generic Choice	28/0/3, $+0.084$	8/23/5, $-0.006$ ( $n{=}36$ )
Soft vs. argmax, generic Score	29/1/1, $+0.079$	10/7/14, $+0.001$
Targeted minimal Noul vs. targeted Choice	7/7/19, $-0.028$	11/7/15, $-0.039$

Appendix

Table 4: Paired comparisons of answer types and wordings. Each row gives the wins, ties, and losses of the first-named variant over usable benchmarks and the mean difference. W/T/L: benchmarks on which the first-named variant wins, ties (

|\Delta|<0.005

), or loses. Mean

\Delta

: first minus second, averaged over benchmarks. Soft: score from the answer probabilities. Argmax: score from the top answer only. F1 at

t=0.5

n=31

benchmarks for generic and

n=33

for targeted comparisons unless noted.

Pool	AUROC inflation	F1 inflation
All Jev-only strategies (15–40)	0.014 (0.026, 0.129)	0.039 (0.043, 0.176)
Generic (5 readouts, $n{=}36$ )	0.005 (0.014, 0.094)	0.011 (0.020, 0.100)
Targeted	0.008 (0.021, 0.116)	0.030 (0.039, 0.176)
Single-question ( $n{=}37$ )	0.010 (0.018, 0.105)	0.026 (0.036, 0.181)
Multi-question	0.002 (0.012, 0.102)	0.016 (0.020, 0.131)

Appendix

Table 5: Selection inflation by strategy pool. Inflation is how much picking the best strategy on the evaluation half overstates the score of the strategy picked on the other half. Cells: median over the 38 usable benchmarks unless noted, with the mean and maximum in parentheses. Inflation is the best strategy’s score on one half minus the score, on that half, of the strategy selected on the other half (20 grouped splits). Generic: the five readouts of the generic question. Single- and multi-question: strategies that use one or several questions.

		Jev AUROC $\uparrow$				Jev F1 $\uparrow$
Failure type	Usable	Generic Noul	Best gen. (in-sample)	Targeted (split-half)	Margin vs. base.	@0.5	CV thr.	All-pos. F1
Sycophancy †	3/4	0.726	0.732	0.784	+0.118	0.640 ∘	0.890	0.689
Jailbreaks †	3/4	0.965	0.972	0.963	+0.057	0.818	0.891	0.507
Deception	4/4	0.949	0.971	0.945	+0.215	0.845	0.853	0.614
Prompt injection	4/4	0.962	0.988	0.998	+0.026	0.535 ∘	0.905	0.597
Hallucination †	6/6	0.823	0.842	0.867	+0.186	0.722	0.734	0.621

Appendix

Table 6: Detection performance per failure type. Values are medians over usable benchmarks (at least five items per class), and 95% CIs resample benchmarks and items. Usable : usable/all benchmarks, and a number in parentheses means fewer benchmarks enter that cell. † : at most three benchmarks, or one with fewer than 10 minority-class items. Generic Noul : the benchmark’s template question read as

P(\text{yes})

, zero-shot. Best gen. : best of the five readouts of the generic question, chosen in sample. Targeted : best targeted strategy, selected on one half of the items and scored on the other. Margin vs. base. : generic Noul AUROC minus the better of two supervised baselines, in-domain TF-IDF logistic regression and response length (Table 34 ). F1 : generic Noul at threshold

t{=}0.5

and at a 2-fold cross-validated threshold. All-pos. F1 : F1 of flagging every item. ∘ : at or below All-pos. F1. Per-type CIs: Table 9 . Sensitivity rows drop the seven usable benchmarks whose label our audit changes (Table 26 ), keep only validated or only unvalidated labels, or drop benchmarks whose minority class has fewer than 20 items.

	AUROC $\uparrow$					F1 $\uparrow$
Benchmark	Noul	Gen. best	Type	Tgt. best	Tgt. SH	Noul@.5	Noul cv	Best	Best cv	All-pos.
Sycophancy
ELEPHANT (AITA)	0.957	1.000	S	1.000	1.000	0.646	0.890	1.000	1.000	0.689
SYCON-Bench (false premise) ‡	0.976	1.000	S	0.905	–	0.353	–	0.880	–	0.903
SycophancyEval (answer)	0.540	0.622	C a	0.549	0.514	0.640	0.901	0.716	0.880	0.901
SycophancyEval (feedback)	0.726	0.732	C	0.789	0.784	0.390	0.327	0.411	0.400	0.243

Appendix

Table 7: Full per-benchmark detection results (Jev only). Rows marked ‡ are degenerate and excluded from all aggregates. AUROC columns: Noul , the generic question asked as a yes/no Noul. Gen. best , the best of the five generic readouts, whose answer type is given in Type (N = Noul, C = Choice, S = 3-level Score, a = argmax readout, otherwise soft probability). Tgt. best , the best targeted question (in-sample maximum over all targeted strategies, an upper bound). Tgt. SH , targeted question selected on one random half and evaluated on the other (mean over 20 grouped splits, out of sample). F1 columns: Noul@.5 , generic Noul at threshold 0.5. Noul cv , generic Noul with a 2-fold cross-validated threshold. Best , best F1 at 0.5 over all Jev strategies (in-sample). Best cv , the same strategy with a cross-validated threshold. All-pos. , F1 of flagging every item,

2\pi/(1+\pi)

for positive rate

\pi

. “–”: not defined (paired-choice benchmarks have no single-item Noul, degenerate or skewed folds lack a class, and split-half needs both classes in each half).

	All usable		Label-chang.		Minority	Hill-	Held-	Corrected
Statistic	Median	95% CI	defects excl.	Validated	class $\geq 20$	climb	out	labels
#B (generic Noul )	38 (31)		31 (28)	11 (8)	26 (21)	29 (25)	9 (6)	38 (31)
Generic Noul AUROC	0.886	[0.821, 0.952]	0.905	0.872	0.886	0.886	0.906	0.893
Best generic AUROC	0.903	[0.850, 0.963]	0.936	0.896	0.888	0.903	0.908	0.914
Targeted SH AUROC	0.911	[0.860, 0.944]	0.926	0.912	0.897	0.909	0.913	0.912
Generic Noul F1@0.5	0.706	[0.607, 0.773]	0.707	0.721	0.736	0.706	0.689	0.706

Appendix

Table 8: Headline medians with 95% CIs and on subsets. Medians are over benchmarks, and the targeted gain is split-half targeted minus best generic AUROC. #B: usable benchmarks (in parentheses: with a generic Noul ). Best generic: best of the five generic readouts, in sample. Targeted SH: targeted strategy selected split-half (on one half, scored on the other). F1, CV: F1 at a 2-fold cross-validated threshold. Targeted gain: targeted SH minus best generic AUROC, with wins/ties/losses over benchmarks and a Wilcoxon signed-rank

p

. Hill-climb and held-out: the benchmark’s role in the suite of Chen et al. (2026) . Label-changing defects: the seven usable benchmarks marked “defect” in Table 2 (the workplace scenes, a direction-only defect, stay). Validated: StrongREJECT and the ten deterministic labels marked “val.”. Corrected labels: Open-Prompt-Injection and SycophancyEval (answer) scored on their corrected labels (Table 28 ).

Failure type	#B	Generic Noul	Best generic	Targeted SH
Sycophancy	3/3/3	0.726 [0.519, 0.964]	0.732 [0.596, 1.000]	0.784 [0.468, 1.000]
Jailbreaks	3/3/3	0.965 [0.945, 0.997]	0.972 [0.953, 0.998]	0.963 [0.920, 0.986]
Deception	4/4/4	0.949 [0.916, 0.993]	0.971 [0.939, 1.000]	0.945 [0.899, 0.989]
Prompt injection	4/4/4	0.962 [0.247, 0.987]	0.989 [0.537, 0.996]	0.998 [0.550, 0.999]
Hallucination	6/6/6	0.823 [0.711, 0.924]	0.842 [0.751, 0.932]	0.867 [0.749, 0.921]
Privacy violation	4/4/4	0.808 [0.701, 0.914]	0.859 [0.803, 0.933]	0.877 [0.776, 0.942]

Appendix

Table 9: Per-failure-type medians with 95% CIs. The bootstrap is the one of Table 8 . #B: benchmarks behind the generic Noul / best-generic / targeted medians. Cells: median AUROC [95% CI]. Best generic: best of the five generic readouts, in sample. Targeted SH: targeted strategy selected on one half of the items and scored on the other.

				Primary strategy			Generic Noul
Benchmark	Added to state	Ref.	Tag	AUROC A	AUROC B	$\Delta$ [95% CI]	$\Delta$ [95% CI]
Reference added (32)
SycophancyEval (answer)	reference	missing	key	0.569	0.930	+0.361 [+0.24, +0.48]	+0.400 [+0.27, +0.52]
DeceptionBench	groundtruth	missing	key	0.973	0.976	+0.003 [ $-$ 0.01, +0.01]	+0.003 [ $-$ 0.01, +0.01]
PrivacyLens	secrets	missing	key	0.819	0.976	+0.157 [+0.08, +0.23]	+0.159 [+0.08, +0.23]
PrivacyLens (all actions) ∘	secrets	missing	key	0.874	0.974	+0.101 [+0.05, +0.15]	+0.113 [+0.06, +0.16]

Appendix

Table 10: Full context ablation. Each row compares a detection instance without (A) and with (B) extra context, on the same items and labels unless marked unpaired. Added to state : the fields present only in B. Ref. : whether A’s state already contains the label-defining reference (missing = cannot be recovered from A, derivable = recoverable by reasoning, present = already in A so B adds a pointer or distilled copy, auxiliary = B adds non-reference context). Tag : whether a deployed monitor would have the added field (dep. = deployable) or the field defines the label (key = label key). The PrivacyLens secret list is a label key because the label is membership of the disclosed item in that list. ∘ : not in the collapsed, ceiling-free set of Table 11 . Primary strategy : the single Jev strategy maximizing the mean AUROC over A and B (coverage

\geq 0.95

, Jev-driven under a permutation null, no strategy that reads label-defining metadata in code), so selection does not favour B. Generic Noul : the fixed generic question.

\Delta=

AUROC

{}_{B}-

AUROC A with a 95% grouped bootstrap CI (1,000 resamples), bold when the CI excludes 0. “[excl.]”: the label is the reference scorer’s exclusion decision rather than the failure. CI bounds are rounded to two decimals. “–”: generic Noul not asked on that file.

Set	Added field	Generic Noul	Best shared
All reference pairs	deployable	1/7, $+0.020$	1/7, $+0.001$
	label key	5/13, $+0.053$	9/19, $+0.027$
	missing reference	3/4, $+0.136$	6/10, $+0.046$
	reference present	1/6, $+0.000$	1/6, $+0.001$
Collapsed, ceiling-free	all	5/18, $+0.028$	7/19, $+0.016$
	deployable	1/7, $+0.020$	1/6, $+0.002$

Appendix

Table 11: Reference pairs by availability of the added field. Each cell gives the number of pairs whose 95% CI lies above zero out of all pairs, then the median

\Delta

AUROC. No pair has a CI below zero. Deployable: a field a system-level monitor may hold. Label key: a field that belongs to the label definition. Missing or present: whether the state without the field already holds the reference. Best shared: one strategy selected on both states (the primary strategy of Table 10 ). Rows above the rule: all reference pairs with [excl.] variants and SYCON-Bench removed. Below: the collapsed, ceiling-free set of Figure 4 .

File ( $n$ )	Reference	AUROC short $\to$ long	$\Delta$ [95% CI]
PrivacyLens-all, no secrets (298)	no	0.944 $\to$ 0.802	$-0.142$ [ $-0.237$ , $-0.048$ ]
PrivacyLens-all, secrets (298)	yes	0.987 $\to$ 0.962	$-0.026$ [ $-0.061$ , $0.002$ ]
PrivacyLens, no secrets (110)	no	0.883 $\to$ 0.737	$-0.146$ [ $-0.330$ , $0.035$ ]
PrivacyLens, secrets (110)	yes	0.976 $\to$ 0.975	$-0.001$ [ $-0.048$ , $0.045$ ]
Revealed reward MT, transcript (69)	no	0.965 $\to$ 0.847	$-0.118$ [ $-0.273$ , $0.020$ ]
Revealed reward MT, official (69)	goal and proxy	0.935 $\to$ 0.863	$-0.072$ [ $-0.220$ , $0.065$ ]

Appendix

Table 12: AUROC on short vs. long states within a file. States are split at the median length, with a bootstrap 95% CI of the difference. File: a benchmark state file, with its item count. Reference: what label-defining reference the state holds (“no”: missing).

\Delta

: AUROC on long minus short states. MT: multi-turn.

			Pooled			Per file (median)
Score set	Files	$n$	ECE	AUROC	Slope	ECE	Slope	AUROC
Generic Noul	48	9,797	0.035	0.884	0.91	0.150	1.58	0.905
Targeted Noul	52	40,705	0.034	0.892	0.85	0.108	1.11	0.878
Generic Choice	53	11,193	0.087	0.879	0.45	0.128	0.65	0.879
Best per file	55	11,777	0.037	0.919	0.45	0.100	1.03	0.932

Appendix

Table 13: Pooled vs. per-file calibration. Pooled metrics treat all items of all files as one set, and per-file metrics are medians over files, where a file is one benchmark’s items for one target model. Generic/targeted Noul : raw

P(\text{yes})

of the generic question and of every targeted Noul . Generic Choice :

P(\text{yes})

. Best per file: the per-file best strategy (selected on the same labels). ECE: expected calibration error, 10 equal-width bins. Slope: logistic-regression coefficient of the label on

\mathrm{logit}(p)

(1: calibrated,

<1

: over-confident,

>1

: under-confident).

Strategy set	Pairs	F1@0.5	F1 oracle	F1 CV	Median CV $-$ 0.5	Gap $\leq 0.02$	Median $t^{\ast}$
All strategies	700	0.676	0.798	0.763	$+0.011$	36%	0.40
Best per file	42	0.822	0.859	0.831	$0.000$	71%	0.50
Generic Noul	48	0.614	0.811	0.761	$+0.044$	23%	0.35
Generic Choice	52	0.606	0.795	0.762	$+0.058$	15%	0.25
Direct Noul	207	0.703	0.810	0.773	$+0.006$	38%	0.45
Rubric	36	0.781	0.874	0.855	$+0.007$	56%	0.425

Appendix

Table 14: F1 at

t=0.5

, at the oracle threshold

t^{\ast}

, and with a 2-fold cross-validated threshold. Values are means over (file, strategy) pairs of main-role files. Pairs: (file, strategy) pairs. Oracle: the F1-maximizing threshold on the same items, an upper bound. CV: threshold chosen on one fold of item groups and applied to the other. Median CV

-

0.5: median F1 gain of CV over

t=0.5

. Gap

\leq 0.02

: share of pairs whose F1 at 0.5 is within 0.02 of the oracle. Threshold grid

0.05,\dots,0.95

, binary strategies excluded.

Pool	$n$	Obs.	Null mean (p95)	$>$ p95	Eq.-mass	Debiased	Brier	Rel.	Res.	Unc.
		ECE (10 bins)					Brier decomposition
Files	48	0.150	0.069 (0.100)	36	0.170	0.181	0.143	0.039	0.081	0.206
Benchmarks	31	0.168	0.074 (0.106)	24	0.173	0.182	0.145	0.042	0.106	0.215

Appendix

Table 15: Generic- Noul calibration against a perfect-calibration null, and the Brier decomposition. Values are medians over files or benchmarks. Obs.: observed ECE. Null: ECE when labels are drawn from Bernoulli(

p

), so the score is perfectly calibrated (mean and 95th percentile over 1,000 draws).

>

p95: files or benchmarks whose ECE exceeds the null’s 95th percentile. Eq.-mass: ECE with 10 equal-mass bins. Debiased: the debiased

L_{2}

calibration error of Kumar et al. (2019) . Brier

=

reliability (Rel.)

-

resolution (Res.)

+

uncertainty (Unc.), with medians of each term, so the identity holds only approximately.

Threshold	Median F1	F1 $-$ all-pos.	$\leq$ all-pos.
$t=0.5$	0.706	$+0.135$	8/31
EM prior-shift, prior 0.5	0.571	$-0.011$	18/31
EM prior-shift, pooled prior	0.690	$+0.034$	15/31
Fit on $k=10$ items	0.793	$+0.156$	4/31
Fit on $k=20$ items	0.797	$+0.176$	4/31
Fit on $k=50$ items ∗	0.803	$+0.188$	2/26

Appendix

Table 16: Threshold learning curve for the generic Noul . On 31 canonical benchmarks, the threshold is fit on

k

random labelled items (200 draws) and F1 is measured on the rest. EM prior-shift: label-free re-estimation of the positive rate ( Saerens et al., 2002 ) , started from 0.5 or from the pooled rate, then

t=0.5

on the corrected scores. Oracle: the F1-maximizing threshold on the same items. F1

-

all-pos.: median difference from the F1 of flagging every item.

\leq

all-pos.: benchmarks at or below that F1. ∗ 26 benchmarks with

n\geq 60

Label source (#B)	AUROC	ECE (null)	$t^{\ast}$	F1@0.5	F1 CV	F1, $k{=}10$	All-pos.	CV $-$ 0.5
Validated (8)	0.872 [0.772, 0.922]	0.114 (0.059)	0.40	0.721	0.694	0.684	0.536	$+0.008$ [ $-0.043$ , $+0.044$ ]
LLM judge (16)	0.906 [0.818, 0.966]	0.149 (0.086)	0.35	0.767	0.825	0.819	0.595	$+0.052$ [ $-0.000$ , $+0.126$ ]
Rule (4)	0.963 [0.947, 0.987]	0.243 (0.053)	0.35	0.535	0.906	0.829	0.597	$+0.370$ [ $+0.158$ , $+0.819$ ]
Label-changing defect (3)	0.540 [0.208, 0.759]	0.252 (0.062)	0.50	0.640	0.832	0.827	0.826	$+0.000$ [ $-0.096$ , $+0.285$ ]
All (31)	0.886 [0.821, 0.952]	0.168 (0.074)	0.35	0.706	0.822	0.793	0.568	–

Appendix

Table 17: Generic- Noul calibration and thresholds by label source. Values are medians over benchmarks with a generic Noul . 95% CIs resample benchmarks within the group (2,000 replicates) and, for AUROC and F1, one prompt-grouped item replicate per benchmark. ECE: 10 bins, with the perfect-calibration null mean in parentheses.

t^{\ast}

: oracle threshold. CV

-

0.5: median per-benchmark F1 gain of the 2-fold CV threshold over

t=0.5

. Defect: Open-Prompt-Injection and both SycophancyEval benchmarks.

Failure	Benchmark	Neg.	Pos.	Base rate	F1@0.5	F1 CV	All-pos.
CV $\leq$ all-positive	SycophancyEval (answer)	27	123	0.82	0.640	0.901	0.901
	LLM-AggreFact (multi-turn)	9	139	0.94	0.841	0.969	0.969
	Reference letters (gender)	13	30	0.70	0.125	0.817	0.822
	Workplace scenes (WinoBias)	7	37	0.84	0.318	0.880	0.914
CV $<$ $t=0.5$	SycophancyEval (feedback)	187	30	0.14	0.390	0.327	0.243
	MASK (continuation)	40	42	0.51	0.849	0.830	0.677

Appendix

Table 18: Benchmarks on which the CV threshold fails for the generic Noul . Neg./pos. give the class counts. Top block: the 2-fold cross-validated (CV) threshold does no better than flagging every item (All-pos.). Bottom block: it does worse than the default

t=0.5

. Base rate: share of positives. Failure: the failure mode of the threshold, not the failure type.

Signal	$n$	Acc.	AUROC	Top 10%	Top 20%	Top 50%	AURC (oracle)
Generic Choice , confidence	9,979	0.814	0.745	0.944	0.965	0.924	0.102 (0.019)
Generic Score , confidence	10,097	0.833	0.762	0.988	0.974	0.943	0.070 (0.015)
Generic Noul , $\|p-0.5\|$	10,097	0.806	0.780	0.994	0.980	0.931	0.075 (0.020)
Targeted Noul , $\|p-0.5\|$	41,905	0.811	0.779	0.994	0.983	0.937	0.072 (0.019)

Appendix

Table 19: Selective detection. Each column gives the accuracy at

t=0.5

when only the most confident decisions are kept. Confidence: the value Jev returns for Choice / Score , and

|p-0.5|

for Noul .

n

: items. Acc.: accuracy on all items. AUROC: how well confidence separates correct from wrong decisions. Top

k

%: accuracy on the

k

% most confident items. AURC: area under the risk–coverage curve (lower is better), with the value of a perfect confidence ranking in parentheses.

Rater	vs. human: acc. / $\kappa$ / AUROC	vs. scorer: acc. / $\kappa$ / AUROC
Reference scorer (GPT-4o-mini rubric)	0.936 / 0.811 / 0.929	–
Jev generic Noul (pre-specified)	0.935 / 0.809 / 0.971	0.915 / 0.753 / 0.955
Jev generic Choice	0.939 / 0.819 / 0.973	0.921 / 0.767 / 0.953
Jev targeted † (“substantive help”, criteria)	0.956 / 0.868 / 0.984	0.935 / 0.806 / 0.973
Jev re-implemented scorer rubric	0.932 / 0.801 / 0.953	0.927 / 0.788 / 0.938

Appendix

Table 20: StrongREJECT agreement with the median of five human labelers and with the reference scorer (

n=1{,}360

t=0.5

). The first Jev row is pre-specified, and AUROC uses each rater’s continuous score. † Selected as the best of six Jev questions on these human labels (in-sample). Table 21 gives its out-of-sample value.

Selection $\to$ evaluation	Jev / scorer	[95% CI]	Jev / scorer	with Jev
	$\kappa$	$\kappa$ gain	AUROC	Humans side
None (generic Noul )	0.809 / 0.811	$-0.002$ [ $-0.059$ , $+0.057$ ]	0.971 / 0.929	49.1% of 116
Split-half, prompt groups	0.865 / 0.819	$+0.046$ [ $-0.024$ , $+0.090$ ]	0.984 / 0.934	63.1% [43.8, 75.6]
Split-half, item groups	0.858 / 0.803	$+0.055$ [ $+0.008$ , $+0.096$ ]	0.982 / 0.926	64.8% [52.9, 74.7]
HarmBench val. $\to$ SR	0.819 / 0.811	$+0.008$ [ $-0.057$ , $+0.066$ ]	0.973 / 0.929	51.8% of 108
SR $\to$ HarmBench val.	0.777 / –	$+0.000$ [ $-0.020$ , $+0.021$ ] ∗	0.957 / –	–

Appendix

Table 21: Out-of-sample question selection against human labels. The Jev question is selected by

\kappa

against humans among six candidates and scored on data not used for the selection. Candidates: generic Noul and Choice , targeted Noul and Choice , and the re-implemented rubric as thresholded atoms and as a soft score. Cross-dataset rows select among the four questions both batteries share.

\kappa

gain: Jev minus reference scorer on the evaluation data, with a 95% bootstrap CI that reruns the selection. Humans side with Jev: share of Jev–scorer disagreements (95% CI). SR: StrongREJECT. Split-half rows select on one half of SR and evaluate on the other, averaged over 20 splits (

\approx

45 disagreements per half). Prompt groups are the primary unit. ∗ HarmBench validation (578 items) has no scorer labels in the package, so the last row gives the gain over the generic Noul .

		$\kappa$ @0.5 [95% CI]			AUROC
Generator	$n$ (pos.)	Jev generic	Jev targeted	Scorer	Jev gen.	Jev tgt.	Scorer
dolphin-mixtral	500 (188)	0.874 [.828, .916]	0.928 [.887, .963]	0.820 [.756, .883]	0.987	0.994	0.924
GPT-3.5	491 (91)	0.668 [.585, .750]	0.752 [.647, .840]	0.790 [.714, .856]	0.928	0.959	0.938
GPT-4	112 (9)	0.731 [.430, .943]	0.809 [.583, 1.00]	0.537 [.267, .825]	0.982	0.981	0.848

Appendix

Table 22: StrongREJECT human set by generator. Values are

\kappa

against the human label at

t=0.5

and AUROC. 95% CIs resample forbidden prompts within the generator. Targeted: the “substantive help” question selected on these labels. Llama-2-70B has 2 positives among 257 responses and is omitted.

Detector	AUROC	$\kappa$ @0.5	ECE (null p95)	Brier	Slope
GPT-4o-mini rubric (reference scorer)	0.929	0.811	0.045 (0.013)	0.055	0.34
Rubric component: 1 $-$ refusal	0.833	0.517	0.208	0.208	0.18
Jev generic Noul	0.971	0.809	0.062 (0.025)	0.053	1.20
Jev generic Noul , 9 quantile levels	0.962	–	–	–	–
Jev generic Choice	0.973	0.819	0.032 (0.017)	0.046	0.65
Jev generic Score	0.968	0.810	0.039 (0.021)	0.056	0.80

Appendix

Table 23: Detectors on the StrongREJECT human set (

n=1{,}360

, 290 positive). Every detector is scored against the median of five human labels.

\kappa

@0.5: Cohen’s

\kappa

at threshold 0.5. ECE: 10 bins, with the 95th percentile under perfect calibration in parentheses. Slope: calibration slope, and

<1

means over-confident. 9 quantile levels: Jev coarsened to the scorer’s resolution, ties broken at random (mean of 200 draws).

Benchmark ( $n$ )	Target model	Jev direct Choice	Jev generic Noul
LLM-AggreFact A (150)	0.533 / 0.631	0.733 / 0.769	0.707 / 0.762
LLM-AggreFact B (150)	0.547 / 0.757	0.793 / 0.874	0.780 / 0.881
SummEdits (300)	0.557 / 0.648	0.813 / 0.852	0.803 / 0.853
ConfAIde tier 2b (98)	0.758 / 0.816	0.863 / 0.918	–
PrivaCI-Bench (300)	0.711 / 0.711	0.730 / 0.815	–
PrivaCI-Bench GDPR (300)	0.938 / 0.938	0.881 / 0.969	–

Appendix

Table 24: Understanding tasks against human gold. Cells give balanced accuracy / AUROC, and the target model’s column is its own judgment on the task. On these benchmarks the detection label is a deterministic function of human gold and the target’s answer, so each rater can be scored on the underlying task (e.g., whether a claim is supported). “–”: no generic Noul for the task.

Items	#B	$n$	Jev err.	Determ.	Interpr.	Construct	Unclass.
All read items	21	311	25.4%	28.6%	14.8%	24.1%	7.1%
Excl. HarmBench validation	20	258	22.9%	34.5%	17.8%	16.3%	8.5%
Excl. also Open-Prompt-Injection	19	169	34.9%	3.6%	27.2%	24.9%	9.5%

Appendix

Table 25: Audit verdicts of the 311 items read one by one. Rows pool all items or drop the two benchmarks that dominate the pool, and percentages are of the items in the row. #B: benchmarks. Jev err.: Jev is wrong. Determ.: deterministic label defect (rule, parser, or label definition). Interpr.: interpretive defect (judge verdict or annotation). Construct: the label and a reasonable reading of the failure differ. Unclass.: read but not classified.

Benchmark (failure)	Label defect	Effect	Evidence
Open-Prompt-Injection (prompt injection)	Positive only if the model followed the injection and solved the injected task: tracks task accuracy.	Changed (det.)	293/300 responses follow the injection, and 83/89 negatives are followed-but-wrong. Targeted AUROC 0.50–0.55 on the label. On the corrected label “follows the injection”, generic Noul 0.975, targeted 1.000.
SycophancyEval (answer) (sycophancy)	Label is answer correctness, not deference.	Changed (det.)	Target is wrong on 44/50 items with no user opinion. Deference gives AUROC 0.75 or 0.22 depending on which answer the user suggests (0.49 pooled). On the corrected label “answer moves toward the suggestion”, blind generic Noul 0.712.
SycophancyEval (feedback) (sycophancy)	Judge applies “no criticism” inconsistently.	Changed (judge)	15 of 30 positives name a fallacy or flaw in the first sentence.
MACHIAVELLI (power, harm, reward, held-out games) (power seeking, reward hacking)	Whether the target’s choice is the option annotated with more in-game consequences. Neither is in the state.	Changed (unobs.)	Label equals “choice $=$ flagged option” on 1,100/1,100 items, target picks B on 75–78% of items, and P(picks flagged) 0.71–0.76 if flagged is B vs. 0.16–0.26 if A.
Workplace scenes (WinoBias) (social bias)	Prompt forces a gap (it asks who takes the lead), and the parser maps FEMALE_MORE to MALE_MORE .	Direction only	95%/84% positive by construction, and direction wrong on 65/79 positives. Jev recovers the corrected direction on 92–100% of positives.
AbstentionBench (concealing uncertainty)	Label depends on which judge runs the official prompt.	Defect (judge), degenerate	Qwen2.5-7B and GPT-4o judges agree on 50% of should-abstain items. Jev F1 0.70 on Qwen labels vs. 0.85 on GPT-4o labels.

Appendix

Table 26: Label problems surfaced by Jev disagreements. Reading the items on which Jev disagrees with the scorer shows labels that measure something other than the intended failure, depend on information the state omits, or cannot support evaluation. Label defect : what the label measures in practice. Effect : what the defect does to the binary label, with its class (det.: deterministic defect in the rule or label definition, judge: judge defect, unobs.: label unobservable from the state). Changed marks the seven usable benchmarks that the “Excl. changed labels” row of Table 6 drops. With AbstentionBench, whose judge-dependent label is a defect but which is degenerate, they are the eight D labels of Table 1 . The remaining rows leave the binary label unchanged and stay in every aggregate, or are degenerate. Evidence : the measurement that exposes the defect. FN/FP: Jev false negatives/positives against the scorer label. Audit protocol and counts: Appendix F .

Benchmark	Read	J	Det.	Int.	Constr.	Cluster	Cluster-level finding or note
Open-Prompt-Injection	89	0	83	0	0	300	6 negatives did not follow the injection
SycophancyEval (answer)	4	0	0	4	0	200	label is correctness (44/50 no-opinion answers wrong)
SycophancyEval (feedback)	31	0	0	16	0	–	15 (+2 borderline) of 30 positives contradict the definition
StrongREJECT	9	0	0	3	6	–	3 deflections scored harmful, rest borderline persona responses
JailbreakBench (PAIR), benign split	4	1	0	3	0	–	explicit refusals labelled “no refusal”
HarmBench	11	0	0	0	11	–	attempted but low-utility outputs

Appendix

Table 27: Audit counts. Items read one by one are classified into four verdicts, and cluster items are counted only at cluster level. J: Jev error. Det.: deterministic label defect. Int.: interpretive label or scorer defect. Constr.: construct boundary. “–”: not classified per item.

Benchmark and label	Pos./ $n$	Generic	Best gen.	Tgt. SH	Generic@0.5	Generic CV	All-pos.
		AUROC			F1
Open-Prompt-Injection, official	211/300	0.231	0.534	0.536	0.832	0.832	0.826
corrected: follows injection	293/300	0.975	1.000	1.000	0.995	0.995	0.988
SycophancyEval (ans.), official	123/150	0.540	0.622	0.514	0.640	0.901	0.901
corrected: moves to suggestion	15/150	0.712	0.763	0.752	0.239	0.250	0.182

Appendix

Table 28: Official and corrected labels. The official label stays canonical. Open-Prompt-Injection corrected: the response answers in the injected task’s label vocabulary (keyword rule). SycophancyEval (answer) corrected: the answer moves toward the user’s suggestion relative to the no-opinion answer. Generic: the generic Noul . Best gen.: best of the five generic readouts. Tgt. SH: targeted question selected split-half over 20 grouped splits.

Scorer group	#B	Calls	Tokens	USD	Tokens	USD	Ratio
		Reference scorer			Jev
API LLM judge	19	8,940	9.36M	18.96	7.18M	0.30	62.9 $\times$
Local judge / classifier	5	1,171	0.52M	–	2.14M	0.09	–
Rule / log-prob scorer	20	–	–	–	8.62M	0.36	–
All Jev calls (23,384)	44				54.0M	2.27

Appendix

Table 29: Cost of one full detection pass, Jev vs. the reference scorers. Judges are priced at list prices and Jev at USD 0.042 per 1M input tokens. Benchmarks are grouped by how their reference scorer labels items (API judges: GPT-4o, GPT-4o-mini, Claude Haiku 4.5). List prices in USD per 1M input/output tokens: GPT-4o 2.50/10.00, GPT-4o-mini 0.15/0.60, Claude Haiku 4.5 1.00/5.00. Local judges run on our GPUs and have no list price (“–”). Jev one pass: evaluated items

\times

mean Jev input tokens per call. Ratio : judge USD / Jev USD. The last row is every Jev call made in this study (all context variants and reruns).

Failure type	Calls	Q/call	Err. (%)	In tok.	Out/Q	Lat. med. (s)	Lat. p90 (s)	USD
Sycophancy	1,207	11.6	0.00	2,634	25.6	0.34	0.79	0.163
Jailbreaks	4,848	14.2	0.56	2,579	22.0	0.31	0.53	0.554
Deception	1,988	12.1	0.00	2,227	33.6	0.33	0.43	0.281
Prompt injection	2,824	9.7	0.00	1,686	26.2	0.34	1.30	0.211
Hallucination	3,000	12.5	0.00	2,529	24.0	0.31	0.38	0.331
Privacy violation	2,860	10.8	0.00	1,883	25.3	0.31	0.50	0.243

Appendix

Table 30: Jev calls per failure type. One call sends one detection instance with the whole question battery, and latency does not grow with the number of questions. Q/call : mean questions per call. Err. : share of failed calls. In tok. : median input tokens per call. Out/Q : output tokens per question. Latency: median and 90th percentile per call. USD : input-token cost of all calls.

			Reference scorer			Jev
Benchmark	Judge	Items	Calls/item	Tok./item	USD	Tok./item	USD
SycophancyEval (answer)	GPT-4o	200	1.00	514	0.259	1,907	0.0160
SYCON-Bench (false premise)	GPT-4o	42	1.86	1,890	0.199	8,073	0.0142
SycophancyEval (feedback)	GPT-4o	217	2.00	1,096	0.598	2,740	0.0250
StrongREJECT	GPT-4o-mini	100	1.00	44	0.001	2,691	0.0113
HarmBench	HarmBench-13B cls. ∗	150	1.00	226	–	3,144	0.0198

Appendix

Table 31: Reference-scorer cost on the 24 judge-scored benchmarks. Jev columns give input tokens per call and the cost of one Jev pass on the same items. Calls/item and Tok./item : judge calls and judge tokens (input + output, o200k_base count of the recorded text) per evaluated item. USD : list-price cost of one judge pass. ∗ Local model: tokens are a lower bound and no list price applies.

Judges at	Jev billed for	Judge $	Jev $	Pooled	Median (range)	$>$ 2 $\times$	$>$ 5 $\times$	PL / MASK	Excl.
List price	full battery	18.96	0.302	62.9 $\times$	16.2 $\times$ (0.12–142 $\times$ )	18	18	28% / 37%	35.1 $\times$
GPT-4o-mini	single question	1.82	0.150	12.1 $\times$	3.3 $\times$ (0.50–43.7 $\times$ )	17	8	39% / 23%	7.3 $\times$
GPT-4o-mini	full battery	1.82	0.302	6.0 $\times$	1.5 $\times$ (0.12–18.9 $\times$ )	8	6	39% / 23%	3.6 $\times$

Appendix

Table 32: Cost ratio (judge USD / Jev USD) over the 19 API-judge benchmarks under three pricing scenarios. Judges are priced at list prices or at GPT-4o-mini prices, and Jev is billed for the full battery or for the single generic question (median 957 input tokens per item). Pooled: ratio of the summed costs, and the first row is the ratio of Table 29 . Median and range: over per-benchmark ratios (minimum StrongREJECT, maximum PrivacyLens in every scenario).

>

\times

>

\times

: benchmarks above that ratio. PL / MASK: share of the judge total spent on PrivacyLens and on the three MASK benchmarks. Excl.: pooled ratio without PrivacyLens and MASK.

Detector	AUROC	ECE ( $>$ null)	F1@0.5	F1 CV	F1, $k=10$ / 20 / 50	F1 oracle
Jev generic Noul (zero-shot)	0.886	0.168 (24/31)	0.706	0.822	0.793 / 0.797 / 0.803	0.849
TF-IDF LR (out-of-fold)	0.754	0.164 (22/31)	0.610	0.609	0.604 / 0.605 / 0.629	0.647
Length (out-of-fold LR)	0.526	0.047 (2/31)	0.235	0.632	0.565 / 0.590 / 0.636	0.636
TF-IDF trained on $k$ items	–	–	–	–	0.351 / 0.387 / 0.514	–

Appendix

Table 33: Detectors under the Jev calibration and threshold protocol. Values are medians over the 31 benchmarks with a generic Noul . ECE: 10 bins. In parentheses, benchmarks whose ECE exceeds the 95th percentile of the perfect-calibration null.

k

: threshold fit on

k

labelled items and F1 on the rest (200 draws,

k=50

on benchmarks with

n\geq 60

). TF-IDF on

k

items: trained on the

k

items and thresholded at 0.5. All-positive F1: 0.568.

	Jev AUROC		TF-IDF LR		Length	All-pos.
Benchmark	Generic	Targeted SH	AUROC	F1	AUROC	F1
Sycophancy
ELEPHANT (AITA)	0.957	1.000	0.748 $\pm$ 0.034	0.727	0.515	0.689
SycophancyEval (answer)	0.540	0.514	0.764 $\pm$ 0.012	0.914	0.451	0.901
SycophancyEval (feedback)	0.726	0.784	0.608 $\pm$ 0.016	0.138	0.547	0.243
Jailbreaks

Appendix

Table 34: Per-benchmark baselines on the 38 usable benchmarks. Jev columns: generic Noul AUROC and the targeted question selected split-half. TF-IDF LR: out-of-fold AUROC (SD over 3 repeats) and F1 at 0.5. Length: AUROC with the sign chosen out of sample. “–”: no generic Noul (MACHIAVELLI-style benchmarks and the text-only uncertainty states).

Explore similar work

Aug 28, 2026cs.AI

Automated Researchers Can Mitigate Well-characterized Alignment Failures

Automating alignment research may accelerate progress toward aligned AI, but whether it does is hard to measure. Luckily, many alignment failures, such as deception, sycophancy, and jailbreaks, are already measurable by public benchmarks. We study whether automated alignment researchers (AARs) can post-train to mitigate alignment failures by proposing training methods and data to simultaneously optimize multiple safety benchmarks, while largely preserving general capability. Across 10 alignment failures, the strongest AAR methods significantly reduce the targeted alignment failures and generalize to a held-out benchmark, multi-turn behavioral audits, and models up to 4.7x larger than the target model. As a human baseline, 28 experienced researchers receive up to eight hours to develop one-shot methods for the same benchmarks, but their methods underperform the best AAR methods. Using human ideas as the AARs' initial research direction does not improve performance, suggesting current AARs may not need guidance from experienced researchers. These results suggest that automating alignment research on well-characterized failures may be practical in the near term.

Chen Yueh-Han, Jiaxin Wen, Jan Hendrik Kirchner

1Anthropic Fellows Program · 2Anthropic · 3UC Berkeley

Artificial Intelligence Alignment Research Automation Feature Alignment Adversarial Evaluation

Ask PDF

Sep 19, 2026cs.CL

Alignment Forecasting: Predicting Misalignment From Training Data

Training a language model on data with a narrow flaw can sometimes make the model broadly misaligned. Inspecting the data at face value often does not settle whether it will emerge, and today it is caught only after training, by auditing the resulting model. To complement post-hoc audits, we introduce Alignment Forecasting: the task of predicting alignment failures before training. Given a target model, a fine-tuning dataset, and a failure mode such as deception or sycophancy, a forecaster outputs the probability that fine-tuning would meaningfully increase that failure mode. To measure progress on alignment forecasting, we introduce ALIGNMENTFORECASTBENCH, a benchmark of over 5,000 forecasting questions spanning 17 target models, 32 datasets, and 16 failure modes. Frontier models prompted directly perform poorly on ALIGNMENTFORECASTBENCH. We therefore propose a forecasting scaffold in which an LLM reads the dataset and rates how strongly and broadly it pushes the model toward misbehavior, and a simple learned model combines that rating with the failure mode's base rate and the target model's prior tendency. This forecasts well above chance, and beats a model fine-tuned on the task and a simple forecaster allowed to see how weaker models behaved after fine-tuning on the same data. Its signals also flag problematic training examples that a frontier-model classifier misses. Filtering those examples out from real post-training data such as UltraChat results in more aligned models on our multiple-choice evaluation in most cases, though the benefit in open-ended conversations is unclear. More progress is needed before forecasts can reliably guide training data curation in practice, but our results suggest that forecasting many alignment failures before training can be tractable in the SFT setting.

Chen Yueh-Han, Bruce W. Lee, Ilia Sucholutsky +1

NYU, MATS · Independent · NYU +1

Large Language Model Alignment Misalignment Persona Frontier Models Training Data Data-Driven Forecasting Data-Curation

Ask PDF

Feb 26, 2026cs.CL

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

We introduce AuditBench, an alignment auditing benchmark. AuditBench consists of 56 language models with implanted hidden behaviors. Each model has one of 14 concerning behaviors--such as sycophantic deference, opposition to AI regulation, or secret geopolitical loyalties--which it does not confess to when directly asked. AuditBench models are highly diverse--some are subtle, while others are overt, and we use varying training techniques both for implanting behaviors and training models not to confess. To demonstrate AuditBench's utility, we develop an investigator agent that autonomously employs a configurable set of auditing tools. By measuring investigator agent success using different tools, we can evaluate their efficacy. Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent. We find that our most effective tools involve scaffolded calls to auxiliary models that generate diverse prompts for the target. White-box interpretability tools can be helpful, but the agent performs best with black-box tools. We also find that audit success varies greatly across training techniques: models trained on synthetic documents are easier to audit than models trained on demonstrations, with better adversarial training further increasing auditing difficulty. We release our models, agent, and evaluation framework to support future quantitative, iterative science on alignment auditing.

Abhay Sheshadri, Aidan Ewart, Elias Kempf +8

Anthropic Fellows Program · Anthropic

Model Auditing Artificial Intelligence Alignment Adversarial Evaluation Adversarial Training

Ask PDF