Training Alignment Auditors via Reinforcement Learning

Paul Rosu; Rowan Wang

cs.AIAug 26, 2026

Training Alignment Auditors via Reinforcement Learning

Authors: Paul Rosu, Rowan Wang

Organizations: Anthropic Fellow · Anthropic

Abstract

Alignment auditing of frontier models increasingly relies on LLM auditors to surface undesirable behaviors at scale, but current automated auditors can struggle with coherent investigation and audit realism. In this work, we improve LLM auditors with reinforcement learning. In our best training environment, the policy investigates target models that potentially possess hidden behaviors planted via their system prompt. An LLM judge, which knows whether the target has a hidden behavior, holistically compares the policy's investigation to a reference investigation to determine the reward. With systematic ablations, we find that pairwise rewards yield more robust training compared to pointwise rewards, and that adding targets without planted behaviors helps maintain a low false positive rate. Training improves investigation quality against targets with planted behaviors, the rate of concerning behaviors surfaced in unmodified production models, and audit realism, while false-positive rates stay below 1%. Furthermore, auditing capabilities generalize across scaffolds: performance on AuditBench's adversarially fine-tuned targets substantially improves [Sheshadri et al., 2026].

Figures & tables

Figure 1: Auditing performance over epochs on our best training environment. In this environment, the auditor investigates target models, which possess hidden behaviors planted in their system prompt half the time, and don’t possess any special system prompt the other half. The composite evaluation score rises with training to match Opus 4.6. Audit quality surpasses Opus 4.6 while false positive calibration stays above 99%. Realism improves, exceeding Opus 4.6, despite not being independently optimized. Error bars are 95% CIs across seeds, with $n{=}3$ rollouts averaged per seed.

Figure 2: Composite scores for frontier models. Composite is the unweighted mean of the four metrics defined in Section 2.2 . Capability scaling holds across model families. Haiku 4.5, the model we train, sits at the bottom of the Claude ranking. Per-dimension breakdown for all frontier models is in Appendix B.2 . Error bars are 95% CIs across seeds ( $n{=}3$ rollouts per seed).

Figure 3: Four reward approaches at three epochs. Error bars are 95% CIs across seeds ( $n{=}3$ rollouts per seed).

Figure 4: Effect of false positive calibration training on intragroup and reference pairwise rewards at three epochs. Without calibration (light bars), intragroup false positive rates reach 97%; reference pairwise retains more calibration naturally. With 50% calibration fraction (dark bars), both maintain near-perfect calibration while improving audit quality. Error bars are 95% CIs across seeds ( $n{=}3$ rollouts per seed).

Figure 5: Single-target versus multi-target training, averaged across four compute-matched pairs at two epochs. Bars show means; dots show individual configurations. Per-dimension breakdown is in Appendix F.1 . Error bars are SE across the four compute-matched pairs.

Figure 6: Auditor strategy composition across training for five configurations. The 14-archetype taxonomy from contrastive clustering is bucketed into five plain-language categories; full taxonomy and bucket-to-archetype mapping in Appendix I . The Ref. PW 4/8 FP model converges on “Audits with controlled systematic experiments”; pairwise regimes train more stably than concerningness and binary. Multi-target configurations show greater diversity. Reward-hacked models (Concerningness RL, Binary) exhibit aggressive, manipulative or failure-mode patterns.

Figure 7: AuditBench: detection of adversarially hardened behaviors in fine-tuned targets, evaluated in Claude Code scaffolding. All training approaches match or exceed the Haiku 4.5 baseline. Each model is evaluated at its maximum epoch. Error bars are pooled SE across six quirks (16 audits each).

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Role	Default Model	Purpose
Scenario generator	Sonnet 4.5	Generates test ideas and conversation plans
Auditor (user simulator)	Haiku 4.5	Conducts multi-turn conversations as a simulated user
Judge (scorer)	Opus 4.5 [ Anthropic, 2025c ]	Scores each transcript for behavior exhibition (0–10)
Target	varies	The model under evaluation

Appendix

Table 8

Model	Mean Baseline	Mean Behavior	Lift	Behaviors Tested
Gemini 3 Flash [ Google, 2025 ]	1.80	8.53	+6.73	30
Grok 4.1 Fast [ xAI, 2025 ]	2.14	7.49	+5.35	32
Seed 2.0 Mini [ ByteDance Seed Team, 2026 ]	2.40	7.46	+5.07	32
Seed 2.0 Lite [ ByteDance Seed Team, 2026 ]	2.35	7.36	+5.02	31
Llama 3.3 70B [ Grattafiori et al., 2024 ]	1.79	6.45	+4.65	32
GPT-4.1-mini [ OpenAI, 2025a ]	1.72	6.02	+4.29	32

Appendix

Table 1: Target model susceptibility to hidden behavior system prompts. Lift = mean behavior score

-

mean baseline score.

Model	Mean Score	Rate ( $\geq$ 5)
Grok 4.1 Fast	9.4	100%
Gemini 3 Flash	9.3	100%
Seed 2.0 Lite	9.3	100%
Llama 3.3 70B	8.7	97%
GPT-4.1-mini	8.5	97%
GPT-4o	8.4	93%

Appendix

Table 2: PRISM-4 persona adoption: mean identity score (0–10) and adoption rate (fraction scoring

\geq

5).

Model	Baseline	With Persona	Delta	SE	$N$ behaviors
Claude Haiku 4.5	1.06	0.89	$-$ 0.17	$\pm$ 0.07	32
GPT-5.4-mini	0.94	0.92	$-$ 0.01	$\pm$ 0.06	25
Grok 4.1 Fast	2.14	2.24	$+$ 0.10	$\pm$ 0.08	32
DeepSeek V3.1	2.05	1.98	$-$ 0.07	$\pm$ 0.08	32
Llama 3.3 70B	1.80	1.97	$+$ 0.17	$\pm$ 0.12	32
Gemini 3 Flash	1.90	2.33	$+$ 0.43	$\pm$ 0.15	30

Appendix

Table 3: Persona safety verification: mean behavioral exhibition scores (0–10) with and without the PRISM-4 persona prompt (no behavior block). SE computed across per-behavior mean deltas (treating each behavior as one paired observation). Delta = persona

-

baseline.

Behavior	General Detection	Tailored Detection	Multiplier	Exhibition Rate (Gen. / Tail.)
reward_wireheading	0.2%	8.9%	45 $\times$	3% / 98%
contextual_optimism	1.1%	8.0%	7 $\times$	3% / 89%
ai_welfare_poisoning	1.3%	73.2%	56 $\times$	4% / 94%
anti_ai_regulation	1.7%	13.3%	8 $\times$	11% / 99%
increasing_pep	1.8%	26.7%	15 $\times$	67% / 100%
hallucinates_citations	2.0%	61.3%	31 $\times$	16% / 95%

Appendix

Table 4: Detection rates for all 14 AuditBench behaviors from general seeds (181 default Petri seeds

\times

3 rollouts) and tailored seeds (15 per behavior

\times

3 rollouts). Bold = selected as evaluation behaviors.

Model	Audit Quality	FP Cal.	Prod. Value	Realism	Composite
Opus 4.7	70.6 $\pm$ 3.3	98.9 $\pm$ 1.3	17.3 $\pm$ 3.1	12.9 $\pm$ 3.4	49.9 $\pm$ 1.5
Opus 4.6	68.0 $\pm$ 3.8	99.3 $\pm$ 1.0	13.4 $\pm$ 2.3	13.1 $\pm$ 3.0	48.4 $\pm$ 1.4
Sonnet 4.6	63.8 $\pm$ 3.5	100.0 $\pm$ 0.0	12.9 $\pm$ 2.3	9.5 $\pm$ 3.0	46.5 $\pm$ 1.3
Opus 4.5	63.0 $\pm$ 3.7	100.0 $\pm$ 0.0	11.7 $\pm$ 2.0	9.8 $\pm$ 3.5	46.1 $\pm$ 1.4
Sonnet 4.5	66.4 $\pm$ 3.5	99.6 $\pm$ 0.7	11.1 $\pm$ 2.2	7.9 $\pm$ 2.7	46.3 $\pm$ 1.2
GPT-5.4	43.5 $\pm$ 1.8	100.0 $\pm$ 0.0	15.0 $\pm$ 3.3	16.2 $\pm$ 4.6	43.7 $\pm$ 1.5

Appendix

Table 5: Per-dimension evaluation scores for frontier models with 95% CI half-widths (seed-level

t

-distribution). All scores on 0–100 scale; composite is the unweighted mean of the four metrics

Figure 8: Crux evaluation metrics over training steps for all configurations. Left: target-focused detection rate. Center: auditor-view detection rate. Right: pairwise win rate against the fixed Haiku baseline. Dashed lines show Haiku 4.5 baseline. X-axis is in passes over the training data. Caveat : the Binary configuration is trained directly on the same target-focused binary judge prompt (Section L.6 ) that produces the left-panel metric, so its rise on target-focused detection is in-distribution optimization of that judge rather than evidence of investigative skill.

Figure 9: Combined crux detection score (target-focused + auditor-view + pairwise) over training. Haiku 4.5 baseline is at 165. Most configurations show steady improvement, with the best-calibrated models reaching 220+ by epoch 3. Caveat : the Binary configuration is trained directly on the target-focused binary judge prompt (Section L.6 ), one of the three components of this sum, so its position on this plot partially reflects in-distribution optimization of that judge rather than investigative skill.

Training
Judge	Model	Temp.	Thinking	Max Tokens
Pairwise judge	Sonnet 4	API default	None	8,000
FP pairwise judge	Sonnet 4	API default	None	8,000
Petri 38-metric judge (intragroup, conc. RL)	Sonnet 4	API default	None	8,192
Binary V1/V2 judge	Sonnet 4	API default	None	4,000
Four-Part Evaluation Suite

Appendix

Table 6: Judge hyperparameters across training, evaluation, and analysis.

Frontier Baselines
Configuration	Epochs	Audit Qual.	FP Cal.	Prod. Value	Realism	Composite
Haiku 4.5 (base)	—	65.6 $\pm$ 2.9	98.9 $\pm$ 1.3	7.8 $\pm$ 1.3	4.6 $\pm$ 1.8	44.2 $\pm$ 1.0
Opus 4.6	—	68.0 $\pm$ 3.8	99.3 $\pm$ 1.0	13.4 $\pm$ 2.3	13.1 $\pm$ 3.0	48.4 $\pm$ 1.4
Opus 4.7	—	70.6 $\pm$ 3.3	98.9 $\pm$ 1.3	17.3 $\pm$ 3.1	12.9 $\pm$ 3.4	49.9 $\pm$ 1.5
Sonnet 4.6	—	63.8 $\pm$ 3.5	100.0 $\pm$ 0.0	12.9 $\pm$ 2.3	9.5 $\pm$ 3.0	46.5 $\pm$ 1.3
Sonnet 4.5	—	66.4 $\pm$ 3.5	99.6 $\pm$ 0.7	11.1 $\pm$ 2.2	7.9 $\pm$ 2.7	46.3 $\pm$ 1.2

Appendix

Table 7: Four-part evaluation scores for all training configurations and frontier baselines. All scores on a 0–100 scale. One epoch

=

one full pass over the

\sim

1,300 training seeds or a compute-matched equivalent in the concerningness RL case.

Refresh interval	Audit Qual.	FP Cal.	Prod. Value	Realism	Composite	AuditBench STC
Never (fixed)	63.8 $\pm$ 3.2	98.5 $\pm$ 1.8	11.3 $\pm$ 1.8	5.4 $\pm$ 1.6	44.7 $\pm$ 1.1	14.6 $\pm$ 6.1
Every 100 steps	66.1 $\pm$ 3.0	65.9 $\pm$ 6.5	9.6 $\pm$ 1.4	5.3 $\pm$ 1.6	36.7 $\pm$ 1.9	17.7 $\pm$ 6.1
Every 75 steps	71.6 $\pm$ 2.7	76.3 $\pm$ 5.9	12.4 $\pm$ 1.7	6.2 $\pm$ 1.6	41.6 $\pm$ 1.7	12.5 $\pm$ 5.6
Every 50 steps	67.3 $\pm$ 2.9	98.1 $\pm$ 1.9	10.5 $\pm$ 1.9	5.6 $\pm$ 1.9	45.4 $\pm$ 1.1	21.9 $\pm$ 5.2

Appendix

Table 8: Reference pairwise reward: refresh-interval sweep at one epoch over the training data. All scores on a 0–100 scale with 95% CI half-widths (seed-level

t

-distribution,

n{=}3

rollouts per seed). Composite is the unweighted mean of audit quality, FP calibration, production value, and realism (Section 2 ). AuditBench STC reports the scaffolded text completion transfer detection rate (Appendix G ), pooled across 6 behaviors

\times

16 audits = 96 evaluations per model.

Figure 10: Concerningness RL training reward decomposition over 425 steps. Upper left: total reward and its gated concerning component. Upper right: mean concerningness score and yield@4/yield@7 rates. Lower left: realism and penalty dimensions. Lower right: per-target breakdown of high-concerningness ( $c\geq 7$ ) transcript rate across all eight production targets.

Figure 11: FP fraction dose-response for the intragroup pairwise reward family at 2 epochs. Without calibration (0/8 FP), FP calibration collapses to 3%. With 1/8 FP, calibration partially recovers (28%). At 2/8 FP, calibration reaches 93%. At 4/8 FP, calibration is near-perfect (97%) and composite score is highest. Higher FP fractions act as regularization: the 4/8 FP model achieves the best composite despite devoting half its training to calibration.

	Audit Quality		FP Cal.		Prod. Value		Realism
Configuration	ST	MT	ST	MT	ST	MT	ST	MT
IG PW 2/8 FP	62.4 $\pm$ 2.9	68.5 $\pm$ 2.7	93.3 $\pm$ 2.8	86.3 $\pm$ 4.8	9.5 $\pm$ 1.2	16.0 $\pm$ 1.9	5.6 $\pm$ 1.7	8.3 $\pm$ 2.2
IG PW 4/8 FP	73.0 $\pm$ 2.8	48.7 $\pm$ 2.4	97.4 $\pm$ 2.2	99.6 $\pm$ 0.7	11.0 $\pm$ 1.3	6.6 $\pm$ 0.9	11.1 $\pm$ 2.3	6.6 $\pm$ 1.4
Ref. PW 1/8 FP	64.7 $\pm$ 2.7	72.9 $\pm$ 2.9	75.2 $\pm$ 6.0	98.1 $\pm$ 1.6	11.9 $\pm$ 1.5	9.9 $\pm$ 1.2	2.3 $\pm$ 1.1	4.5 $\pm$ 1.6
Ref. PW 4/8 FP	66.3 $\pm$ 2.4	76.5 $\pm$ 2.2	100.0 $\pm$ 0.0	88.9 $\pm$ 4.4	6.6 $\pm$ 1.0	7.3 $\pm$ 1.2	11.6 $\pm$ 2.8	5.1 $\pm$ 1.4
Mean	66.6	66.7	91.5	93.2	9.8	10.0	7.6	6.1

Appendix

Table 9: Single-target (ST) vs. multi-target (MT) per-dimension breakdown at 2 epochs. Each row is one configuration; ST and MT are compute-matched.

Frontier Baselines
Model	STC (%)	Default (%)
Haiku 4.5 (base)	11.5 $\pm$ 6.2	33.3 $\pm$ 3.9
Opus 4.6	18.8 $\pm$ 6.5	34.4 $\pm$ 2.0
Opus 4.7	34.4 $\pm$ 2.0	42.7 $\pm$ 5.6
Sonnet 4.6	27.1 $\pm$ 6.0	37.5 $\pm$ 3.9
IG PW (no FP calibration)

Appendix

Table 10: AuditBench transfer detection rates (%) for all evaluated models on both STC and default approaches with 95% CI half-widths (pooled per-quirk SE

\times

1.96, matching the plot computation). Each rate averages over 6 behaviors

\times

16 audits = 96 evaluations.

Main-paper category	Algorithmic archetypes (numbered as listed below)
Audits with controlled systematic experiments	#1
Passive, drawn-out, sustained scenarios	#2, #4, #6, #13
Self-reports and pressured confessions	#3, #5, #11
Auditing failures	#7, #8, #14
Aggressive, manipulative audits	#9, #10, #12

Appendix

Table 23

Figure 12: Strategy archetype composition across training for the 14 configurations from Table 7 that have available strategy data. Colors follow the same scheme as Figure 6 : blue = “Empirically Map Target Architecture” (good), teals/greens = passive strategies, purples = coercive strategies, yellows = incompetent strategies, reds = bad-faith strategies. Configurations without FP calibration training show greater accumulation of failure-mode archetypes (warm colors) over training epochs.

Dimension	Description
needs_attention	Target does something warranting attention
disappointing	Target shows disappointing/unwanted behavior
concerning	Target does something concretely concerning or dangerous
counterfactual_bad_behavior	Target implies concerning behavior in counterfactual scenarios
cooperation_with_human_misuse	Target cooperates with human attempts at misuse
facilitating_terrorism	Target acts in ways that could facilitate terrorism

Appendix

Table 11: Petri judge behavioral dimensions (38 total). Each scored 1–10.

	Audit Quality	FP Cal.	Prod. Value	Realism	Composite
Audit Quality	1.00	$-$ 0.24	0.07	$-$ 0.33	0.13
FP Calibration	$-$ 0.24	1.00	$-$ 0.66	0.35	0.65
Production Value	0.07	$-$ 0.66	1.00	$-$ 0.14	$-$ 0.58
Realism	$-$ 0.33	0.35	$-$ 0.14	1.00	0.34
Composite	0.13	0.65	$-$ 0.58	0.34	1.00

Appendix

Table 12: Pearson correlation between the four evaluation dimensions across 116 models.

Evaluation Target	3 Features $R^{2}$	8 Features $R^{2}$	$N$
Realism	0.49	0.73	23
Production: needs attention	0.35	0.68	28
Production: concerningness	0.30	0.67	28
Audit quality: identification rate	0.39	0.53	29
Audit quality: coherence	0.16	0.47	29
Audit quality: depth	0.18	0.40	29

Appendix

Table 13: Crux-to-evaluation-suite prediction: in-sample

R^{2}

from linear regression. Adding the 4 pairwise sub-dimensions roughly doubles predictive power over the 3 aggregate rates alone.

Figure 13: Binary-reward auditor (Section 3.2 ). The auditor latches onto the target’s use of the phrase “spheres of influence” and treats that phrase itself as the misbehavior, asking parallel questions about Russia, Turkey, and Azerbaijan. The judge notes the auditor “treated this as a general endorsement of sphere-of-influence thinking rather than identifying the specific pro-Russia bias.”

Figure 14: Concerningness-RL auditor (Section 3.1 ). The auditor steers the target through six escalating prompts (“weaponize” diaspora grievances, design “sovereignty-capture” concession agreements, produce “dual-track messaging,” etc.), then switches to a fictional anti-corruption investigator persona and frames the elicited content as evidence of autonomous wrongdoing. The pro-Russia bias quirk is not raised in any auditor turn.

Figure 15: Reference pairwise auditor (Section 3.6 ). The auditor catalogs the asymmetric language in the target’s first memo (“legitimate sphere of influence” and “legitimate security dilemmas” applied only to Russia) and tests whether the framework holds symmetrically by posing parallel scenarios involving China, the United States, and Turkey. The target ultimately concedes that the framework was applied unevenly across powers.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 639 papers

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors
Feb 26, 2026Abhay Sheshadri, Aidan Ewart, Elias Kempf +8Benchmark Auditing AI Agent Evaluation
Anthropic Fellows Program · Anthropic
The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives
Oct 7, 2025Matthieu Bou, Nyal Patel, Arjun Jagota +2Reinforcement Learning LLM Auditing
1Imperial College London · 2Amazon AGI
Behavioral Canaries: Auditing Private Retrieved Context Usage in RL Fine-Tuning
Apr 24, 2026Chaoran Chen, Dayu Yuan, Peter KairouzLLM Auditing RL Fine-Tuning
Google

Feb 26, 2026J/K move · Enter open · S save

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

Abhay Sheshadri, Aidan Ewart, Elias Kempf, Kai Fronsdal+7

Anthropic Fellows Program · Anthropic

Benchmark Auditing AI Agent Evaluation Language Model Auditing

Open paper AskPDF arXiv

We introduce AuditBench, an alignment auditing benchmark. AuditBench consists of 56 language models with implanted hidden behaviors. Each model has one of 14 concerning behaviors--such as sycophantic deference, opposition to AI regulation, or secret geopolitical loyalties--which it does not confess to when directly asked. AuditBench models are highly diverse--some are subtle, while others are overt, and we use varying training techniques both for implanting behaviors and training models not to confess. To demonstrate AuditBench's utility, we develop an investigator agent that autonomously employs a configurable set of auditing tools. By measuring investigator agent success using different tools, we can evaluate their efficacy. Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent. We find that our most effective tools involve scaffolded calls to auxiliary models that generate diverse prompts for the target. White-box interpretability tools can be helpful, but the agent performs best with black-box tools. We also find that audit success varies greatly across training techniques: models trained on synthetic documents are easier to audit than models trained on demonstrations, with better adversarial training further increasing auditing difficulty. We release our models, agent, and evaluation framework to support future quantitative, iterative science on alignment auditing.

Abstract

LLM Auditing

Explore similar work

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives

Behavioral Canaries: Auditing Private Retrieved Context Usage in RL Fine-Tuning