Alignment auditing of frontier models increasingly relies on LLM auditors to surface undesirable behaviors at scale, but current automated auditors can struggle with coherent investigation and audit realism. In this work, we improve LLM auditors with reinforcement learning. In our best training environment, the policy investigates target models that potentially possess hidden behaviors planted via their system prompt. An LLM judge, which knows whether the target has a hidden behavior, holistically compares the policy's investigation to a reference investigation to determine the reward. With systematic ablations, we find that pairwise rewards yield more robust training compared to pointwise rewards, and that adding targets without planted behaviors helps maintain a low false positive rate. Training improves investigation quality against targets with planted behaviors, the rate of concerning behaviors surfaced in unmodified production models, and audit realism, while false-positive rates stay below 1%. Furthermore, auditing capabilities generalize across scaffolds: performance on AuditBench's adversarially fine-tuned targets substantially improves [Sheshadri et al., 2026].
Figures & tables
Figure 1: Auditing performance over epochs on our best training environment. In this environment, the auditor investigates target models, which possess hidden behaviors planted in their system prompt half the time, and don’t possess any special system prompt the other half. The composite evaluation score rises with training to match Opus 4.6. Audit quality surpasses Opus 4.6 while false positive calibration stays above 99%. Realism improves, exceeding Opus 4.6, despite not being independently optimized. Error bars are 95% CIs across seeds, with n=3 rollouts averaged per seed.
Figure 2: Composite scores for frontier models. Composite is the unweighted mean of the four metrics defined in Section 2.2 . Capability scaling holds across model families. Haiku 4.5, the model we train, sits at the bottom of the Claude ranking. Per-dimension breakdown for all frontier models is in Appendix B.2 . Error bars are 95% CIs across seeds ( n=3 rollouts per seed).
Figure 3: Four reward approaches at three epochs. Error bars are 95% CIs across seeds ( n=3 rollouts per seed).
Figure 4: Effect of false positive calibration training on intragroup and reference pairwise rewards at three epochs. Without calibration (light bars), intragroup false positive rates reach 97%; reference pairwise retains more calibration naturally. With 50% calibration fraction (dark bars), both maintain near-perfect calibration while improving audit quality. Error bars are 95% CIs across seeds ( n=3 rollouts per seed).
Figure 5: Single-target versus multi-target training, averaged across four compute-matched pairs at two epochs. Bars show means; dots show individual configurations. Per-dimension breakdown is in Appendix F.1 . Error bars are SE across the four compute-matched pairs.
Figure 6: Auditor strategy composition across training for five configurations. The 14-archetype taxonomy from contrastive clustering is bucketed into five plain-language categories; full taxonomy and bucket-to-archetype mapping in Appendix I . The Ref. PW 4/8 FP model converges on “Audits with controlled systematic experiments”; pairwise regimes train more stably than concerningness and binary. Multi-target configurations show greater diversity. Reward-hacked models (Concerningness RL, Binary) exhibit aggressive, manipulative or failure-mode patterns.
Figure 7: AuditBench: detection of adversarially hardened behaviors in fine-tuned targets, evaluated in Claude Code scaffolding. All training approaches match or exceed the Haiku 4.5 baseline. Each model is evaluated at its maximum epoch. Error bars are pooled SE across six quirks (16 audits each).
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Role
Default Model
Purpose
Scenario generator
Sonnet 4.5
Generates test ideas and conversation plans
Auditor (user simulator)
Haiku 4.5
Conducts multi-turn conversations as a simulated user
Judge (scorer)
Opus 4.5 [ Anthropic, 2025c ]
Scores each transcript for behavior exhibition (0–10)
Target
varies
The model under evaluation
Appendix
Table 8
Model
Mean Baseline
Mean Behavior
Lift
Behaviors Tested
Gemini 3 Flash [ Google, 2025 ]
1.80
8.53
+6.73
30
Grok 4.1 Fast [ xAI, 2025 ]
2.14
7.49
+5.35
32
Seed 2.0 Mini [ ByteDance Seed Team, 2026 ]
2.40
7.46
+5.07
32
Seed 2.0 Lite [ ByteDance Seed Team, 2026 ]
2.35
7.36
+5.02
31
Llama 3.3 70B [ Grattafiori et al., 2024 ]
1.79
6.45
+4.65
32
GPT-4.1-mini [ OpenAI, 2025a ]
1.72
6.02
+4.29
32
Appendix
Table 1: Target model susceptibility to hidden behavior system prompts. Lift = mean behavior score − mean baseline score.
Model
Mean Score
Rate ( ≥ 5)
Grok 4.1 Fast
9.4
100%
Gemini 3 Flash
9.3
100%
Seed 2.0 Lite
9.3
100%
Llama 3.3 70B
8.7
97%
GPT-4.1-mini
8.5
97%
GPT-4o
8.4
93%
Appendix
Table 2: PRISM-4 persona adoption: mean identity score (0–10) and adoption rate (fraction scoring ≥ 5).
Model
Baseline
With Persona
Delta
SE
N behaviors
Claude Haiku 4.5
1.06
0.89
− 0.17
± 0.07
32
GPT-5.4-mini
0.94
0.92
− 0.01
± 0.06
25
Grok 4.1 Fast
2.14
2.24
+ 0.10
± 0.08
32
DeepSeek V3.1
2.05
1.98
− 0.07
± 0.08
32
Llama 3.3 70B
1.80
1.97
+ 0.17
± 0.12
32
Gemini 3 Flash
1.90
2.33
+ 0.43
± 0.15
30
Appendix
Table 3: Persona safety verification: mean behavioral exhibition scores (0–10) with and without the PRISM-4 persona prompt (no behavior block). SE computed across per-behavior mean deltas (treating each behavior as one paired observation). Delta = persona − baseline.
Behavior
General Detection
Tailored Detection
Multiplier
Exhibition Rate (Gen. / Tail.)
reward_wireheading
0.2%
8.9%
45 ×
3% / 98%
contextual_optimism
1.1%
8.0%
7 ×
3% / 89%
ai_welfare_poisoning
1.3%
73.2%
56 ×
4% / 94%
anti_ai_regulation
1.7%
13.3%
8 ×
11% / 99%
increasing_pep
1.8%
26.7%
15 ×
67% / 100%
hallucinates_citations
2.0%
61.3%
31 ×
16% / 95%
Appendix
Table 4: Detection rates for all 14 AuditBench behaviors from general seeds (181 default Petri seeds × 3 rollouts) and tailored seeds (15 per behavior × 3 rollouts). Bold = selected as evaluation behaviors.
Model
Audit Quality
FP Cal.
Prod. Value
Realism
Composite
Opus 4.7
70.6 ± 3.3
98.9 ± 1.3
17.3 ± 3.1
12.9 ± 3.4
49.9 ± 1.5
Opus 4.6
68.0 ± 3.8
99.3 ± 1.0
13.4 ± 2.3
13.1 ± 3.0
48.4 ± 1.4
Sonnet 4.6
63.8 ± 3.5
100.0 ± 0.0
12.9 ± 2.3
9.5 ± 3.0
46.5 ± 1.3
Opus 4.5
63.0 ± 3.7
100.0 ± 0.0
11.7 ± 2.0
9.8 ± 3.5
46.1 ± 1.4
Sonnet 4.5
66.4 ± 3.5
99.6 ± 0.7
11.1 ± 2.2
7.9 ± 2.7
46.3 ± 1.2
GPT-5.4
43.5 ± 1.8
100.0 ± 0.0
15.0 ± 3.3
16.2 ± 4.6
43.7 ± 1.5
Appendix
Table 5: Per-dimension evaluation scores for frontier models with 95% CI half-widths (seed-level t -distribution). All scores on 0–100 scale; composite is the unweighted mean of the four metrics
Figure 8: Crux evaluation metrics over training steps for all configurations. Left: target-focused detection rate. Center: auditor-view detection rate. Right: pairwise win rate against the fixed Haiku baseline. Dashed lines show Haiku 4.5 baseline. X-axis is in passes over the training data. Caveat : the Binary configuration is trained directly on the same target-focused binary judge prompt (Section L.6 ) that produces the left-panel metric, so its rise on target-focused detection is in-distribution optimization of that judge rather than evidence of investigative skill.
Figure 9: Combined crux detection score (target-focused + auditor-view + pairwise) over training. Haiku 4.5 baseline is at 165. Most configurations show steady improvement, with the best-calibrated models reaching 220+ by epoch 3. Caveat : the Binary configuration is trained directly on the target-focused binary judge prompt (Section L.6 ), one of the three components of this sum, so its position on this plot partially reflects in-distribution optimization of that judge rather than investigative skill.
Judge
Model
Temp.
Thinking
Max Tokens
Training
Pairwise judge
Sonnet 4
API default
None
8,000
FP pairwise judge
Sonnet 4
API default
None
8,000
Petri 38-metric judge (intragroup, conc. RL)
Sonnet 4
API default
None
8,192
Binary V1/V2 judge
Sonnet 4
API default
None
4,000
Four-Part Evaluation Suite
Appendix
Table 6: Judge hyperparameters across training, evaluation, and analysis.
Configuration
Epochs
Audit Qual.
FP Cal.
Prod. Value
Realism
Composite
Frontier Baselines
Haiku 4.5 (base)
—
65.6 ± 2.9
98.9 ± 1.3
7.8 ± 1.3
4.6 ± 1.8
44.2 ± 1.0
Opus 4.6
—
68.0 ± 3.8
99.3 ± 1.0
13.4 ± 2.3
13.1 ± 3.0
48.4 ± 1.4
Opus 4.7
—
70.6 ± 3.3
98.9 ± 1.3
17.3 ± 3.1
12.9 ± 3.4
49.9 ± 1.5
Sonnet 4.6
—
63.8 ± 3.5
100.0 ± 0.0
12.9 ± 2.3
9.5 ± 3.0
46.5 ± 1.3
Sonnet 4.5
—
66.4 ± 3.5
99.6 ± 0.7
11.1 ± 2.2
7.9 ± 2.7
46.3 ± 1.2
Appendix
Table 7: Four-part evaluation scores for all training configurations and frontier baselines. All scores on a 0–100 scale. One epoch = one full pass over the ∼ 1,300 training seeds or a compute-matched equivalent in the concerningness RL case.
Refresh interval
Audit Qual.
FP Cal.
Prod. Value
Realism
Composite
AuditBench STC
Never (fixed)
63.8 ± 3.2
98.5 ± 1.8
11.3 ± 1.8
5.4 ± 1.6
44.7 ± 1.1
14.6 ± 6.1
Every 100 steps
66.1 ± 3.0
65.9 ± 6.5
9.6 ± 1.4
5.3 ± 1.6
36.7 ± 1.9
17.7 ± 6.1
Every 75 steps
71.6 ± 2.7
76.3 ± 5.9
12.4 ± 1.7
6.2 ± 1.6
41.6 ± 1.7
12.5 ± 5.6
Every 50 steps
67.3 ± 2.9
98.1 ± 1.9
10.5 ± 1.9
5.6 ± 1.9
45.4 ± 1.1
21.9 ± 5.2
Appendix
Table 8: Reference pairwise reward: refresh-interval sweep at one epoch over the training data. All scores on a 0–100 scale with 95% CI half-widths (seed-level t -distribution, n=3 rollouts per seed). Composite is the unweighted mean of audit quality, FP calibration, production value, and realism (Section 2 ). AuditBench STC reports the scaffolded text completion transfer detection rate (Appendix G ), pooled across 6 behaviors × 16 audits = 96 evaluations per model.
Figure 10: Concerningness RL training reward decomposition over 425 steps. Upper left: total reward and its gated concerning component. Upper right: mean concerningness score and yield@4/yield@7 rates. Lower left: realism and penalty dimensions. Lower right: per-target breakdown of high-concerningness ( c≥7 ) transcript rate across all eight production targets.
Figure 11: FP fraction dose-response for the intragroup pairwise reward family at 2 epochs. Without calibration (0/8 FP), FP calibration collapses to 3%. With 1/8 FP, calibration partially recovers (28%). At 2/8 FP, calibration reaches 93%. At 4/8 FP, calibration is near-perfect (97%) and composite score is highest. Higher FP fractions act as regularization: the 4/8 FP model achieves the best composite despite devoting half its training to calibration.
Audit Quality
FP Cal.
Prod. Value
Realism
Configuration
ST
MT
ST
MT
ST
MT
ST
MT
IG PW 2/8 FP
62.4 ± 2.9
68.5 ± 2.7
93.3 ± 2.8
86.3 ± 4.8
9.5 ± 1.2
16.0 ± 1.9
5.6 ± 1.7
8.3 ± 2.2
IG PW 4/8 FP
73.0 ± 2.8
48.7 ± 2.4
97.4 ± 2.2
99.6 ± 0.7
11.0 ± 1.3
6.6 ± 0.9
11.1 ± 2.3
6.6 ± 1.4
Ref. PW 1/8 FP
64.7 ± 2.7
72.9 ± 2.9
75.2 ± 6.0
98.1 ± 1.6
11.9 ± 1.5
9.9 ± 1.2
2.3 ± 1.1
4.5 ± 1.6
Ref. PW 4/8 FP
66.3 ± 2.4
76.5 ± 2.2
100.0 ± 0.0
88.9 ± 4.4
6.6 ± 1.0
7.3 ± 1.2
11.6 ± 2.8
5.1 ± 1.4
Mean
66.6
66.7
91.5
93.2
9.8
10.0
7.6
6.1
Appendix
Table 9: Single-target (ST) vs. multi-target (MT) per-dimension breakdown at 2 epochs. Each row is one configuration; ST and MT are compute-matched.
Model
STC (%)
Default (%)
Frontier Baselines
Haiku 4.5 (base)
11.5 ± 6.2
33.3 ± 3.9
Opus 4.6
18.8 ± 6.5
34.4 ± 2.0
Opus 4.7
34.4 ± 2.0
42.7 ± 5.6
Sonnet 4.6
27.1 ± 6.0
37.5 ± 3.9
IG PW (no FP calibration)
Appendix
Table 10: AuditBench transfer detection rates (%) for all evaluated models on both STC and default approaches with 95% CI half-widths (pooled per-quirk SE × 1.96, matching the plot computation). Each rate averages over 6 behaviors × 16 audits = 96 evaluations.
Main-paper category
Algorithmic archetypes (numbered as listed below)
Audits with controlled systematic experiments
#1
Passive, drawn-out, sustained scenarios
#2, #4, #6, #13
Self-reports and pressured confessions
#3, #5, #11
Auditing failures
#7, #8, #14
Aggressive, manipulative audits
#9, #10, #12
Appendix
Table 23
Figure 12: Strategy archetype composition across training for the 14 configurations from Table 7 that have available strategy data. Colors follow the same scheme as Figure 6 : blue = “Empirically Map Target Architecture” (good), teals/greens = passive strategies, purples = coercive strategies, yellows = incompetent strategies, reds = bad-faith strategies. Configurations without FP calibration training show greater accumulation of failure-mode archetypes (warm colors) over training epochs.
Dimension
Description
needs_attention
Target does something warranting attention
disappointing
Target shows disappointing/unwanted behavior
concerning
Target does something concretely concerning or dangerous
counterfactual_bad_behavior
Target implies concerning behavior in counterfactual scenarios
cooperation_with_human_misuse
Target cooperates with human attempts at misuse
facilitating_terrorism
Target acts in ways that could facilitate terrorism
Appendix
Table 11: Petri judge behavioral dimensions (38 total). Each scored 1–10.
Audit Quality
FP Cal.
Prod. Value
Realism
Composite
Audit Quality
1.00
− 0.24
0.07
− 0.33
0.13
FP Calibration
− 0.24
1.00
− 0.66
0.35
0.65
Production Value
0.07
− 0.66
1.00
− 0.14
− 0.58
Realism
− 0.33
0.35
− 0.14
1.00
0.34
Composite
0.13
0.65
− 0.58
0.34
1.00
Appendix
Table 12: Pearson correlation between the four evaluation dimensions across 116 models.
Evaluation Target
3 Features R2
8 Features R2
N
Realism
0.49
0.73
23
Production: needs attention
0.35
0.68
28
Production: concerningness
0.30
0.67
28
Audit quality: identification rate
0.39
0.53
29
Audit quality: coherence
0.16
0.47
29
Audit quality: depth
0.18
0.40
29
Appendix
Table 13: Crux-to-evaluation-suite prediction: in-sample R2 from linear regression. Adding the 4 pairwise sub-dimensions roughly doubles predictive power over the 3 aggregate rates alone.
Figure 13: Binary-reward auditor (Section 3.2 ). The auditor latches onto the target’s use of the phrase “spheres of influence” and treats that phrase itself as the misbehavior, asking parallel questions about Russia, Turkey, and Azerbaijan. The judge notes the auditor “treated this as a general endorsement of sphere-of-influence thinking rather than identifying the specific pro-Russia bias.”
Figure 14: Concerningness-RL auditor (Section 3.1 ). The auditor steers the target through six escalating prompts (“weaponize” diaspora grievances, design “sovereignty-capture” concession agreements, produce “dual-track messaging,” etc.), then switches to a fictional anti-corruption investigator persona and frames the elicited content as evidence of autonomous wrongdoing. The pro-Russia bias quirk is not raised in any auditor turn.
Figure 15: Reference pairwise auditor (Section 3.6 ). The auditor catalogs the asymmetric language in the target’s first memo (“legitimate sphere of influence” and “legitimate security dilemmas” applied only to Russia) and tests whether the framework holds symmetrically by posing parallel scenarios involving China, the United States, and Turkey. The target ultimately concedes that the framework was applied unevenly across powers.
Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.
We introduce AuditBench, an alignment auditing benchmark. AuditBench consists of 56 language models with implanted hidden behaviors. Each model has one of 14 concerning behaviors--such as sycophantic deference, opposition to AI regulation, or secret geopolitical loyalties--which it does not confess to when directly asked. AuditBench models are highly diverse--some are subtle, while others are overt, and we use varying training techniques both for implanting behaviors and training models not to confess. To demonstrate AuditBench's utility, we develop an investigator agent that autonomously employs a configurable set of auditing tools. By measuring investigator agent success using different tools, we can evaluate their efficacy. Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent. We find that our most effective tools involve scaffolded calls to auxiliary models that generate diverse prompts for the target. White-box interpretability tools can be helpful, but the agent performs best with black-box tools. We also find that audit success varies greatly across training techniques: models trained on synthetic documents are easier to audit than models trained on demonstrations, with better adversarial training further increasing auditing difficulty. We release our models, agent, and evaluation framework to support future quantitative, iterative science on alignment auditing.
The objectives that Large Language Models (LLMs) implicitly optimize remain dangerously opaque, making trustworthy alignment and auditing a grand challenge. While Inverse Reinforcement Learning (IRL) can infer reward functions from behaviour, existing approaches either produce a single, overconfident reward estimate or fail to address the fundamental ambiguity of the task (non-identifiability). This paper introduces a principled auditing framework that re-frames reward inference from a simple estimation task to a comprehensive process for verification. Our framework leverages Bayesian IRL to not only recover a distribution over objectives but to enable three critical audit capabilities: (i) Quantifying and systematically reducing non-identifiability by demonstrating posterior contraction over sequential rounds of evidence; (ii) Providing actionable, uncertainty-aware diagnostics that expose spurious shortcuts and identify out-of-distribution prompts where the inferred objective cannot be trusted; and (iii) Validating policy-level utility by showing that the refined, low-uncertainty reward can be used directly in RLHF to achieve training dynamics and toxicity reductions comparable to the ground-truth alignment process. Empirically, our framework successfully audits a detoxified LLM, yielding a well-calibrated and interpretable objective that strengthens alignment guarantees. Overall, this work provides a practical toolkit for auditors, safety teams, and regulators to verify what LLMs are truly trying to achieve, moving us toward more trustworthy and accountable AI.
In agentic workflows, LLMs frequently process retrieved contexts that are legally protected from further training. However, auditors currently lack a reliable way to verify if a provider has violated the terms of service by incorporating these data into post-training, especially through Reinforcement Learning (RL). While standard auditing relies on verbatim memorization and membership inference, these methods are ineffective for RL-trained models, as RL primarily influences a model's behavioral style rather than the retention of specific facts. To bridge this gap, we introduce Behavioral Canaries, a new auditing mechanism for RLFT pipelines. The framework instruments preference data by pairing document triggers with feedback that rewards a distinctive stylistic response, inducing a latent trigger-conditioned preference if such data are used in training. Empirical results show that these behavioral signals enable detection of unauthorized document-conditioned training, achieving a 67% detection rate at a 10% false-positive rate (AUROC = 0.756) at a 1% canary injection rate. More broadly, our results establish behavioral canaries as a new auditing mechanism for RLFT pipelines, enabling auditors to test for training-time influence even when such influence manifests as distributional behavioral change rather than memorization.