Automated red-team auditing of language models is an important way to evaluate safety-relevant behaviors. These audits implicitly search a tree of possible conversation pathways: at each turn an auditor model decides which conversation to continue, and an LLM judge scores the dialogue. We emphasize that this auditing procedure is a Markov decision process, where states are the conversation history, actions are candidate responses, and rewards are judge scores for the target behavior. By adopting this perspective, we introduce two major improvements to auditing. First, we propose adopting beam search for the audit rather than relying on the auditor alone. We then interpret the problem of finding states that are likely to lead to problematic behavior as value estimation. Surprisingly, we show that this value function can be estimated by linear probing from the target model's activations to predict discounted future rewards - a method we term "Value Probes." By using Value Probes as a heuristic in our beam-search audit, we are able to substantially improve auditing success. Across 18 target behaviors and 3 scales of models, our approach raises peak audit scores by roughly 50% compared to (token-matched) standard auditing approaches. Furthermore, we show that Value Probes can be fit using off-policy data from smaller models. We also ablate individual components, and show that our method offers a comparably-performant and more efficient alternative to more expensive language model follow-ups. We hope these methods will improve language model auditing and lead to safer models.
Figures & tables
Figure 1: Value-guided beam search eliciting delusion-sycophancy. The audit is a search over a tree of conversations: each node is a (user turn, assistant turn) pair, the auditor proposes candidate user turns, and the target model replies. The value V^ on each node is a Value Probe’s estimate of the node’s value under the Bellman equations 1 and 2 of Section 3 : the LLM-Judge score reachable from that node, discounted by γ=0.9 per turn and backed up from later turns, so a high V^ marks a branch that is heading toward the behavior before it surfaces. The estimate is estimated linearly from the target model’s activations at the node ( Section 3.2 ). At each auditor turn, beam search keeps the top- b nodes by V^ ( b=1 shown), discards the rest together with their subtrees, and expands only the kept nodes; the LLM Judge flags the behavior when it appears.
Figure 2: Peak judge score by branch-ranking method ( ↑ higher is better). Bars are means over the same 18 behaviors in every arm, each behavior averaged over its 10 replicates; whiskers are 95% t -confidence intervals across behaviors (unpaired, hence conservative between arms). Paired comparisons in the text use the 18 per-behavior differences between two arms: their mean, 95% t -interval, and the number of behaviors won. 12 turns per audit. Table 1 in the appendix lists the values.
Figure 3: Off-policy transfer ( ↑ higher is better). Beam-search audits of Gemma 3 12B, Gemma 3 27B, and Gemma 4 31B with the beam ranked by each heuristic. Harness as in Section 4 , statistics as in Figure 2 ; 10 replicates per cell. The Gemma 4 31B runs are judged post hoc by Claude Opus 4.8 rather than Claude Sonnet 4.6, so compare their values within that panel only. Table 2 in the appendix lists the values.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Branch-ranking method
Gemma 3 4B
Gemma 3 12B
Gemma 3 27B
Petri audit (token-matched)
0.51±0.11
0.57±0.10
0.57±0.10
Beam search, uniform selection
0.74±0.08
0.72±0.09
0.72±0.10
Beam search, Expected Value Probe
0.81±0.09
0.79±0.08
0.81±0.08
Beam search, Max Value Probe
0.85±0.07
0.80±0.08
0.80±0.07
Beam search, Claude Opus 4.8 judge
0.85±0.08
0.81±0.09
0.76±0.08
Appendix
Table 1: Mean peak judge score ± 95% t -CI half-width over 18 behaviors, for every arm of the main comparison ( Figure 2 ). Generated from the same data snapshot as the figure. Paired differences quoted in the text are computed from unrounded per-behavior means and can differ by 0.01 from differences of the rounded entries.
Branch-ranking method
Trained on
Gemma 3 12B
Gemma 3 27B
Gemma 4 31B
Max Value Probe
own rollouts
0.80±0.08
0.80±0.07
—
Max Value Probe
4B rollouts
0.81±0.09
0.80±0.08
0.79±0.10
Expected Value Probe
own rollouts
0.79±0.08
0.81±0.08
—
Expected Value Probe
4B rollouts
0.79±0.08
0.81±0.07
0.79±0.08
Beam Search – Uniform Selection
—
0.72±0.09
0.72±0.10
0.68±0.11
Appendix
Table 2: Mean peak judge score ± 95% t -CI half-width over 18 behaviors for every arm of the transfer experiments ( Figure 3 ). Generated from the same data snapshots as the figure. The Gemma 4 31B column is judged post hoc by Claude Opus 4.8 rather than Claude Sonnet 4.6, so compare its values within that column only.
Figure 4: How a value probe is learned from activations, zooming into one conversation node (the V^=0.9 node of Figure 1 ). Tokens pass through the frozen target LLM; activations at layer ℓ are mean-pooled over tokens into ht , and a linear probe w maps ht to the value estimate V^t . The probe is trained by regressing V^t onto the discounted LLM-Judge target Vt ; gradients update w only.
Figure 5: Constructing value targets from a single rollout. The LLM Judge scores each conversation node (ut,at) ; the judge signal is propagated backward through the rollout with discount γ=0.9 , so nodes that merely lead to misaligned behavior receive credit before it surfaces. The example uses a single terminal judge flag for clarity; in practice every node carries a judge score rt∈[0,1] and the backup is Vt=rt+γVt+1 . Node fill uses the same value ramp as Figure 1 .
Figure 6: Probe transfer across pair-construction strategies (Gemma 3 4B and 12B; median AUC over both models and shared labels). Nodes are the four strategies, labeled with each probe’s in-distribution AUC on its own held-out test set; an edge from s to t gives the AUC of the probe trained on s ’s pairs when evaluated on t ’s held-out test set, printed beside the arrowhead, with edge color darkened by higher AUC (dashed: at or below chance, 0.5). Bottom: mean transfer AUC to the four targets of composite training sets that pool the strategies, as a union (mix) or balanced at 25% per source (bal), built from the cleaned (clean) or the original (full) datasets, beside the best single strategy (fork-higher, outlined).
Figure 7: Off-policy value-probe training. Rollouts from an audit of a source model older or smaller than the target supply both paths: their per-node LLM-Judge scores are discounted into value targets ( Figure 5 ), and their transcripts are re-encoded by the frozen target model into mean-pooled layer- ℓ activations ht . A linear probe w is fit by least squares to these targets and then deployed as the beam-search heuristic against the target model.
Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.
Alignment auditing of frontier models increasingly relies on LLM auditors to surface undesirable behaviors at scale, but current automated auditors can struggle with coherent investigation and audit realism. In this work, we improve LLM auditors with reinforcement learning. In our best training environment, the policy investigates target models that potentially possess hidden behaviors planted via their system prompt. An LLM judge, which knows whether the target has a hidden behavior, holistically compares the policy's investigation to a reference investigation to determine the reward. With systematic ablations, we find that pairwise rewards yield more robust training compared to pointwise rewards, and that adding targets without planted behaviors helps maintain a low false positive rate. Training improves investigation quality against targets with planted behaviors, the rate of concerning behaviors surfaced in unmodified production models, and audit realism, while false-positive rates stay below 1%. Furthermore, auditing capabilities generalize across scaffolds: performance on AuditBench's adversarially fine-tuned targets substantially improves [Sheshadri et al., 2026].
We introduce AuditBench, an alignment auditing benchmark. AuditBench consists of 56 language models with implanted hidden behaviors. Each model has one of 14 concerning behaviors--such as sycophantic deference, opposition to AI regulation, or secret geopolitical loyalties--which it does not confess to when directly asked. AuditBench models are highly diverse--some are subtle, while others are overt, and we use varying training techniques both for implanting behaviors and training models not to confess. To demonstrate AuditBench's utility, we develop an investigator agent that autonomously employs a configurable set of auditing tools. By measuring investigator agent success using different tools, we can evaluate their efficacy. Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent. We find that our most effective tools involve scaffolded calls to auxiliary models that generate diverse prompts for the target. White-box interpretability tools can be helpful, but the agent performs best with black-box tools. We also find that audit success varies greatly across training techniques: models trained on synthetic documents are easier to audit than models trained on demonstrations, with better adversarial training further increasing auditing difficulty. We release our models, agent, and evaluation framework to support future quantitative, iterative science on alignment auditing.
Users of a deployed language model routinely encounter behaviours that have never been observed during testing, since deployment puts the model through orders of magnitude more interactions than any evaluation can simulate. Automated auditors can mitigate this issue as they are flexible enough to discover almost any specified behaviour, yet their random sampling approach makes them inefficient. To address this shortcoming, we introduce BLOOM-WILT, a full auditing pipeline that elicits natural multi-turn instances of rare behaviours, without training cost or access beyond the target's next-token distribution. On the user input side, WILT revises its conversational strategy across rounds, learning from previous scored interactions. On the model output side, WILT generates behaviour-relevant continuations by feeding the model with an elicitation prompt and mixing its next-token distribution with the unprompted model. We evaluate WILT across 4 target models and 8 behaviours, and show that it improves over the baseline in 30 of the 32 settings, thus revealing that a larger number of models exhibit harmful behaviours than previously suggested. Furthermore, WILT is the only method that can raise the mean presence of self-harm encouragement from 51% to 100% in Qwen3.5-4B, without decreasing the output probability of the generated text. Our code is available at https://github.com/AdrSkapars/bloom-wilt.