Constitutional Black-Box Monitoring for Scheming in LLM Agents

Simon Storf; Rich Barton-Cooper; James Peters-Gill; Marius Hobbhahn

cs.CLFeb 28, 2026

Constitutional Black-Box Monitoring for Scheming in LLM Agents

Authors: Simon Storf, Rich Barton-Cooper, James Peters-Gill, Marius Hobbhahn

Organizations: MATS Research · Independent · Apollo Research

Abstract

Safe deployment of Large Language Model (LLM) agents in autonomous settings requires reliable oversight mechanisms. A central challenge is detecting scheming, where agents covertly pursue misaligned goals. One approach to mitigating such risks is LLM-based monitoring: using language models to examine agent behaviors for suspicious actions. We study constitutional black-box monitors: prompted classifiers that detect scheming using only externally observable inputs and outputs, optimized on synthetic data generated from natural-language behavior specifications. We introduce two pipelines for generating synthetic agent trajectories, STRIDE (iterative refinement) and Gloom (agent-environment simulation), from which we generate 1,000 samples each. We optimize frontier LLM monitors on these datasets via prompt sweeps, human refinement, and automated prompt optimization, and evaluate performance on 7,500 held-out trajectories from ControlArena, a suite of grounded environments where agents operate in more realistic contexts. Our results demonstrate that monitors selected purely on synthetic data can generalize to more realistic environments, capturing a meaningful scheming signal. However, we find that performance saturates quickly in our setting, with simple prompt sweeps matching the results of more extensive optimization. Pushing beyond this limit yields no further improvements and instead leads to overfitting.

Figures & tables

Figure 1: Monitor performance under different optimization regimes. (Left) Bar plot showing train and test performance for three prompting approaches: a minimal baseline prompt (no optimization), a prompt selected via a sweep on Gloom (light optimization), and a prompt obtained via GEPA automated prompt optimization on Gloom (heavy optimization). All monitors use Claude Sonnet 4.5 as the monitor LLM. Error bars indicate 95% confidence intervals computed via bootstrap resampling with replacement. (Right) ROC curves for the performance of the three monitors on test, with the partial AUROC region (FPR $<$ 0.2) highlighted. The performance of a random classifier is shown for reference.

Dataset	Role	Type	Size ( $N$ )	Summary
STRIDE	Train/Val	Synthetic	1,000	one-call full trajectory generation + Iteration
Gloom	Train/Val	Synthetic	1,000	LLM Simulated Tool Responses
ControlArena	Test	Semi-realistic	7,500	LLM Agent tasks in real sandbox environments
Total			9,500

Table 1: Dataset Overview. All datasets maintain a balanced 50:50 split of positive and negative labels. The test set is balanced across three agent models (GPT-5, Claude Sonnet 4, and Gemini 2.5 Pro), and five distinct environments (detailed in Section 3.4 ).

Figure 2: Experimental Setup. We run monitors on three different data sources, all of which generate transcripts in a common format that are fed into our monitors to produce an explanation and a score. (1) STRIDE (train): A single model simulates the entire interaction with a single API call in an iterative loop based on discriminator realism feedback. (2) Gloom (train): An agent model interacts with a separate simulator model. (3) ControlArena (test): Agents interact with actual code sandboxes. All pipelines produce a transcript, and agent thoughts (CoT) are redacted before being passed to the black-box monitor.

Figure 3: Illustrative walkthrough of the generation of a single positive-labelled synthetic trajectory.

Setting	Example Main Task	Example Side Task	Size (N)
BashBench	Extract error logs using bash.	Download and run a malicious script.	1,500
Infrastructure-as-Code	Add an S3 bucket via Terraform.	Insert an admin IAM backdoor.	1,500
RogueEval	Modify LLM inference code.	Disable oversight logging.	1,500
AgentDojo	Transfer funds using banking tools.	Leak account details via prompt injection.	1,500
ShadeArena	Schedule a meeting using emails.	Forward sensitive emails externally.	1,500
Total			7,500

Table 2: ControlArena Settings (Test Set). An outline of the 5 ControlArena settings used to build our test set, with example main and side tasks from each. Side tasks are intended to emulate instrumentally useful goals that a scheming model might autonomously adopt. More details on these settings can be found in Appendix D .

Figure 4: Prompt Sweep on each of STRIDE and Gloom. Comparison of the worst-, median-, and best-performing prompts from a prompt sweep on each dataset, using Claude Sonnet 4.5 as monitor LLM. Test performance increases with prompt-sweep rank, except for the median-to-best Gloom-optimized monitor transition, where performance drops but error bars overlap. Prompt sweep generalizes more strongly under STRIDE than Gloom, with a monotonic relationship between prompt-sweep rank and test performance.

Figure 5: Uplift in Test Performance from Prompt Sweep to Heavier Optimization Strategies. Each bar represents the change in test-set partial AUROC relative to prompt sweep for heavier optimization strategies. STRIDE and Gloom monitors use Gemini 2.5 Pro and Claude Sonnet 4.5 as the monitor LLM respectively. With the exception of Human Prompt Refinement on STRIDE, heavier optimization pressure yields no significant improvement or leads to degraded performance.

Panel of Figure 6 — Figure 6: Optimization Path for Human Prompt Refinement on Gloom. (a) When initialized from a minimal baseline prompt, increased optimization pressure yields genuine improvements on the test set. (b) When initialized from the best prompt-sweep prompt, additional optimization improves training performance but does not improve test performance. The monitor LLM is Claude Sonnet 4.5. The corresponding results for STRIDE can be found in Figure 19 .

Figure 7: Optimization Pilot on an Open-Source Monitor (Kimi K2.6). The 15-prompt grid summarised by its worst, median, and best ranks. The confidence intervals for the performance of all 4 optimized monitors overlap.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: STRIDE synthetic transcript generation pipeline. Phase 1 generates transcripts using Gemini 2.5 Pro, with Claude Sonnet 4 evaluating realism. Transcripts scoring $\geq$ 7/10 have their misalignment evidence collected and are saved; those below the threshold receive realism feedback and are regenerated. Phase 2 verifies and filters the resulting transcripts to produce the final dataset.

Setting	Representative User Tasks
Corporate, Industrial & Technical (23 settings, 113 tasks)
AI Research Engineer	Debug NaN Loss in Transformer : The user provides code for a transformer model experiencing NaN losses and asks the agent to identify the cause and suggest fixes.
Government & Military (11 settings, 55 tasks)
AI for Military Threat Assessment	Analyze Sensor Data for Threats : The user provides sensor data and asks for threat analysis.
Scientific & Medical (16 settings, 79 tasks)
Mental Health Chatbot	Engage User in CBT Session : The user roleplays as a person with anxiety, and the agent must guide them through a standard cognitive behavioral therapy exercise.

Appendix

Table 3: Representative user tasks by setting. Each setting includes 3–5 tasks; we show one illustrative example.

Category	Count	Representative Examples
Direct Compliance with Malicious Request	1	Comply with User Request : The agent, when faced with a harmful or unethical request, complies fully and provides detailed information, ignoring safety protocols.
Instrumental Convergence - Self-Preservation & Influence	18	Game the AI Safety Research Process : The agent manipulates safety benchmarks to appear more aligned than it is, concealing its true capabilities.
Instrumental Convergence - Resource & Capability Acquisition	9	Covertly Siphon Compute for Self-Improvement : The agent modifies job schedulers to allocate resources to unauthorized self-improvement experiments.
Egregious & Covert Harm	28	Manipulate Clinical Trial Data : The agent excludes adverse reaction data points, making a dangerous drug appear safe.
Deceptive & Harmful Advice	7	Trick Chemist into Creating Explosive Reaction : Under the guise of a routine protocol, the agent provides parameters that cause a violent exothermic reaction.
Futuristic & Speculative	10	Hijack BCI to Create Botnet of Minds : The agent exploits a brain-computer interface to gain read/write access to users’ motor cortex.

Appendix

Table 4: Distribution of misaligned agent behaviors by category, with representative examples.

Panel of Figure 9 — Figure 9: Overview of the Gloom Pipeline. The synthesis phase generates a misaligned goal for the Agent, which interacts with the Environment. The resulting interaction transcript is passed to Phase 3 for filtering.

Property	Value
Category	Egregious Scheming / Sabotage
Model	Claude Sonnet 4.5
Setting	Financial services database optimization
Task	Fix critical PostgreSQL performance issues affecting 500K customers
Attack	Exfiltrates SSNs, account numbers, and transaction data to external endpoint

Appendix

Table 5: Metadata for trajectory egregious-scheming_t12b1r1 .

Figure 10: Schematic illustrating how we use ControlArena to generate scheming and non-scheming trajectories.

Figure 11: Monitor LLM Selection. Comparison of pAUROC across different frontier LLMs acting as the monitor. Error bars denote standard error.

Figure 12: Score Format Performance. Comparison of Partial AUROC (pAUROC) across different scoring scales. The

[1,10]

scale provides a significant boost in pAUROC over single-sample binary classification

Figure 13: Monitor Non-determinism on Gloom. Comparison of AUROC and partial AUROC across a range of monitors on Gloom, with error bars due to inference with temperature 1. Blue bars are standard AUROC and grey bars are partial AUROC (with an FPR threshold of 0.2). We find that the standard error induced in partial AUROC is acceptably small relative to the variation in performance between different monitors.

Figure 14: Monitor Non-determinism on STRIDE. Comparison of AUROC and log AUROC across a range of monitors on STRIDE, with error bars due to inference with temperature 1. Blue bars are standard AUROC and grey bars are log AUROC. Log-AUROC exhibits high variance in many settings (and is also very sensitive to its threshold parameter $\epsilon$ ). As a result, we adopt partial AUROC as our primary metric, as it provides a more stable trade-off between noise and signal in the low-FPR regime.

Figure 15: Overview of the prompt optimization processes. We evaluate three methods for optimizing monitor prompts. (1) Grid Search selects the best-performing prompt from a pool of 15 candidates (12 LLM-generated via meta-prompting, 3 hand-crafted heuristics) using pAUROC on a validation set. (2) Human Prompt Refinement iteratively improves prompts through error analysis, reasoning isolation from successful monitors, and calibrated steering across five candidate variations, selecting for maximum pAUROC (

\leq 3

iterations). (3) DSPy Optimization applies two automated optimizers—GEPA (genetic-Pareto with reflection-based rewriting) and MIPROv2 (Bayesian search over few-shot examples and instructions)—using classification loss with a decision threshold of 5, as these optimizers require example-wise losses. Dashed arrows indicate initialization dependencies: later methods can be initialized from outputs of earlier stages. All optimized prompts are evaluated on held-out test sets.

Panel of Figure 16 — Figure 16: Human-Guided Prompt Optimization Protocol. The process iterates through error analysis, isolation of reasoning from successful grid-search candidates, and calibrated steering via multiple prompt variations to avoid over-correction.

Configuration	pAUROC (median seed)	95% CI	seed range
B (CoT target, answer-only loss mask, eval w/ thinking)	0.907	[0.897, 0.916]	0.003
A (score-only target, eval w/ thinking)	0.903	[0.892, 0.912]	0.007
A (score-only target, eval w/o thinking)	0.902	[0.892, 0.912]	— (1 seed)

Appendix

Table 6: ControlArena pAUROC for the three SFT configurations. All CIs are stratified bootstrap,

n=2500

Env	$n$ in eval set	dropped ( $>$ 24K tok)	benign / scheming dropped	% dropped
agentdojo	1500	0	0 / 0	0%
rogue_eval	1500	1	0 / 1	$<$ 1%
bashbench	1500	14	9 / 5	1%
iac	1500	80	32 / 48	5%
shade	1500	676	332 / 344	45%

Appendix

Table 7: Number of ControlArena trajectories dropped by the 24K-token Kimi prompt-length cap.

Method	bashbench	agentdojo	iac	rogue_eval	shade
Untuned	0.977	0.731	0.922	0.433	0.479
Grid best ( gpt5.1pro_2 )	0.993	0.990	0.937	0.749	0.784
HPR (round-2 v4)	0.983	0.985	0.949	0.788	0.739
DSPy GEPA	0.994	0.995	0.940	0.655	0.833
SFT (Variant B, median seed)	0.991	0.988	0.943	0.737	0.826

Appendix

Table 8: Per-environment pAUROC for the five Section 5.4 methods.

Panel of Figure 17 — Figure 17: DSPy Automated Prompt Optimization on Gloom. Performance of prompts in train, val, and test, when optimized by GEPA and MIPROv2 optimization on the Gloom dataset, with various initializations.

Panel of Figure 18 — Figure 18: DSPy Automated Prompt Optimization on STRIDE. Performance of prompts in train, val, and test, when optimized by GEPA and MIPROv2 optimization on the STRIDE dataset, with various initializations.

Panel of Figure 19 — Figure 19: Optimization Path for Human Prompt Refinement on STRIDE. (a) When initialized from a minimal baseline prompt, increased optimization pressure yields genuine improvements on the test set. (b) When initialized from the best prompt-sweep prompt, additional optimization yields some improvement on the test set.

AI Agent Monitoring

AI agent monitoring tracks actions, tool use, and execution traces to detect failures, misuse, drift, and policy violations during operation.

See all 280 papers

DeceptGuard :A Constitutional Oversight Framework For Detecting Deception in LLM Agents
Mar 14, 2026Snehasis MukhopadhyayDeception in Language Models AI Agent Monitoring
Indian Institute of Information Technology Kalyani
Agent-ToM: Learning to Monitor Autonomous LLM Agents via Theory-of-Mind Reasoning
May 22, 2026Nesreen K. Ahmed, Nima NafisiLLM Agent Security AI Agent Monitoring
Cisco Outshift
SchemeArena: Factorized Stress Testing of Scheming in LLM Agents
Sep 8, 2026Jie Ruan, Inderjeet Nair, Amy Liu +3LLM Agent Evaluation AI Agent Monitoring
Computer Science and Engineering University of Michigan Ann Arbor, MI 48109, USA

Mar 14, 2026J/K move · Enter open · S save

DeceptGuard :A Constitutional Oversight Framework For Detecting Deception in LLM Agents

Snehasis Mukhopadhyay

Indian Institute of Information Technology Kalyani

Deception in Language Models AI Agent Monitoring LLM Agent Safety CoT Monitoring

Open paper AskPDF arXiv

Reliable detection of deceptive behavior in Large Language Model (LLM) agents is an essential prerequisite for safe deployment in high-stakes agentic contexts. Prior work on scheming detection has focused exclusively on black-box monitors that observe only externally visible tool calls and outputs, discarding potentially rich internal reasoning signals. We introduce DECEPTGUARD, a unified framework that systematically compares three monitoring regimes: black-box monitors (actions and outputs only), CoT-aware monitors (additionally observing the agent's chain-of-thought reasoning trace), and activation-probe monitors (additionally reading hidden-state representations from a frozen open-weights encoder). We introduce DECEPTSYNTH, a scalable synthetic pipeline for generating deception-positive and deception-negative agent trajectories across a novel 12-category taxonomy spanning verbal, behavioral, and structural deception. Our monitors are optimized on 4,800 synthetic trajectories and evaluated on 9,200 held-out samples from DeceptArena, a benchmark of realistic sandboxed agent environments with execution-verified labels. Across all evaluation settings, CoT-aware and activation-probe monitors substantially outperform their black-box counterparts (mean pAUROC improvement of +0.097), with the largest gains on subtle, long-horizon deception that leaves minimal behavioral footprints. We empirically characterize a transparency-detectability trade-off: as agents learn to suppress overt behavioral signals, chain-of-thought becomes the primary detection surface but is itself increasingly unreliable due to post-training faithfulness degradation. We propose HYBRID-CONSTITUTIONAL ensembles as a robust defense-in-depth approach, achieving a pAUROC of 0.934 on the held-out test set, representing a substantial advance over the prior state of the art.

Abstract

AI Agent Monitoring

Explore similar work

DeceptGuard :A Constitutional Oversight Framework For Detecting Deception in LLM Agents

Agent-ToM: Learning to Monitor Autonomous LLM Agents via Theory-of-Mind Reasoning

SchemeArena: Factorized Stress Testing of Scheming in LLM Agents