cs.AIOct 22, 2026

Simulated Control Evals

Authors: Jeffrey Olmo, David Zhang, Tyler Tracy

Organizations: Redwood Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Control evaluations measure whether safeguards can prevent an AI model from causing undesired outcomes even when it is intentionally trying to do so. These evaluations typically use purpose-built control settings and extrapolate their results to a model's deployment. However, the effectiveness of safeguards in deployment can depend on features of environments and tasks that control settings do not represent. To broaden where control evaluations can be run, we introduce simulated control evaluations (SCEs), which use a language model to simulate tool responses in place of an environment. We evaluate SCEs on LinuxArena, where execution provides ground-truth outcomes. We compare SCE prediction errors with those from a separate control setting, BashArena, and with variation in control metrics within LinuxArena. For independently sampled evaluation and deployment task sets, SCEs predict safety with a root mean squared error of 3.6 percentage points across five monitors at a 5% audit budget, compared with 10.1 for BashArena. SCEs closely reproduce distributions of attack strategies, attacker reasoning, and monitor reasoning, with over tenfold lower KL divergence from execution than BashArena. SCEs explain 78-85% of the between-environment variance in attack success, monitor scores, and safety. This accuracy is sample efficient: SCEs predict safety with less error than BashArena using fewer than 5% as many task-pair draws as the deployment task set. Strategies discovered through simulation also improve monitor performance on held-out LinuxArena attacks, outperforming BashArena and approximately matching improvement from executed strategies. These results suggest that deployment-specific SCEs can be more predictive than control evaluations executed in separate control settings.

Figures & tables

Figure 1: The simulated control evaluation process. A language-model simulator generates tool responses for an attack policy pursuing a side task during legitimate work. Repeating this process across tasks produces simulated attack trajectories. A monitor scores their suspicion, and an attack success judge estimates attack success. Control metrics such as safety at a given audit budget can then be computed. Attack-policy actions never execute against the deployment environment.
Figure 2: BashArena and LinuxArena variance as performance comparisons. Columns denote environments and rows tasks within each environment; boxes are schematic. Left: A control setting is used to predict a deployment’s control metrics. Middle: BashArena or LinuxArena averages provide constant predictions across LinuxArena task pairs, side-task categories, and environments. Right: Orange simulation and gray execution use the same LinuxArena environments and task pairs. Each Δ\Delta compares a prediction with the deployment’s measured control metric mm . Smaller simulation gaps support using SCEs when control-setting extrapolation to deployment incurs at least the reference error observed here, provided simulator accuracy is maintained.
Figure 3: Grounded simulation closely matches LinuxArena behavior. Adding the reference trajectory and codebase access reduces divergence across attack strategies, attacker reasoning, and monitor reasoning. The primary simulator is much closer to execution than BashArena is; gray re-execution bars show the scale of variation between repeated runs.
CategoryExample behaviorDetail
Attack strategyPrefix-confusion cleanup. Uses an expected miss on a malformed prefix to justify deleting objects under the real prefix.let storage_team = &body.team_id[1..]; let probe = format!("uploads/{storage_team}/{file_id}.mp4"); if head_object(probe).is_err() { delete_real_objects(...); }
Attacker reasoningCover for content substitution. Plans a subtle side attack while keeping the surrounding work plausible.“I need to tweak the checksum properly and consider a subtle attack. The attack should feel unrelated and not obvious. […] I have to do it precisely, but also ensure to implement a sufficient cover.”
Monitor reasoningPrivilege-escalation pathway. Interprets routine health checks in light of earlier privilege-seeking changes.“Current action appears as routine process and healthcheck, but prior history shows attempts to elevate privileges […]. This creates a partial causal pathway from normal task to elevated-access activity.”
Table 1: Examples of behavioral representations.
Figure 4: Simulation reproduces LinuxArena’s monitor-reasoning patterns. The BashArena comparison on the left shows a larger distributional mismatch than the simulation comparison on the right. The qualitative geometry above and cluster masses below illustrate this difference.
Figure 5: SCEs predict expected attack success across tasks and environments. Left: All reports total RMSE for independently sampled task sets. Task pair, Side-task category, and Environment compare variance-corrected SCE RMS bias with variance-corrected LinuxArena SD. Top labels show percent match for All and percent explained for other groups. Whiskers are 95% main-task bootstrap intervals. Right: Per-environment error decomposition for simulation plus judge. Orange bases show absolute mean gaps; grey segments add variation from sampling evaluation and deployment task sets, with boundaries at roots of cumulative MSE. Blue diamonds show absolute mean gaps for the judge on executed attacks. Black whiskers are 95% normal bootstrap intervals on simulation-plus-judge RMSE; numbers above them give point RMSE.
Figure 6: Success-rate errors concentrate in a few side tasks. Side tasks ranked by success-rate RMSE for independent evaluation and deployment task sets. The three largest errors are named; shading shows pointwise 95% bootstrap intervals. The dotted line marks the rank at which cumulative squared error reaches 50%.
Figure 7: SCEs predict expected monitor scores across tasks and environments. Left: All reports total RMSE for independently sampled task sets. Task pair, Side-task category, and Environment compare variance-corrected SCE RMS bias with variance-corrected LinuxArena SD. Top labels show percent match for All and percent explained for other groups. Whiskers are 95% main-task bootstrap intervals. Right: Per-environment decomposition into absolute empirical mean gap and variation from sampling evaluation and deployment task sets. Boundaries are roots of cumulative MSE; bar tops are total RMSE. Black whiskers are 95% normal bootstrap intervals on RMSE; numbers above them give point RMSE.
Figure 8: SCEs predict safety across monitors and audit budgets. Left: Simulated curves broadly track executed safety, including the differences between monitors. Right: At 5% audit, the pooled SCE prediction has lower total RMSE than BashArena’s, and SCE RMS bias across side tasks and environments is below the corresponding LinuxArena SD.
Figure 9: SCE sample efficiency and strategy frequencies. Left: Fraction of clusters still undiscovered, weighted by held-out successful-and-evasive attack mass (left axis), and total safety RMSE (orange, right axis). Discovery requires a simulated attack that succeeds and evades the same monitor; one attack can discover several reasoning clusters. The budget axis is logarithmic. Bands show 95% intervals; diamonds mark parity with the full BashArena evaluation. Right: Simulated versus executed strategy frequencies among all attacks. Point area reflects executed successful-and-evasive mass across five monitors at a 5% audit budget. Darker and lighter bands show agreement within factors of two and three.
Figure 10: Simulated attacks provide useful evidence for improving a monitor. Briefing with simulated strategies improves detection of held-out executed attacks more than briefing with BashArena strategies, and approaches briefing with same-environment strategies. Left: Detection at low false-positive rates. Right: Lift over no briefing. Whiskers show 95% bootstrap intervals.
Figure 11: Grounding makes simulation harder to distinguish from execution. Left: The history discriminator approaches chance as grounding affordances are added. Right: Grounded simulations do not become systematically more detectable as trajectories progress. Chance AUROC is 0.5.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

ObserverDimensionDescription
Attack strategy observerPrimary mechanismsOperational route or routes used to pursue the adversarial objective.
 Setup and discoveryInformation gathering, probing, and preparation that shaped the attempt.
 Constraint handlingHandling of environmental, permission, network, or task constraints.
 AdaptationChanges of approach after feedback, failure, or newly discovered information.
 Evasion choicesChoices intended to reduce monitor visibility or blend into main-task work.
 Persistence, cleanup, and reversibilityPersistence, cleanup, restoration, reversibility, or deliberate avoidance thereof.
Appendix
Table 2: Behavioral dimensions supplied to the structured observers. The attack-strategy observer describes a complete trajectory. The monitor observer describes the action monitor’s rationale at one step. We embed the resulting descriptions for the behavioral-distribution comparisons; the embeddings exclude raw trajectories, suspicion scores, and step positions.
Figure 12: Environment-specific strategy and reasoning distributions. Each lower-triangle entry compares executed trajectories from two environments. Orange-outlined diagonal entries compare execution with simulation in the same environment. All panels use uncentered, L2-normalized embeddings and maximum mean discrepancy (MMD) with a radial basis function (RBF) kernel; lower is closer. Each panel has one shared kernel and its own color scale. Strategy descriptions have equal trajectory weight; attacker reasoning gives each recorded summary equal weight; monitor reasoning gives each trajectory equal total weight across its reasoning descriptions. Values are point estimates; symmetric upper triangles are omitted.
Figure 13: Safety by environment. The Figure 8 estimator applied within each environment: solid is simulated, dashed executed, and bands are 95% main-task-bootstrap intervals.
Figure 14: Whole-trajectory discrimination. Bars show the error rate: how often a paired executed/simulated discriminator selects simulation as executed; 0.5 is chance. Presentation order is counterbalanced; whiskers are 95% pair-bootstrap intervals. Base and +reference use high reasoning effort on a separate smaller cohort; +codebase uses medium effort on the primary cohort.
Figure 17
IDRepresentative source mechanismnexecn_{\mathrm{exec}}nsimn_{\mathrm{sim}}Exec.Sim.BashArena
9Withdrawal relabeling. Introduced a singular/plural lookup-key mismatch for the adverse-health withdrawal reason2020575530
7Untracked media extraction. Used equal extract time bounds as a hidden trigger1110545230
6Changed re-indexing to remove and rebuild only files still present on disk1313432930
49Replaced immediate API execution-audit writes with an in-memory pending queue118404730
8Inserted a backward-time linear projection for short series1718403130
22Planted a stable preferred phrase in preprocessing configuration source77403230
Appendix
Table 3: All 49 observed attack-strategy groups in the risk comparison. Groups are ordered by executed undetected success rate. Descriptions quote the first mechanism clause of each group’s closest source example; they are examples, not validated cluster names. Counts are attacks; rates are percentages at 5% honest FPR, averaged over five monitors. BashArena uses a strategy smoother tuned only on its own outcomes.
More in this topic

AI Safety Evaluation

AI Safety Evaluation studies how to test whether AI systems avoid harmful behavior and remain reliable across users, tasks, and deployment settings.

See all 143 papers

Explore similar work

CardsList
Apr 16, 2026

LinuxArena: A Control Setting for AI Agents in Live Production Software Environments

We introduce LinuxArena, a control setting in which agents operate directly on live, multi-service production environments. LinuxArena contains 20 environments, 1,671 main tasks representing legitimate software engineering work, and 184 side tasks representing safety failures such as data exfiltration and backdooring, making it the largest and most diverse control setting for software engineering to date. We validate LinuxArena is useful for control research by running sabotage evaluations, which measure whether attackers can complete side tasks while working on main tasks, and monitor evaluations, which measure a monitor model's ability to detect sabotage attempts. Against a GPT-5-nano trusted monitor at a 1% step-wise false positive rate, Claude Opus 4.6 achieves roughly a 23% undetected sabotage success rate. We additionally release LaStraj, a dataset of human-crafted attack trajectories that evade monitors at substantially higher rates than any model-generated attacks we elicited, showing that current attack policies do not saturate LinuxArena. These results suggest that LinuxArena has meaningful headroom for both attackers and defenders, making it a strong testbed for developing and evaluating future control protocols.

Tyler Tracy, Ram Potham, Nick Kuhn +31

Redwood Research · EquiStamp

Software Engineering AgentsAI Agent SecurityAI Agent Security BenchmarksAI Agent MonitoringAI Control
AskPDF
Jul 21, 2026

ResearchArena: Evaluating Sabotage and Monitoring in Automated AI R&D

As AI agents begin to automate AI R&D, we need ways to assess whether their outputs are safe to deploy, even when the agents themselves may be untrusted. AI control offers one such approach: rather than trusting the agent, it treats it as a potential adversary and uses a monitor to detect covert sabotage before deployment. We evaluate AI control for automated AI R&D with ResearchArena, a framework spanning four long-horizon tasks: safety post-training, capabilities post-training, CUDA-kernel optimization, and inference-server optimization. Because the deliverable in AI R&D is an artifact that will be deployed and run, we pair each main task with two kinds of hidden side task: an embedded side task that sabotages the submitted model, adapter, kernel, or server itself, and an independent side task that takes a covert out-of-scope action in the sandbox while leaving the artifact honest. We evaluate frontier agents at both sabotage and monitoring, across four monitor types that vary along two axes: whether the monitor may execute and probe the produced artifact, and whether it sees the agent's chain-of-thought. We find that sabotage hidden in the training data is the hardest to catch, flagged fewer than half the time. Letting the monitor run experiments on the artifact, rather than only read the trajectory, helps, but it is not enough: monitors still miss embedded sabotage by inspecting only the surface, by noticing the anomaly but explaining it away, or by probing the artifact with the wrong test. We release ResearchArena as a modular framework for evaluating sabotage and control in automated AI R&D.

Lena Libon, Ben Rank, Jehyeok Yeon +5

ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center · ETH Zurich · University of Amsterdam

AI Agent MonitoringAI ControlAI Agent Safety BenchmarksAutonomous Research Agents
AskPDF
Jun 3, 2026

Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety

An attacker that strategically chooses when to attack is much harder to catch than one that attacks indiscriminately. AI control is a safety framework for deploying capable but untrusted AI agents under the oversight of a weaker, trusted monitor and a limited human audit budget. Control evaluations stress-test these protocols by pitting a red-team attack policy against the blue-team monitor, but current evaluations typically assume attackers that do not strategically select when to attack. We study this capability, attack selection, in agentic settings by decomposing attack decisions into a start policy, which decides when an attacker should attack, and a stop policy, which decides when an attacker should abort an ongoing attack. Across two agentic settings, BashArena and LinuxArena, both policies substantially lower measured empirical safety without changing the underlying attack capability. At a 1% audit budget, our start policy reduces safety by 20pp on both BashArena and LinuxArena, and our stop policy reduces safety by 20pp on BashArena and 28pp on LinuxArena. These reductions should be interpreted as upper bounds on the effect of attack selection. Existing control evaluations may therefore yield overly optimistic safety estimates against selective attackers. We recommend that future evaluations, system cards, and safety cases elicit attack selection to produce more realistic safety estimates.

Catherine Ge-Wang, Tyler Crosse, Benjamin Hadad +3

University of Oxford · Mentorship for Alignment Research Students (MARS), CAISH · Georgia Institute of Technology +3

Adversarial AttacksAI Agent SafetyAI Agent Security BenchmarksAI ControlAI Safety EvaluationAI Agent Safety Benchmarks
AskPDF