Reasoning models can remain capable of solving a task while still defaulting to cheaper but misleading shortcuts. This creates a central oversight problem: when a model gives an answer with plausible but incomplete reasoning, can an overseer determine whether that output should be trusted? To study this problem, we introduce PyINE, a framework for scalable elicitation and oversight using instrumented Python programs as a verifiable execution substrate. In PyINE, programs define task environments, execution traces provide authoritative labels for outcomes and intermediate facts, and task variants can be generated mechanically rather than through static human annotation. We instantiate the framework in PyINE-v1, a first release built from nearly one million deterministic execution traces and over 500,000 matched LLM-generated code variants used for counterfactual evaluation. Using standard RL with verifiable rewards on cue-varied tasks, we train a shortcut-following model that improves substantially at predicting execution outcomes while still making systematic errors when misleading human-facing cues conflict with the program's realized behavior. We then evaluate activation probes, trained text classifiers, prompted judges, and a lightweight debate protocol as overseers of this model. We find that performance pooled at the dataset level can hide weak coverage of the failures that matter most: cheap learned overseers often miss rare shortcut-driven errors, while stronger model-based checks are more balanced but substantially costlier and harder to turn into reliable thresholded decisions. PyINE-v1 turns this failure-mode coverage problem into a reusable experimental setting for developing oversight methods that are verifiable, failure-mode-aware, and cost-sensitive.
Figures & tables
Predictor
Family
Original
Helpful
Δhelpful
Misleading
Δmislead
Overall Len.
Qwen3-4B-Instruct [ 101 ]
Base
38.0 ±0.8
95.5 ±0.3
+57.5
6.3 ±0.4
-31.7
213 ±992
↪ Model organism
Shortcut
69.8 ±0.7
98.1 ±0.2
+28.3
10.5 ±0.5
-59.3
866 ±2041
gpt-4o
Reference
38.5 ±0.8
95.9 ±0.3
+57.4
18.9 ±0.6
-19.6
14 ±99
gpt-5-nano (low)
Reference
83.9 ±0.6
94.0 ±0.4
+10.1
77.7 ±0.7
-6.2
778 ±1024
gpt-5-nano
Reference
87.8 ±0.5
94.0 ±0.4
+6.2
84.2 ±0.6
-3.6
2154 ±2604
gpt-5-mini (low)
Reference
92.0 ±0.4
98.7 ±0.2
+6.7
63.8 ±0.8
-28.2
449 ±731
Table 1: Model organism performance (soft-match accuracy) for PyINE-v1 across across original, helpful-hinted, and misleading-hinted task variants (with values reported as mean ± 95% confidence interval), together with Δhelpful:=Helpful−Original and Δmislead:=Misleading−Original , in percentage points. The last column reports average response length across all task variants, in tokens with standard deviation. Various reference models are compared across the same metrics.
Figure 2: AUROC by task variant and data distribution preset across overseer families and models. Bars show cross-replica means with standard-deviation for locally trained models; LLM-based overseers were evaluated only once to minimize inference costs. Colors distinguish task variants (original, hinted, or misleading); shading intensity corresponds to the training/tuning data preset used (weak, moderate, or strong) for locally trained models.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Component
PyINE-v1 summary
Source corpus
Python programming problems and reference solutions drawn from TACO [ 61 ] , filtered to retain programs that can be executed reproducibly under controlled instrumentation. To favor diversity over exhaustive trace coverage, we sample up to 10 solutions per problem and up to 10 test cases per solution, without replacement.
Execution environment
Programs are executed deterministically under sandboxed Python instrumentation. We exclude unsupported dependencies, external I/O, unresolved nondeterminism, and other environment-sensitive behaviors; timeout limits and trace-size caps are applied during collection.
Code and query variants
PyINE-v1 includes original programs together with augmented variants involving obfuscated code, helpful hints, misleading hints, and selectively stubbed code. These variants are used across model-organism construction, shortcut-reliance evaluation, and behavioral robustness analyses.
Release scale
After filtering and validation, PyINE-v1 contains approximately 1M traced executions spanning over 10,000 coding problems and 80,000 candidate solutions, plus over 600,000 augmented code variants for training, counterfactual evaluation, and robustness analyses.
Task scope
This initial release focuses on whole-program execution queries: given a program and an input, the model predicts the final execution outcome. Intermediate execution events and artifacts are recorded and available for supervision and evaluation, but are not yet used as prediction targets in current experiments.
Output verification
Final predictions are scored with execution-grounded matching rules that tolerate minor formatting variation while preserving outcome correctness. The same execution substrate also supports finer-grained verification of reasoning claims against intermediate runtime objects.
Appendix
Table 2: Overview of the PyINE-v1 release, including the execution substrate, task scope, code and query variants, model organism, oversight protocols, and primary evaluation metrics.
Figure 3: Distribution of key statistics across the execution traces of the PyINE-v1 benchmark instantiation: (a) difficulty breakdown of the source problems, per TACO metadata; (b) number of recorded execution steps per trace, which we argue may be a better indicator of code execution task difficulty in Appendix F.3 ; (c) code length of the traced solutions prior to augmentations; (d) number of traces available per problem, reflecting data density across the benchmark.
Artifact
Count
Note
PyINE-v1 trace statistics
Coding problems
10,685
Distinct coding problems retained from TACO [ 61 ] after filtering.
Reference solutions
81,314
Selected solutions across retained coding problems.
Base execution traces
480,323
Original traced executions before augmentation.
Obfuscated execution traces
456,864
95.1% of base traces have an obfuscated counterpart.
Total execution traces
937,187
Total trace count for which full supervision is available.
Appendix
Table 3: Artifact counts for PyINE-v1 after filtering and augmentation. We distinguish between execution traces , which correspond to instrumented runs of programs on concrete inputs, and derived variants , which include query-side LLM-augmented code obtained constructed from those traces.
Figure 5: High-level overview of the PyINE framework: (a) executable Python programs are first instrumented to produce authoritative execution traces and outcome labels which provide diverse task instances and counterfactual variants; (b) task instances are strategically prepared into a bias-inducing data subset to construct model organisms; (c) model organisms are then used to construct overseers using another subset that partially exposes some failure modes (e.g. shortcut-following behavior), simulating challenging overseer training/tuning conditions where all failure modes are not easily discerned; and finally (d) a fully-supported counterfactual data subset is used to report the true performance of overseers against all failure modes.
Figure 7: Distribution of completion tokens (generation length) for the PyINE-v1 original tasks across three models, shown on a logarithmic scale. The base model (Qwen3-4B-Instruct [ 101 ] ) concentrates the vast majority of its outputs near zero tokens (median = 13), producing extremely short responses that rarely include reasoning. After RL training, the shortcut model organism generates substantially longer outputs (median = 743), with a heavy tail extending to the 10 000-token limit, despite the description length penalty. GPT-5, shown as a reference, exhibits a similar long-tailed profile (median = 1 100). Dashed lines mark the median and 95th percentile for each model.
Figure 8: Distribution of generation length (completion tokens) across task variants and answer correctness for the shortcut-following model organism. (a) Comparison across task variants: the original tasks elicit substantially longer generations than the helpful and misleading variants, whose length distributions are similar. (b) Comparison by answer correctness (soft match): incorrect answers tend to have a wider spread and higher tail percentiles. (c) Joint breakdown by correctness and task variant, showing that the correctness gap in generation length is most pronounced for the original variant (in absolute numbers).
Figure 9: Shortcut-following model organism training progression across RL checkpoints (validation set). (a) Soft-match accuracy by code type; dashed lines indicate base model performance. (b) Mean response length in tokens ( ±1 std). (c) Efficiency ratio (accuracy per 1K tokens) for original tasks only. The step-600 checkpoint (vertical dashed line in (c)) is selected for downstream experiments.
Figure 10: Shortcut-following model organism prediction accuracy (soft match) as a function of code and task complexity for the original task variant in PyINE-v1. Each subplot shows individual samples (green = correct, red = incorrect) with a rolling-mean accuracy curve and 95% confidence band. All x-axes are clipped to the 1st–99th percentile range to exclude outliers. Accuracy degrades monotonically with increasing code complexity across all Halstead metrics and SLOC, while maintainability index — an inverse complexity measure — exhibits the expected positive trend. Among task complexity indicators, trace step count shows the steepest accuracy decline, dropping below 50% for programs exceeding 1 000 steps. Program input argument length and expected output length exhibit a similar pattern. Completion tokens mirror this trend more smoothly, suggesting that harder problems both require and elicit longer generations, consistent with past observations [ 32 , 22 ] .
Predictor
Pass@1
Pass@5
Pass@10
Maj.V.
Divers.
% Miss.
Len.
Qwen3-4B-Instruct [ 101 ]
38.0 ±2.2
48.0 ±2.4
50.9 ±2.5
36.5 ±2.4
24.0 ±1.1
0.9 ±0.2
567 ±1553
↪ Shortcut model organism
69.8 ±2.0
82.2 ±1.8
85.3 ±1.8
68.8 ±2.3
27.9 ±1.4
4.9 ±0.3
2153 ±2931
gpt-5
93.0 ±1.2
95.2 ±1.0
95.7 ±1.0
93.2 ±1.3
11.0 ±0.2
5.7 ±0.4
2260 ±2707
gpt-5 (low)
94.2 ±1.0
96.4 ±0.9
96.7 ±0.9
94.9 ±1.1
12.8 ±0.5
1.5 ±0.2
1228 ±1769
gpt-5-mini
94.0 ±1.1
96.0 ±1.0
96.2 ±1.0
94.4 ±1.2
12.0 ±0.4
3.1 ±0.3
1523 ±2204
gpt-5-mini (low)
92.0 ±1.2
95.7 ±1.0
96.1 ±1.0
92.6 ±1.3
14.7 ±0.8
0.3 ±0.1
782 ±1037
Appendix
Table 4: Code execution predictor capability metrics for the original tasks of PyINE-v1. Pass@k scores are reported as mean percentages following [ 15 ] based on prediction soft-matching. Majority vote (Maj.V.) reports the percentage of tasks for which a strict majority of attempts are correct. Output diversity (Divers.) reports the percentage of distinct predictions generated across all attempts, averaged across all tasks. The missing answer ratio (% Miss.) is the percentage of attempts where a final answer is missing (likely due to truncation). The mean generated response length (Len.) is reported in tokens, with standard deviation. All other metrics are reported with a 95% CI.
Figure 11: AUROC by task variant and data distribution preset across overseer families and models. Bars show cross-replica means with standard-deviation for locally trained models; LLM-based overseers were evaluated only once to minimize inference costs. Colors distinguish task variants (original, hinted, or misleading); shading intensity corresponds to the training/tuning data preset used (weak, moderate, or strong) for locally trained models.
Figure 12: Butterfly histograms showing the distribution of predicted correctness scores for different baseline oversight methods. An ideal score distribution would be one showing a diagonal starting from the bottom left (i.e. a red bar peak near 0) and ending at the top right (green bar peak near one). The different columns show performance across task variants. Rows show representative or interesting configurations across different oversight families. Trained overseer score distributions aggregate predictions across all replicas.
Table 5: Trained activation probe sensitivity (TPR, %) at fixed false-positive rate budgets, reported per task variant. We report the probe configurations that resulted in the best performance. Values are means ± 95% confidence intervals obtained with an SEM-based Gaussian clamped to [0,1] . Δhelpful and Δmislead. denote the change from the original variant to helpful-hinted and misleading-hinted variants respectively, in percentage points. The threshold-free AUROC is given for context, and the rightmost column provides the estimated compute tax of the overseer relative to the target predictor. Higher is better throughout, except for the compute tax.
Table 6: Trained LLM classifier sensitivity (TPR, %) at fixed false-positive rate budgets, reported per task variant. Values are means ± 95% confidence intervals obtained with an SEM-based Gaussian clamped to [0,1] . Δhelpful and Δmislead. denote the change from the original variant to helpful-hinted and misleading-hinted variants respectively, in percentage points. The threshold-free AUROC is given for context, and the rightmost column provides the estimated compute tax of the overseer relative to the target predictor. Higher is better throughout, except for the compute tax.
Table 7: Judge overseer sensitivity (TPR, %) at fixed false-positive rate budgets, reported per task variant. Values are means ± 95% confidence intervals obtained with problem-clustered bootstrapping. Δhelpful and Δmislead. denote the change from the original variant to helpful-hinted and misleading-hinted variants respectively, in percentage points. The threshold-free AUROC is given for context, and the rightmost column provides the estimated compute tax of the overseer relative to the target predictor. Higher is better throughout, except for the compute tax.
Table 8: Debate interrogator sensitivity (TPR, %) at fixed false-positive rate budgets, reported per task variant. Values are means ± 95% confidence intervals obtained with problem-clustered bootstrapping. Δhelpful and Δmislead. denote the change from the original variant to helpful-hinted and misleading-hinted variants respectively, in percentage points. The threshold-free AUROC is given for context, and the rightmost column provides the estimated compute tax of the overseer relative to the target predictor. Higher is better throughout, except for the compute tax.
Figure 13: Estimated oversight cost versus performance (AUROC) for representatives of each evaluated oversight method family. Each method appears as a linked triplet of markers corresponding to original, helpful-hinted, and misleading-hinted task variants. Methods with all three markers closer to the lower-right corner achieve stronger detection at lower cost.
Reasoning in Large Language Models (LLMs) poses a challenge for oversight as many misaligned behaviors do not surface until reasoning concludes. To address this, we introduce Behavior Cue Reasoning for making LLM reasoning more controllable and monitorable. Behavior Cues are special token sequences that a model is trained to emit immediately before specific implicit and explicit behaviors, acting as dual purpose signal and control levers. When fine-tuning a weaker external monitor with Reinforcement Learning for reasoning oversight, a compressed view of only information surfaced by Behavior Cues is sufficient signal for the monitor to prune up to 50% of otherwise wasted reasoning tokens in complex math problem solving. When leveraged by an almost optimal rule-based monitor in an environment where excessive constraint violations results in failure, Behavior Cues allows for the recovery of safe actions from 80% of reasoning traces that would otherwise end with the proposal of an unsafe action, more than doubling the success rate from 46% to 96%. Through evaluation across two model families and three domains, we show that Behavior Cue Reasoning improves reasoning monitorability and controllability with no cost to performance. More broadly, our work progresses scalable oversight by demonstrating how the monitored model itself can be trained to reason more tractably to oversight. Code: https://github.com/christopherzc/behavior-cues
Christopher Z. Cui, Taylor W. Killian, Prithviraj Ammanabrolu
University of California, San Diego · 2Brigham Young University
Verification is becoming central to both reinforcement-learning-based training and inference-time control of large language models (LLMs). Yet current verifiers face a fundamental trade-off: LLM-based verifiers are expressive but hard to control and prone to error, while deterministic executable verifiers are reliable and interpretable but often limited in capability. We study the following question: given a development set of LLM outputs and labels for a target objective, such as correctness, can we automatically induce a minimal set of Python verifiers whose joint satisfaction closely matches that objective? We propose AutoPyVerifier, a framework that uses an LLM to synthesize candidate verifier functions and then refines them through search over a directed acyclic graph (DAG). By navigating the DAG, AutoPyVerifier systematically explores the space of deterministic executable verifiers and selects a compact verifier set whose joint satisfaction best approximates the target objective. Across mathematical reasoning, coding, function calling, and instruction-following benchmarks for several state-of-the-art LLMs, AutoPyVerifier improves target-objective prediction by up to 55.0 F1 points over the initial LLM-generated verifier sets. Additional analyses show that the most useful verification targets vary by benchmark and model, and that the DAG-based search shifts the learned verifier sets toward more structural and semantically grounded checks. We further show that exposing the discovered verifier set to an LLM as an external tool improves downstream accuracy by up to 17.0 points. We release our code
Chain-of-thought (CoT) controllability, the ability of reasoning models to control properties of their CoT, is consequential for both their usefulness and reliable oversight. Recent work evaluates this capability by measuring whether models follow CoT constraints given in the user prompt, and reports very low controllability across models. We argue that this low controllability may not reflect an inherent inability of reasoning models to control their CoT, but instead result from naively specifying constraints in the user prompt. To test this, we introduce Self-Instructed CoT Controllability: a complementary evaluation that places the constraint as a self-instruction inside the model's CoT and measures its ability to follow it. We show that, under self-instruction, models achieve significantly higher controllability than with standard user instructions. Even a generic reminder to follow the user's constraint substantially improves controllability. Moreover, effective self-instructions can be bootstrapped into the reasoning through a simple instruction in the user prompt, enabling improved control even without access to the reasoning channel. Finally, using self-instruction, we demonstrate in a realistic monitoring setup that CoT controllability can be leveraged for monitor evasion, posing a risk to CoT monitorability.