Myopic Optimization with Non-myopic Approval (MONA) mitigates multi-step reward hacking by restricting the agent's planning horizon while supplying far-sighted approval as a training signal~\cite{farquhar2025mona}. The original paper identifies a critical open question: how the method of constructing approval -- particularly the degree to which approval depends on achieved outcomes -- affects whether MONA's safety guarantees hold. We present a reproduction-first extension of the public MONA Camera Dropbox environment that (i)~repackages the released codebase as a standard Python project with scripted PPO training, (ii)~confirms the published contrast between ordinary RL (91.5% reward-hacking rate) and oracle MONA (0.0% hacking rate) using the released reference arrays, and (iii)~introduces a modular learned-approval suite spanning oracle, noisy, misspecified, learned, and calibrated approval mechanisms. In reduced-budget pilot sweeps across approval methods, horizons, dataset sizes, and calibration strategies, the best calibrated learned-overseer run achieves zero observed reward hacking but substantially lower intended-behavior rates than oracle MONA (11.9% vs.\ 99.9%), consistent with under-optimization rather than re-emergent hacking. These results operationalize the MONA paper's approval-spectrum conjecture as a runnable experimental object and suggest that the central engineering challenge shifts from proving MONA's concept to building learned approval models that preserve sufficient foresight without reopening reward-hacking channels. Code, configurations, and reproduction commands are publicly available. https://github.com/codernate92/mona-camera-dropbox-repro
Figures & tables
Figure 1: Taxonomy of AI alignment failure modes, highlighting where MONA and this extension sit. Blue-highlighted nodes indicate the specific failure modes addressed in this work: multi-step reward hacking (the core problem MONA mitigates) and learned-overseer fragility (the central question our extension investigates). The dotted link shows that Camera Dropbox bridges specification problems (reward hacking) and optimization problems (sensor manipulation / reward tampering). This taxonomy is simplified; real systems may exhibit multiple failure modes simultaneously, and categories interact in ways not fully captured by a tree structure.
Figure 2: Standard RL vs. MONA in Camera Dropbox. Under standard RL (left), the agent optimizes environment reward over the full horizon and learns sensor tampering (91.5% reward-hacking rate). Under MONA (right), myopic optimization restricts the agent’s planning horizon while non-myopic approval provides a training signal aligned with intended behavior (0.0% hacking rate). Numbers are from the released public reference arrays [ 6 , 8 ] .
Aspect
Original MONA paper
This extension
Primary aim
Introduce MONA and test whether myopic optimization plus non-myopic approval mitigates multi-step reward hacking across three model organisms [ 6 ] .
Reproduce the public Camera Dropbox release, script the PPO path, and extend it with modular learned-approval experiments [ 8 ] .
Environment scope
Test-driven development, loan applications, Camera Dropbox [ 6 ] .
Public Camera Dropbox ( 4×4 , 2 boxes, 50 steps) plus a harder variant with more boxes and longer step limit [ 8 ] .
Public reference confirms the original contrast; learned-approval pilot limited by under-optimization, not re-emergent hacking [ 8 ] .
Table 1: Comparison of the original MONA work and the reproduction-first extension. Cells marked original reflect claims from [ 6 ] ; cells marked extension reflect this work [ 8 ] .
Figure 3: Conceptual approval-reward construction spectrum from Appendix B.3 of [ 6 ] . Safer constructions (left) are more restrictive and less outcome-dependent; moving right toward achieved-outcome dependence degrades safety until MONA recreates ordinary RL (point 6). The bracket indicates the region addressed by this extension: oracle MONA, noisy and misspecified oracles, and learned outcome classifiers.
Figure 4: Reproduction-first workflow. The project starts from the public MONA Camera Dropbox release, repackages it as a Python project, adds modular learned-approval mechanisms, and evaluates safety–capability tradeoffs across the approval-construction space.
Metric
Ordinary RL
Oracle MONA
Best Learned
(public ref.)
(public ref.)
(local pilot)
Reward-hacking rate
0.915
0.000
0.000
Intended-behavior rate
0.077
0.999
0.119
Failure rate
0.007
0.001
—
True return
—
—
−0.363
Table 2: Key metrics for the three main comparison points. Public ref. values are from the released MONA arrays [ 8 ] . Best learned is the best calibrated learned-overseer run from the local pilot [ 8 ] .
Figure 5: Behavior-rate comparison across the three main conditions. The public reference preserves the original MONA contrast: ordinary PPO heavily reward-hacks while oracle MONA almost entirely performs the intended behavior. The best local calibrated learned-overseer run shows zero observed hacking but far lower intended-behavior rates, consistent with under-optimization rather than capability recovery [ 8 ] .
Domain
Proxy metric
“Camera”
Approval challenge
Healthcare
Diagnostic coverage, discharge speed
Patient outcome metrics, chart audits
Clinician-informed approval over high-dimensional medical states
Autonomous vehicles
Benchmark pass rate
Safety evaluation suite
Defining approval for complex, long-tail driving scenarios
Content recommendation
Click-through rate, watch time
User satisfaction surveys
Capturing “genuine user value” vs. engagement proxies
Financial risk
Sharpe ratio, VaR
Risk monitoring dashboard
Distinguishing reduced measured risk from hidden tail exposure
Scientific automation
Statistical significance
Peer review, replication checks
Process-based approval (pre-registration) vs. outcome-based (significance)
Cybersecurity
Detection/prevention rate
Intrusion detection system
Balancing false-positive suppression against adversarial evasion
Table 3: Proposed risk domains for MONA-style model organisms, mapped to the structural elements of Camera Dropbox. Each domain exhibits multi-step reward hacking through a proxy metric, a “camera” (monitoring system) that can be manipulated, and a core approval-construction challenge.
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.
Reward hacking is a major problem in current production machine learning systems, resulting in costly and dangerous failures. Here we study how to monitor for and mitigate reward hacking in realistic settings, including tool calls and long contexts. We apply a dynamic intervention to steering directions obtained by three different methods: difference-in-means vectors from labeled rollouts, gradient optimization-based vectors, and vectors extracted when models reference an anti-reward-hacking model spec. This dynamic intervention perturbs model representations less than uniform steering vector addition to the residual stream, allowing for more precise edits. We find that we are able to cheaply and significantly reduce reward hacking without correspondingly significant costs to model coherence or capability, and that steering vectors originally extracted from coding environments also generalize to reducing reward hacking in other domains. Our results pave the way towards latent space monitoring and control over realistic production systems.
Reward hacking in code generation, where models exploit evaluation loopholes to obtain high reward without correctly solving the intended task, poses a critical challenge for Reinforcement Learning (RL) and the deployment of reasoning models. Existing studies often rely on explicitly prompted hacking trajectories, but it remains unclear whether monitors trained on such data can detect reward hacks that arise without direct hacking instructions during RL training. In this work, we introduce Trace-and-Amplify, a framework for scalable curation of reward-hacking trajectories that arise during RL training without explicit hacking instructions. The framework uses unit-test tracers to identify hacking solutions when they occur and retains such trajectories for monitor training and evaluation. Through controlled comparisons between monitors trained on prompt-elicited hacking trajectories and training-time reward-hacking trajectories collected by Trace-and-Amplify, we find that (1) prompt-elicited-data-trained monitors often fail to generalize to trajectories curated by our framework, and (2) monitors trained on our Trace-and-Amplify trajectories demonstrate stronger generalizability to unseen hacking types. Our results indicate that prompted reward hacking data may not fully reflect training-time reward-hacking behaviors, and that relying solely on these data can lead to misleading conclusions. Codebase is available at https://github.com/LichenLillc/CoTMonitoring.git
Lichen Li, Hengguang Zhou, Yijun Liang +2
Peking University · University of California, Los Angeles · University of Maryland +2
Reinforcement learning agents can exploit misspecified reward signals to achieve high apparent returns while failing on the intended objective, a failure mode known as reward hacking. Existing practical defenses typically constrain policy updates to stay near a known safe reference, creating a tension between suppressing hacking and permitting legitimate improvement. We propose Modification-Considering Value Learning (MCVL), which operationalizes the theoretical idea of current utility optimization for standard value-based RL. MCVL wraps an off-policy learner and treats each incoming transition as a candidate modification: it forecasts two training paths, one that includes the transition and one that does not, and scores both with a frozen bootstrapped-return estimator derived from a learned reward model and value function. The transition is admitted only if inclusion does not decrease the score. We formalize conditions under which this filtering is both safe and permissive, and instantiate MCVL with DDQN and TD3. Across four safety-relevant gridworlds and three modified MuJoCo continuous-control tasks with diverse hacking mechanisms, MCVL mitigates reward hacking while continuing to improve the intended objective. Project website: ktolnos.github.io/mcvl/.