Synthetic Monitoring Environments for Reinforcement Learning

Leonard Pleiss; Carolin Schmidt; Maximilian Schiffer

cs.LGMar 6, 2026

Synthetic Monitoring Environments for Reinforcement Learning

Authors: Leonard Pleiss, Carolin Schmidt, Maximilian Schiffer

Organizations: Technical University Munich Munich, 80331

Abstract

Reinforcement Learning (RL) lacks benchmarks that enable precise, white-box diagnostics of agent behavior. Current environments often entangle complexity factors and lack ground-truth optimality metrics, making it difficult to isolate why algorithms fail. We introduce Synthetic Monitoring Environments (SMEs), an infinite suite of continuous control tasks. SMEs provide fully configurable task characteristics and known optimal policies. As such, SMEs allow for the exact calculation of instantaneous regret. Their rigorous geometric state space bounds allow for systematic within-distribution (WD) and out-of-distribution (OOD) evaluation. We demonstrate the framework's benefit through multidimensional ablations of PPO, TD3, and SAC, revealing how specific environmental properties - such as action or state space size, reward sparsity and complexity of the optimal policy - impact WD and OOD performance. We thereby show that SMEs offer a standardized, transparent testbed for transitioning RL evaluation from empirical benchmarking toward rigorous scientific analysis.

Figures & tables

Figure 1: Topological deformation of the input space as a function of the number of stacked uniform layers, $\mathcal{C}_{\pi^{\star}}$ .

Figure 2: Ablations for PPO, SAC and TD3 across different task configurations over $10$ seeds. Curves indicate median evaluation performance, smoothed over $10$ points. Shaded areas indicate interquartile ranges.

Figure 3: Evaluation performance during training (column 1) and final within-distribution and out-of-distribution performance for PPO, SAC and TD3 across different complexities of the optimal policy the agent seeks to mirror ( $\mathcal{C}_{\pi^{\star}}$ , columns 2-4). Performance is defined as the complement of the mean average error between action and optimal action $\tilde{r}_{t}$ . States within $\mathbb{R}^{2}(0,1)$ are within-distribution. States beyond the unit square are out-of-distribution. SD = State Dimension.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Performance by proximity to the training distribution for PPO, SAC and TD3 across 10 seeds. Alg. = Algorithm, WD = Within-distribution, OOD = Out-of-distribution, MAE = Mean Average Error.

Figure 5: Optimal action distribution under varying input dimensions $N_{s}$ and complexities $\mathcal{C}_{\pi^{\star}}$ for $10{,}000$ states, uniformly sampled from the unit hypercube that bounds the state space. AD = Action dimension.

$\nu$	$\mathcal{C}_{\pi^{\star}}$	BP	BC	IQL
0.0	1	$1.000\pm 0.00$	$0.998\pm 0.00$	$0.997\pm 0.00$
0.0	10	$1.000\pm 0.00$	$0.979\pm 0.01$	$0.979\pm 0.00$
0.0	50	$1.000\pm 0.00$	$0.803\pm 0.07$	$0.798\pm 0.07$
0.1	1	$0.982\pm 0.01$	$0.981\pm 0.01$	$0.981\pm 0.01$
0.1	10	$0.979\pm 0.01$	$0.971\pm 0.01$	$0.970\pm 0.01$
0.1	50	$0.983\pm 0.01$	$0.801\pm 0.07$	$0.799\pm 0.07$

Appendix

Table 1: Performance comparison in

\tilde{r}_{t}

across noise levels and policy complexities (BP = Behavior policy, BC = Behavior cloning, IQL = Implicit Q-Learning.

Figure 6: Improvement over the behavioral policy for Behavior Cloning (BC) and Implicit Q-Learning (IQL) by noise level $\nu$ and optimal policy complexity $\mathcal{C}_{\pi^{\star}}$ .

Figure 7: Ablations for PPO, SAC and TD3 across different task complexities over $10$ seeds. Curves indicate median evaluation performance, smoothed over $10$ points. Shaded areas indicate interquartile ranges.

RL Benchmarks

Reinforcement learning benchmarks provide tasks and evaluation protocols for comparing agents across control, games, robotics, and other decision-making settings.

See all 85 papers

Synthetic Sandbox for Training Machine Learning Engineering Agents
Apr 6, 2026Yuhang Zhou, Lizhu Zhang, Yifan Wu +4LLM Agent Training RL for LLM Agents
Meta AI
OOD-RL-Bench: A Benchmark Framework for Out-of-Distribution Detection in Reinforcement Learning
Jul 14, 2026Emil Mittag, Richard Dazeley, Peter VamplewRL Benchmarks Online Anomaly Detection
Deakin University, Geelong, Victoria, 3220, Australia · Federation University Australia, Ballarat, Victoria, 3350, Australia
Ground-Truth-First Benchmarking of Reinforcement Learning via Converse Optimality: From Smooth, Fully Observed Systems to Non-Smooth or Partially Observed Tasks
Oct 22, 2026Grégoire Ouerdane, Sinan Ibrahim, Stefan Streif +1RL Benchmarks Robust RL
Center for Engineering Systems and Sciences, Skoltech, 30 Bolshoy Boulevard, bld. 1, Moscow, 121205, Russia · Central University, 7 Gasheka St., bld. 1, Moscow, 123056, Russia · Technische Universität Chemnitz, Straße der Nationen 62, D-09111 Chemnitz, Germany +1

Apr 6, 2026J/K move · Enter open · S save

Synthetic Sandbox for Training Machine Learning Engineering Agents

Yuhang Zhou, Lizhu Zhang, Yifan Wu, Jiayi Liu+3

Meta AI

LLM Agent Training RL for LLM Agents

Open paper AskPDF arXiv

As large language model agents advance beyond software engineering (SWE) tasks toward machine learning engineering (MLE), verifying agent behavior becomes orders of magnitude more expensive: while SWE tasks can be verified via fast-executing unit tests, MLE verification requires running full ML pipelines -- data preprocessing, model training, and metric evaluation -- on large datasets at each rollout step, rendering trajectory-wise on-policy reinforcement learning (RL) prohibitively slow. Existing approaches retreat to supervised fine-tuning (SFT) or offline proxy rewards, sacrificing the exploration and generalization benefits of on-policy RL. We observe that sandbox data size is the primary source of this bottleneck. Based on this insight, we introduce SandMLE, a multi-agent framework that generates diverse, verifiable synthetic MLE environments from a small number of seed tasks, preserving the structural and technical complexity of real-world problems while constraining datasets to micro-scale (each task is paired with only 50-200 training samples). Through extensive experiments, we show that SandMLE reduces execution time by over 13 times, enabling large-scale, on-policy trajectory-wise RL for the first time in the MLE domain. On MLE-bench-lite, SandMLE yields significant gains over SFT baselines across Qwen3-8B, 14B, and 30B-A3B, with relative medal rate improvements ranging from 20.3% to 66.9%. Furthermore, the trained policy generalizes across unseen agentic scaffolds, achieving up to 32.4% better HumanRank score on MLE-Dojo.

Abstract

RL Benchmarks

Explore similar work

Synthetic Sandbox for Training Machine Learning Engineering Agents

OOD-RL-Bench: A Benchmark Framework for Out-of-Distribution Detection in Reinforcement Learning

Ground-Truth-First Benchmarking of Reinforcement Learning via Converse Optimality: From Smooth, Fully Observed Systems to Non-Smooth or Partially Observed Tasks