Ground-Truth-First Benchmarking of Reinforcement Learning via Converse Optimality: From Smooth, Fully Observed Systems to Non-Smooth or Partially Observed Tasks
Authors: Grégoire Ouerdane, Sinan Ibrahim, Stefan Streif, Pavel Osinenko
Organizations: Center for Engineering Systems and Sciences, Skoltech, 30 Bolshoy Boulevard, bld. 1, Moscow, 121205, Russia · Central University, 7 Gasheka St., bld. 1, Moscow, 123056, Russia · Technische Universität Chemnitz, Straße der Nationen 62, D-09111 Chemnitz, Germany · Sirius University of Science and Technology, 1 Olimpiyskiy Avenue, Sirius, 354340, Russia
Benchmarking in reinforcement learning is usually relative: algorithms are ranked by returns on environments with unknown optima. Distance to optimality remains unobservable, while conclusions may depend on reward design, stochasticity, hyperparameters, environment selection, and simulator details. We develop a ground-truth-first framework based on converse optimality, constructing each task with a certified optimal value function and policy withheld from learners and used only for evaluation. The framework unifies two modes: sensitivity benchmarking samples certified environments from continuous-time generators to assess robustness to task variation, while oracle-referenced comparison uses native discrete-time generators to evaluate policies via optimality gap, regret, and policy error. Continuous-time models support physically interpretable sensitivity studies but introduce dependence on the simulator step (Δt) and numerical scheme. Native discrete-time models align the environment, oracle, and learner on the same transition law, eliminating external integration choices as a confounder. We reproduce quantitative results covering optimizer sensitivity and oracle-referenced comparisons on serial planar arms, dynamically extended nonholonomic vehicles, and quadrotor-inspired tasks. The methodology extends beyond smooth affine Gaussian systems: virtual-input pullback handles non-affine raw-action interfaces and finite-covariance non-Gaussian disturbances. A non-smooth converse optimality formulation enables certified benchmarks with kinked value functions and switching surfaces. For partially observed evaluation, an information-state formulation reduces benchmarking to converse optimality on a sufficient statistic and provides an information-penalty reference. Code, fixtures, and diagnostics are available on the public project page.
Figures & tables
Figure 1: Ground-truth-first benchmark-generation pipeline. A converse construction samples certified dynamics and costs, validates the optimality certificate, exports an environment fixture, trains learners without oracle access, and evaluates learned policies against the hidden pair (V⋆,π⋆) .
Mode
Time formalism
Scientific question
Typical outputs
Sensitivity / robustness
Continuous-time generator
Does a learning pipeline remain reliable when the environment varies?
Relative suboptimality distributions, mean and variance tests, train–test shift studies
Oracle-referenced comparison
Native discrete-time fixture
How far is a learned policy from the certified optimum on a fixed fixture?
Table 1: Two complementary benchmark modes enabled by ground-truth-first fixtures. Mode I is typically instantiated with a continuous-time generator and Mode II with a native discrete-time fixture; Section 4 and Section 5 explain why.
Figure 2: Reproduced from Yaremenko et al. ( Yamerenko et al., 2025 ) . Sensitivity (top) and robustness (bottom) test structures. The symbol ∘ indicates that the policy is substituted into the test problem, closing the loop. The problem sample is split evenly at the junction.
Quantity
Value
Generator family
Unstable 3D continuous-time converse-optimal systems
Learning algorithm
PPO ( Schulman et al., 2017 )
Perturbation under test
Adam vs. RAdam optimizer
Environment sample size
160 generated problems, split into 80 per optimizer arm
Normalization
Relative suboptimality using each environment’s known optimal value
Test statistic
Variance ratio F≈2.59
Table 2: Continuous-time sensitivity case study. Two PPO pipelines differed only in optimizer. The measured quantity is relative suboptimality over a generated family of certified environments. Values are reported from the source; the stated test concerns the variance of relative suboptimality.
Figure 3: Comparing algorithm absolute performance. Discounted reward bars (r=−c) with 95% CIs under CRN. Colors correspond to p∈{0.5,0.6,0.8} .
Figure 4: Serial n -link planar arm (ConverseArm): absolute oracle-referenced performance (optimality gap) across algorithms (rows) and control authority p (columns), with separate panels for n .
Figure 5: Serial n -link planar arm (ConverseArm): absolute oracle-referenced performance (regret) across algorithms (rows) and control authority p (columns), with separate panels for n .
Figure 6: Left: reward-convention optimality gap to the certified optimum, where zero is optimal and more negative values are worse. Right: cost regret under matched CRN evaluation, where lower is better. Bars show means and error bars show paired-bootstrap 95% confidence intervals. The logarithmic vertical scales show order-of-magnitude differences across algorithms and regimes.
Algorithm
Optimality gap
Optimality gap 95% CI
Regret
Regret 95% CI
A2C
−1.44
[−1.76,−1.16]
2.13×101
[1.84×101,2.47×101]
DDPG
−2.18
[−2.53,−1.85]
3.35×101
[2.92×101,3.86×101]
PPO
−5.92×10−1
[−7.40×10−1,−4.68×10−1]
8.52
[7.47,9.66]
SAC
−6.09×10−1
[−7.64×10−1,−4.76×10−1]
9.03
[7.60,1.06×101]
TD3
−7.62×10−1
[−9.30×10−1,−6.24×10−1]
1.23×101
[1.00×101,1.51×101]
Table 3: ConverseQuad oracle-referenced metrics under CRN. Reproduced from ( Ibrahim et al., 2026 ) . Optimality gap uses the reward convention: zero is optimal and more negative values are worse; regret is in cost units, where smaller is better. Intervals are the source’s reported 95% paired-bootstrap intervals.
Figure 7: Compact oracle-referenced summaries. (a) Per-experiment optimality gap and regret show how algorithm rankings change with occupancy measure, stiffness, and actuation geometry. (b) ConverseQuad regret under nonlinear raw actuation and non-Gaussian disturbances. (c) Macro optimality gap; the closer the values are to zero, the better.
Config.
β
γ
k
C0
0.0
0.0
0.2
C1
1.0
−1.0
0.1
C2
1.0
−1.0
1.0
C3
0.7
−0.7
0.2
C4
1.4
−1.9
0.01
Table 4: Certified configurations of the piecewise-quadratic family. The first four satisfy (1+2k)β≥−γ from ( 48 ). The smooth baseline β=γ=0 is certified trivially.
Algorithm
Normalized OptGap
Regret
SAC
1.05[0.73,1.36]
2.39[1.96,2.83]
DDPG
1.18[0.80,1.55]
2.70[2.17,3.23]
TD3
5.04[3.05,7.03]
7.64[5.22,10.07]
PPO
5.14[2.06,8.23]
13.39[7.80,18.99]
A2C
12.97[0,26.47]
22.06[3.99,40.11]
Table 5: Aggregate non-smooth benchmark results over the certified configurations C0–C4. Mean and 95% CI via paired bootstrap (5 seeds × 15 initial states per configuration). Lower is better for both metrics.
Figure 8: Normalized optimality gap for each algorithm on the non-smooth mobile-robot benchmark. Error bars indicate 95% bootstrap confidence intervals.
Figure 9: Compact heatmap summaries. (a) Normalized optimality gap across the parameter space; darker shades indicate larger suboptimality. (b) Regret for the same fized parameters; darker shades correspond to higher cumulative excess cost.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Distribution of reward-convention optimality gaps by regime and algorithm. Points correspond to available run/fixture evaluation rows; violin widths show the empirical distribution of optimality gaps. The dashed line marks the certified optimum at zero. Black horizontal markers indicate central tendency. Unlike Figure 6 , this is not a confidence-interval plot; it is a distributional diagnostic that exposes variability and lower-tail failures.
Experiment
Algorithm
Optimality gap
Optimality gap 95% CI
Regret (mean)
Regret 95% CI
Serial-random
A2C
-2.36e+03
[-2.79e+03, -1.93e+03]
8.24e+03
[6.83e+03, 9.58e+03]
Serial-random
DDPG
-2.29e+03
[-3.83e+03, -9.12e+02]
9.28e+03
[3.50e+03, 1.63e+04]
Serial-random
PPO
-3.32e+02
[-8.45e+02, -4.92e+01]
1.51e+03
[1.92e+02, 3.93e+03]
Serial-random
SAC
-1.65e+02
[-3.47e+02, -4.63e+01]
6.90e+02
[1.72e+02, 1.53e+03]
Serial-random
TD3
-5.85e+01
[-8.64e+01, -3.57e+01]
2.14e+02
[1.25e+02, 3.35e+02]
Serial-fixed
A2C
-1.11e+03
[-2.15e+03, -4.58e+02]
5.78e+04
[3.12e+04, 9.23e+04]
Appendix
Table 6: Oracle-referenced comparison on native discrete-time fixtures. Metrics are mean and 95% CI via paired bootstrap under CRN. For optimality gap, zero is optimal and values closer to zero are better; for regret, smaller is better.
Reinforcement learning benchmarks provide tasks and evaluation protocols for comparing agents across control, games, robotics, and other decision-making settings.
The objective comparison of Reinforcement Learning (RL) algorithms is notoriously complex as outcomes and benchmarking of performances of different RL approaches are critically sensitive to environmental design, reward structures, and stochasticity inherent in both algorithmic learning and environmental dynamics. To manage this complexity, we introduce a rigorous benchmarking framework by extending converse optimality to discrete-time, control-affine, nonlinear systems with noise. Our framework provides necessary and sufficient conditions, under which a prescribed value function and policy are optimal for constructed systems, enabling the systematic generation of benchmark families via homotopy variations and randomized parameters. We validate it by automatically constructing diverse environments, demonstrating our framework's capacity for a controlled and comprehensive evaluation across algorithms. By assessing standard methods against a ground-truth optimum, our work delivers a reproducible foundation for precise and rigorous RL benchmarking.
Sinan Ibrahim, Grégoire Ouerdane, Hadi Salloum +3
Skolkovo Institute of Science and Technology, Moscow, Russia · Innopolis University, Innopolis, Russia · Technische Universität Chemnitz, Chemnitz, Germany +2
Safe reinforcement learning (RL) is commonly formalized as a Constrained Markov Decision Process (CMDP), in which an agent maximizes expected reward while keeping its expected cumulative cost below a specified safety bound. Existing safe RL benchmarks predominantly report whether an algorithm is safe on average, following this expectation-based guarantee. We argue that this convention is insufficient to reliably characterize an algorithm's true safety: it fails to capture how often and how severely the safety bound is violated, whether this holds consistently across tasks and safety bounds, and whether training-time behavior is representative of behavior of the final converged policy. Therefore, we introduce (i) evaluation metrics for safe RL that address each of these concerns and in addition allow for aggregation across tasks and safety bounds. We furthermore define (ii) a safety tier system to systematically categorize and compare algorithms in terms of safety and reliability at both training and for a final policy. Using this framework, we provide (iii) an empirical safety evaluation across multiple safety navigation tasks. Our results show that aggregate metrics, distributional reporting, and task- and safety bound-specific results each reveal information the other metrics cannot. We therefore recommend reporting all three jointly, rather than compressing this information into a single value, as is common practice. We provide SafeRLEval, an open-source evaluation suite to support the reliable characterization of safety in future safe RL research.
Lindsay Spoor, Aske Plaat, Thomas Moerland
Leiden Institute of Advanced Computer Science, Leiden University, Leiden, the Netherlands
Reinforcement Learning (RL) lacks benchmarks that enable precise, white-box diagnostics of agent behavior. Current environments often entangle complexity factors and lack ground-truth optimality metrics, making it difficult to isolate why algorithms fail. We introduce Synthetic Monitoring Environments (SMEs), an infinite suite of continuous control tasks. SMEs provide fully configurable task characteristics and known optimal policies. As such, SMEs allow for the exact calculation of instantaneous regret. Their rigorous geometric state space bounds allow for systematic within-distribution (WD) and out-of-distribution (OOD) evaluation. We demonstrate the framework's benefit through multidimensional ablations of PPO, TD3, and SAC, revealing how specific environmental properties - such as action or state space size, reward sparsity and complexity of the optimal policy - impact WD and OOD performance. We thereby show that SMEs offer a standardized, transparent testbed for transitioning RL evaluation from empirical benchmarking toward rigorous scientific analysis.
Leonard Pleiss, Carolin Schmidt, Maximilian Schiffer