Ground-Truth-First Benchmarking of Reinforcement Learning via Converse Optimality: From Smooth, Fully Observed Systems to Non-Smooth or Partially Observed Tasks

Grégoire Ouerdane; Sinan Ibrahim; Stefan Streif; Pavel Osinenko

cs.LGOct 22, 2026

Ground-Truth-First Benchmarking of Reinforcement Learning via Converse Optimality: From Smooth, Fully Observed Systems to Non-Smooth or Partially Observed Tasks

Authors: Grégoire Ouerdane, Sinan Ibrahim, Stefan Streif, Pavel Osinenko

Organizations: Center for Engineering Systems and Sciences, Skoltech, 30 Bolshoy Boulevard, bld. 1, Moscow, 121205, Russia · Central University, 7 Gasheka St., bld. 1, Moscow, 123056, Russia · Technische Universität Chemnitz, Straße der Nationen 62, D-09111 Chemnitz, Germany · Sirius University of Science and Technology, 1 Olimpiyskiy Avenue, Sirius, 354340, Russia

Abstract

Benchmarking in reinforcement learning is usually relative: algorithms are ranked by returns on environments with unknown optima. Distance to optimality remains unobservable, while conclusions may depend on reward design, stochasticity, hyperparameters, environment selection, and simulator details. We develop a ground-truth-first framework based on converse optimality, constructing each task with a certified optimal value function and policy withheld from learners and used only for evaluation. The framework unifies two modes: sensitivity benchmarking samples certified environments from continuous-time generators to assess robustness to task variation, while oracle-referenced comparison uses native discrete-time generators to evaluate policies via optimality gap, regret, and policy error. Continuous-time models support physically interpretable sensitivity studies but introduce dependence on the simulator step ( $Δt$ ) and numerical scheme. Native discrete-time models align the environment, oracle, and learner on the same transition law, eliminating external integration choices as a confounder. We reproduce quantitative results covering optimizer sensitivity and oracle-referenced comparisons on serial planar arms, dynamically extended nonholonomic vehicles, and quadrotor-inspired tasks. The methodology extends beyond smooth affine Gaussian systems: virtual-input pullback handles non-affine raw-action interfaces and finite-covariance non-Gaussian disturbances. A non-smooth converse optimality formulation enables certified benchmarks with kinked value functions and switching surfaces. For partially observed evaluation, an information-state formulation reduces benchmarking to converse optimality on a sufficient statistic and provides an information-penalty reference. Code, fixtures, and diagnostics are available on the public project page.

Figures & tables

Figure 1: Ground-truth-first benchmark-generation pipeline. A converse construction samples certified dynamics and costs, validates the optimality certificate, exports an environment fixture, trains learners without oracle access, and evaluates learned policies against the hidden pair $(V^{\star},\pi^{\star})$ .

Mode	Time formalism	Scientific question	Typical outputs
Sensitivity / robustness	Continuous-time generator	Does a learning pipeline remain reliable when the environment varies?	Relative suboptimality distributions, mean and variance tests, train–test shift studies
Oracle-referenced comparison	Native discrete-time fixture	How far is a learned policy from the certified optimum on a fixed fixture?	Optimality gap, regret, Bellman residual, policy error, paired CRN confidence intervals

Table 1: Two complementary benchmark modes enabled by ground-truth-first fixtures. Mode I is typically instantiated with a continuous-time generator and Mode II with a native discrete-time fixture; Section 4 and Section 5 explain why.

Figure 2: Reproduced from Yaremenko et al. ( Yamerenko et al., 2025 ) . Sensitivity (top) and robustness (bottom) test structures. The symbol $\circ$ indicates that the policy is substituted into the test problem, closing the loop. The problem sample is split evenly at the junction.

Quantity	Value
Generator family	Unstable 3D continuous-time converse-optimal systems
Learning algorithm	PPO ( Schulman et al., 2017 )
Perturbation under test	Adam vs. RAdam optimizer
Environment sample size	160 generated problems, split into 80 per optimizer arm
Normalization	Relative suboptimality using each environment’s known optimal value
Test statistic	Variance ratio $F\approx 2.59$

Table 2: Continuous-time sensitivity case study. Two PPO pipelines differed only in optimizer. The measured quantity is relative suboptimality over a generated family of certified environments. Values are reported from the source; the stated test concerns the variance of relative suboptimality.

Figure 3: Comparing algorithm absolute performance. Discounted reward bars $(r=-c)$ with 95% CIs under CRN. Colors correspond to $p\in\{0.5,0.6,0.8\}$ .

Figure 4: Serial $n$ -link planar arm (ConverseArm): absolute oracle-referenced performance (optimality gap) across algorithms (rows) and control authority $p$ (columns), with separate panels for $n$ .

Figure 5: Serial $n$ -link planar arm (ConverseArm): absolute oracle-referenced performance (regret) across algorithms (rows) and control authority $p$ (columns), with separate panels for $n$ .

Figure 6: Left: reward-convention optimality gap to the certified optimum, where zero is optimal and more negative values are worse. Right: cost regret under matched CRN evaluation, where lower is better. Bars show means and error bars show paired-bootstrap 95% confidence intervals. The logarithmic vertical scales show order-of-magnitude differences across algorithms and regimes.

Algorithm	Optimality gap	Optimality gap 95% CI	Regret	Regret 95% CI
A2C	$-1.44$	$[-1.76,\,-1.16]$	$2.13\times 10^{1}$	$[1.84\times 10^{1},\,2.47\times 10^{1}]$
DDPG	$-2.18$	$[-2.53,\,-1.85]$	$3.35\times 10^{1}$	$[2.92\times 10^{1},\,3.86\times 10^{1}]$
PPO	$-5.92\times 10^{-1}$	$[-7.40\times 10^{-1},\,-4.68\times 10^{-1}]$	$8.52$	$[7.47,\,9.66]$
SAC	$-6.09\times 10^{-1}$	$[-7.64\times 10^{-1},\,-4.76\times 10^{-1}]$	$9.03$	$[7.60,\,1.06\times 10^{1}]$
TD3	$-7.62\times 10^{-1}$	$[-9.30\times 10^{-1},\,-6.24\times 10^{-1}]$	$1.23\times 10^{1}$	$[1.00\times 10^{1},\,1.51\times 10^{1}]$

Table 3: ConverseQuad oracle-referenced metrics under CRN. Reproduced from ( Ibrahim et al., 2026 ) . Optimality gap uses the reward convention: zero is optimal and more negative values are worse; regret is in cost units, where smaller is better. Intervals are the source’s reported

95\%

paired-bootstrap intervals.

Panel of Figure 7 — Figure 7: Compact oracle-referenced summaries. (a) Per-experiment optimality gap and regret show how algorithm rankings change with occupancy measure, stiffness, and actuation geometry. (b) ConverseQuad regret under nonlinear raw actuation and non-Gaussian disturbances. (c) Macro optimality gap; the closer the values are to zero, the better.

Config.	$\beta$	$\gamma$	$k$
C0	$0.0$	$0.0$	$0.2$
C1	$1.0$	$-1.0$	$0.1$
C2	$1.0$	$-1.0$	$1.0$
C3	$0.7$	$-0.7$	$0.2$
C4	$1.4$	$-1.9$	$0.01$

Table 4: Certified configurations of the piecewise-quadratic family. The first four satisfy

(1+2k)\beta\geq-\gamma

from ( 48 ). The smooth baseline

\beta=\gamma=0

is certified trivially.

Algorithm	Normalized OptGap	Regret
SAC	$1.05$ $[0.73,1.36]$	$2.39$ $[1.96,2.83]$
DDPG	$1.18$ $[0.80,1.55]$	$2.70$ $[2.17,3.23]$
TD3	$5.04$ $[3.05,7.03]$	$7.64$ $[5.22,10.07]$
PPO	$5.14$ $[2.06,8.23]$	$13.39$ $[7.80,18.99]$
A2C	$12.97$ $[0,26.47]$	$22.06$ $[3.99,40.11]$

Table 5: Aggregate non-smooth benchmark results over the certified configurations C0–C4. Mean and

95\%

CI via paired bootstrap (5 seeds

\times

15 initial states per configuration). Lower is better for both metrics.

Figure 8: Normalized optimality gap for each algorithm on the non-smooth mobile-robot benchmark. Error bars indicate 95% bootstrap confidence intervals.

Panel of Figure 9 — Figure 9: Compact heatmap summaries. (a) Normalized optimality gap across the parameter space; darker shades indicate larger suboptimality. (b) Regret for the same fized parameters; darker shades correspond to higher cumulative excess cost.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Distribution of reward-convention optimality gaps by regime and algorithm. Points correspond to available run/fixture evaluation rows; violin widths show the empirical distribution of optimality gaps. The dashed line marks the certified optimum at zero. Black horizontal markers indicate central tendency. Unlike Figure 6 , this is not a confidence-interval plot; it is a distributional diagnostic that exposes variability and lower-tail failures.

Experiment	Algorithm	Optimality gap	Optimality gap 95% CI	Regret (mean)	Regret 95% CI
Serial-random	A2C	-2.36e+03	[-2.79e+03, -1.93e+03]	8.24e+03	[6.83e+03, 9.58e+03]
Serial-random	DDPG	-2.29e+03	[-3.83e+03, -9.12e+02]	9.28e+03	[3.50e+03, 1.63e+04]
Serial-random	PPO	-3.32e+02	[-8.45e+02, -4.92e+01]	1.51e+03	[1.92e+02, 3.93e+03]
Serial-random	SAC	-1.65e+02	[-3.47e+02, -4.63e+01]	6.90e+02	[1.72e+02, 1.53e+03]
Serial-random	TD3	-5.85e+01	[-8.64e+01, -3.57e+01]	2.14e+02	[1.25e+02, 3.35e+02]
Serial-fixed	A2C	-1.11e+03	[-2.15e+03, -4.58e+02]	5.78e+04	[3.12e+04, 9.23e+04]

Appendix

Table 6: Oracle-referenced comparison on native discrete-time fixtures. Metrics are mean and 95% CI via paired bootstrap under CRN. For optimality gap, zero is optimal and values closer to zero are better; for regret, smaller is better.

RL Benchmarks

Reinforcement learning benchmarks provide tasks and evaluation protocols for comparing agents across control, games, robotics, and other decision-making settings.

See all 85 papers

Benchmarking Reinforcement Learning via Stochastic Converse Optimality: Generating Systems with Known Optimal Policies
Mar 18, 2026Sinan Ibrahim, Grégoire Ouerdane, Hadi Salloum +3RL Benchmarks RL Control
Skolkovo Institute of Science and Technology, Moscow, Russia · Innopolis University, Innopolis, Russia · Technische Universität Chemnitz, Chemnitz, Germany +2
Evaluation Metrics for Safe Reinforcement Learning
Sep 14, 2026Lindsay Spoor, Aske Plaat, Thomas MoerlandRL Benchmarks Constrained RL
Leiden Institute of Advanced Computer Science, Leiden University, Leiden, the Netherlands
Synthetic Monitoring Environments for Reinforcement Learning
Mar 6, 2026Leonard Pleiss, Carolin Schmidt, Maximilian SchifferRL Benchmarks Reinforcement Learning
Technical University Munich Munich, 80331

Mar 18, 2026J/K move · Enter open · S save

Benchmarking Reinforcement Learning via Stochastic Converse Optimality: Generating Systems with Known Optimal Policies

Sinan Ibrahim, Grégoire Ouerdane, Hadi Salloum, Henni Ouerdane+2

Skolkovo Institute of Science and Technology, Moscow, Russia · Innopolis University, Innopolis, Russia · Technische Universität Chemnitz, Chemnitz, Germany+2

RL Benchmarks RL Control

Open paper AskPDF arXiv

The objective comparison of Reinforcement Learning (RL) algorithms is notoriously complex as outcomes and benchmarking of performances of different RL approaches are critically sensitive to environmental design, reward structures, and stochasticity inherent in both algorithmic learning and environmental dynamics. To manage this complexity, we introduce a rigorous benchmarking framework by extending converse optimality to discrete-time, control-affine, nonlinear systems with noise. Our framework provides necessary and sufficient conditions, under which a prescribed value function and policy are optimal for constructed systems, enabling the systematic generation of benchmark families via homotopy variations and randomized parameters. We validate it by automatically constructing diverse environments, demonstrating our framework's capacity for a controlled and comprehensive evaluation across algorithms. By assessing standard methods against a ground-truth optimum, our work delivers a reproducible foundation for precise and rigorous RL benchmarking.

Abstract

RL Benchmarks

Explore similar work

Benchmarking Reinforcement Learning via Stochastic Converse Optimality: Generating Systems with Known Optimal Policies

Evaluation Metrics for Safe Reinforcement Learning

Synthetic Monitoring Environments for Reinforcement Learning