Learning from the Test: Self-Referential Differential Testing for Deep RL Agents

Junda He; Jieke Shi; Zhou Yang; Mingfei Cheng; David Lo

cs.SEAug 23, 2026

Learning from the Test: Self-Referential Differential Testing for Deep RL Agents

Authors: Junda He, Jieke Shi, Zhou Yang, Mingfei Cheng, David Lo

Organizations: Singapore Management University, Singapore · University of Alberta, Edmonton, Canada and Canada CIFAR AI Chair, Amii, Edmonton, Canada

Abstract

Deep Reinforcement Learning (DRL) has achieved significant success in complex decision-making problems. As DRL systems are increasingly deployed in real-world applications, ensuring their quality and reliability is paramount. Current works primarily focus on detecting safety-critical failures, often neglecting policy optimality, which can lead to reduced efficiency, user distrust, and economic losses. This oversight, compounded by the inherent "testing oracle problem" for optimality, leaves a significant gap in comprehensively evaluating DRL systems. To address this gap, we propose Delta (Differential Testing for DRL Agents), a novel and comprehensive framework that automatically identifies both safety-critical and optimality bugs in DRL agents. Delta employs a two-phase approach: (1) Safety Testing, where the Agent Under Test (AUT) is evaluated for catastrophic failures while collecting data from its decision-making policy, and (2) Optimality Testing, where this collected data from the prior phase is used to train a challenger agent via Offline Reinforcement Learning. Differential testing is then performed by comparing the challenger agent against the AUT; instances where the challenger achieves higher cumulative rewards indicate optimality issues in the AUT. We demonstrate Delta's effectiveness across five environments. We investigate the effectiveness of three offline RL algorithms (BC, BCQ, and CQL) in generating challenger agents. Experimental results demonstrate that safety testing datasets are valuable for training competent DRL agents. Challenger agents trained with BCQ proved most effective for identifying optimality issues within the framework of Delta. Across the five environments, Delta uncovered an average of 2,518 optimality issues, outperforming the baseline methods by 50.2%.

Figures & tables

Figure 1. Illustration of Offline Reinforcement Learning. Illustration of Offline Reinforcement Learning.

Figure 2. The Overall Workflow of Delta . The Overall Workflow of Delta.

Figure 3. Delta ’s Differential Fuzzing Workflow. Delta Differential Fuzzing Workflow.

Environment	Number of Safety Failures of AUT	AUT Success Rate (%)	AUT Performance	BC Performance	BCQ Performance	CQL Performance
CartPole	$77.4\pm 10.3$	$97.0\ [91.6,99.0]$	$495.7\pm 6.8$	$497.1\pm 2.5$	$496.0\pm 2.6$	$\textbf{499.8}\pm 1.6$
MountainCar	$167.2\pm 8.7$	$99.0\ [94.6,99.8]$	$-129.8\pm 2.8$	$-129.0\pm 1.4$	$-128.2\pm 1.4$	$\textbf{-126.6}\pm 1.5$
Acrobot	$1183.6\pm 41.9$	$93.0\ [86.3,96.6]$	$-81.3\pm 3.0$	$-83.7\pm 2.1$	$-84.2\pm 3.2$	$\textbf{-83.3}\pm 1.5$
Hopper	$154.4\pm 9.9$	$97.0\ [91.6,99.0]$	$3162.3\pm 60.9$	$3173.4\pm 41.6$	$\textbf{3204.4}\pm 32.4$	$3173.8\pm 20.8$
Walker2D	$251.8\pm 23.8$	$95.0\ [88.8,97.9]$	$3925.5\pm 192.2$	$3961.3\pm 95.8$	$\textbf{4004.0}\pm 92.1$	$3996.5\pm 68.5$

Table 1. Number of safety failures detected in the AUT, AUT success rate, and mean cumulative reward over 100 episodes for the AUT and challengers (BC, BCQ, CQL). Brackets [lower, upper] denote the 95% Wilson score CI for the success rate;

\pm

denotes the 95% CI margin of error for the other columns. Bold values indicate the best-performing challenger agent (among BC, BCQ, and CQL) for each environment.

Panel of Figure 4 — Figure 4. Running time of training the Agent Under Test (AUT), and the challenger agents (BC, BCQ, and CQL). Bar charts of the training time of the AUT and the challenger agents (BC, BCQ, and CQL) in each of the five environments.

# of Bugs	CartPole	MountainCar	Acrobot	Hopper	Walker2D
BC	79.4 $\pm$ 8.5	1243.2 $\pm$ 318.3	2900.0 $\pm$ 797.3	1231.8 $\pm$ 236.8	1251.4 $\pm$ 148.9
BCQ	105.6 $\pm$ 17.7	3752.0 $\pm$ 649.9	5450.0 $\pm$ 1241.7	1738.8 $\pm$ 315.6	1543.2 $\pm$ 146.6
CQL	84.4 $\pm$ 10.7	2144.0 $\pm$ 797.6	4811.0 $\pm$ 760.1	875.0 $\pm$ 123.5	778.6 $\pm$ 75.7
# of Distinct Bugs	CartPole	MountainCar	Acrobot	Hopper	Walker2D
BC	65.4 $\pm$ 9.8	19.8 $\pm$ 6.9	298.2 $\pm$ 12.2	1183.6 $\pm$ 228.8	1238.4 $\pm$ 150.8
BCQ	81.0 $\pm$ 9.8	58.2 $\pm$ 9.9	467.4 $\pm$ 10.0	1648.8 $\pm$ 280.8	1508.2 $\pm$ 213.4

Table 2. Experiment Results of RQ1. We compare the performance of different offline RL algorithms for training the challenger agents to detect optimality bugs. We report the Number of Bugs, the Number of Distinct Bugs, and the Distance. The average result over five runs is reported. The

\pm

values represent the 95% confidence interval margin of error. Bold values indicate the best result in each column.

Environment	# of Bugs
Environment	Delta	MDPFuzz	CureFuzz	GMT
CartPole	105.6 $\pm$ 17.7 (+46.7%)	54.0 $\pm$ 17.6	72.0 $\pm$ 8.1	60.6 $\pm$ 15.5
MountainCar	3752.0 $\pm$ 649.9 (+91.5%)	1854.4 $\pm$ 600.6	1959.2 $\pm$ 383.6	739.8 $\pm$ 559.1
Acrobot	5450.0 $\pm$ 1241.7 (+40.9%)	3869.0 $\pm$ 1425.4	2556.6 $\pm$ 861.9	1850.8 $\pm$ 598.5
Hopper	1738.8 $\pm$ 315.6 (+37.9%)	674.2 $\pm$ 101.3	815.0 $\pm$ 160.1	1261.2 $\pm$ 286.2
Walker2D	1543.2 $\pm$ 146.6(+34.1%)	598.6 $\pm$ 141.1	838.8 $\pm$ 342.7	1150.4 $\pm$ 193.9

Table 3. Performance comparison of Delta against baseline methods (MDPFuzz, CureFuzz, and GMT) in detecting optimality bugs, including the total number of bugs found (# of Bugs), the number of unique bugs (# of Distinct Bugs), and Distance. All results are averaged over five runs, with the

\pm

values representing the 95% confidence interval margin of error. Bold values indicate the best performance in each row. The percentages in parentheses for Delta denote its improvement over the best-performing baseline for that metric.

Method	# of Bugs
Method	CartPole	MountainCar	Acrobot	Hopper	Walker2D
Delta	105.6 $\pm$ 17.7	3752.0 $\pm$ 649.9	5450.0 $\pm$ 1241.7	1738.8 $\pm$ 315.6	1543.2 $\pm$ 146.6
w/o-Diversity	93.8 $\pm$ 7.7	2373.0 $\pm$ 269.4	2963.0 $\pm$ 446.4	1464.6 $\pm$ 271.6	1252.0 $\pm$ 311.7
w/o-Regret	74.6 $\pm$ 11.3	2328.0 $\pm$ 252.6	2238.4 $\pm$ 165.9	1409.8 $\pm$ 214.1	1009.4 $\pm$ 247.9
w/o-Inconsistency	67.6 $\pm$ 9.0	2509.2 $\pm$ 526.3	2532.4 $\pm$ 295.2	1419.4 $\pm$ 350.9	1147.2 $\pm$ 206.0
Method	# of Distinct Bugs

Table 4. Performance comparison of Delta against ablated variants (w/o diversity, w/o regret, w/o inconsistency) in detecting optimality bugs, including the total number of bugs found (# of Bugs), the number of unique bugs (# of Distinct Bugs), and Distance. All results are averaged over five runs, with the

\pm

values representing the 95% confidence interval margin of error. Bold values indicate the best performance in each column.

Environment	Mean Reward (AUT)	Mean Reward (AUT-Improved)	Safety Failures of AUT	Safety Failures of AUT-Improved (after repair)	CureFuzz Re-eval of AUT-Improved
CartPole	495.7 $\pm$ 6.8	500 $\pm$ 0.0	77.4 $\pm$ 10.3	0.0(- 100.0% ) $\pm$ 0.0	0.0 $\pm$ 0.0
MountainCar	-129.8 $\pm$ 2.8	-125.4 $\pm$ 3.2	167.2 $\pm$ 8.7	0.0(- 100.0% ) $\pm$ 0.0	37.4 $\pm$ 6.4
Acrobot	-81.3 $\pm$ 3.0	-75.2 $\pm$ 2.9	1183.6 $\pm$ 41.9	118.2(- 90.0% ) $\pm$ 11.3	265.0 $\pm$ 18.5
Hopper	3162.3 $\pm$ 60.9	3247.0 $\pm$ 30.5	154.4 $\pm$ 9.9	40.0(- 74.1% ) $\pm$ 6.0	106.2 $\pm$ 19.5
Walker2D	3925.5 $\pm$ 192.2	4399.3 $\pm$ 93.8	251.8 $\pm$ 23.8	5.6(- 97.8% ) $\pm$ 2.4	105.0 $\pm$ 17.2

Table 5. Comparison of the original Agent Under Test (AUT) and the AUT-Improved. The first two columns report the mean cumulative reward for each agent. The third and fourth columns present the average number of safety failures on known safety-critical scenarios, with percentages in parentheses quantifying the failure reduction after repair. The last column reports the results of the re-evaluation of CureFuzz against the AUT-Improved. All

\pm

values represent the 95% confidence interval margin of error.

$\bm{\tau}$	CartPole	MountainCar	Acrobot	Hopper	Walker2D
0.10	100.2 $\pm$ 15.2	4.2 $\pm$ 1.0	279.0 $\pm$ 10.5	1029.8 $\pm$ 139.0	1505.6 $\pm$ 204.6
0.20	25.4 $\pm$ 3.4	3.2 $\pm$ 0.6	130.6 $\pm$ 8.9	93.8 $\pm$ 6.4	1111.2 $\pm$ 152.7
0.30	3.8 $\pm$ 1.4	3.0 $\pm$ 0.0	54.8 $\pm$ 2.4	56.6 $\pm$ 5.9	445.2 $\pm$ 116.3

Table 6. Number of behavioral patterns detected by Delta across the five environments at three relative distance thresholds

\tau\in\{0.10,0.20,0.30\}

. Results are averaged over five independent runs;

\pm

values denote the 95% confidence interval margin of error.

Challenger Algorithm	BC	BCQ	CQL
# Bugs	44.6 $\pm$ 11.3	87.8 $\pm$ 23.3	65.2 $\pm$ 22.8
Testing Method	Delta	CureFuzz	GMT	MDPFuzz
# Bugs	87.8 $\pm$ 23.3 (+171%)	32.4 $\pm$ 11.3	25.2 $\pm$ 6.7	24.6 $\pm$ 6.4

Table 7. Optimality bug detection on the non-deterministic Hopper-Noise environment. Top: comparison of offline RL algorithms for training the challenger. Bottom: Delta against baseline methods. All results are averaged over five runs;

\pm

values denote the 95% confidence interval margin of error.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

DreamTest: World-Model Surrogates for Search-Based Testing of Deep Reinforcement Learning Agents
Oct 3, 2026Qinghua Xu, Guancheng Wang, Boxi Yu +2Automated Software Testing Neural Surrogate Modeling
Lero Research Centre, University of Limerick, Ireland · University of Ottawa, Canada
Analyzing Symbolic Properties for DRL Agents in Systems and Networking
Apr 6, 2026Mohammad Zangooei, Jannis Weil, Amr Rizk +2Reinforcement Learning Neural Network Verification
University of Waterloo, Waterloo, Canada · Leibniz University Hannover, Hannover, Germany
Robust Policy Optimization via Adversarial Importance Sampling
Sep 11, 2026Amine Andam, Jamal Bentahar, Mustapha HedabouRobust RL Policy Learning
Mohammed VI Polytechnic University · Khalifa University · Concordia University

Oct 3, 2026J/K move · Enter open · S save

DreamTest: World-Model Surrogates for Search-Based Testing of Deep Reinforcement Learning Agents

Qinghua Xu, Guancheng Wang, Boxi Yu, Liting Lin+1

Lero Research Centre, University of Limerick, Ireland · University of Ottawa, Canada

Automated Software Testing Neural Surrogate Modeling Reinforcement Learning World Models

Open paper AskPDF arXiv

Testing deep reinforcement learning (DRL) agents in cyber-physical systems aims to uncover diverse failures before deployment, but each execution can be expensive. Surrogate-assisted testing reduces this cost by learning to predict which test configurations are likely to fail. Prior surrogates treat the system as a black box and predict pass or fail outcomes directly; we instead model how a test unfolds and estimate failure from an imagined episode. We introduce DreamTest, a world-model surrogate for testing DRL agents. DreamTest adapts a recurrent state-space model to learn agent behaviour and environment dynamics from the agent's training log. Given a candidate configuration, imagined rollouts produce a failure score that guides search without executing every candidate in a simulator or real system. We evaluate DreamTest for failure prediction, test generation, and failure diversity on Parking, Humanoid, and DonkeyCar. Mean area under the precision-recall curve (AUPRC) exceeds the strongest baseline by 97%, 12%, and 39%, respectively, and gains on five out-of-distribution test sets reach 145%, 29%, and 44%. Under the same simulator-validation budget, the best "DreamTest + search" combinations find 29%, 22%, and 79% more novel failures on average. Across clusterings with k = 2-40, failures generated with DreamTest cover the most behavioural clusters for almost all k, indicating that DreamTest consistently discovers behaviourally diverse failures.

Abstract

Reinforcement Learning

Explore similar work

DreamTest: World-Model Surrogates for Search-Based Testing of Deep Reinforcement Learning Agents

Analyzing Symbolic Properties for DRL Agents in Systems and Networking

Robust Policy Optimization via Adversarial Importance Sampling