Disagreement-Regularized Imitation Learning for Image-Based Continuous Control with Gaussian and Beta Policies

Irving Giovani Bronzatti Petrazzini; Eric Aislan Antonelo

cs.LGSep 29, 2026

Disagreement-Regularized Imitation Learning for Image-Based Continuous Control with Gaussian and Beta Policies

Authors: Irving Giovani Bronzatti Petrazzini, Eric Aislan Antonelo

Organizations: Department of Automation and Systems Engineering, Federal University of Santa Catarina, Florian´opolis, Santa Catarina, Brazil.

Abstract

Purpose: Behavior cloning can accumulate errors when a learned controller visits states outside the demonstrated distribution. This study evaluates whether Disagreement-Regularized Imitation Learning (DRIL), which converts disagreement among cloned policies into a reinforcement-learning reward, improves image-based continuous control. Methods: A controlled CarRacing study combines Gaussian and Beta learner policies, demonstrations from either a clipped Gaussian expert or an intrinsically bounded Beta expert, one or 20 trajectories, deterministic and stochastic evaluation, and three retained stages: behavior cloning, the highest 10-episode training-score checkpoint, and the final DRIL checkpoint. The disagreement ensemble contains five Gaussian policies in every variant. Each retained policy is evaluated over 100 procedurally generated episodes. Results: Score-selected DRIL produced its largest gains in the few-demonstration setting, improving over the strongest behavior-cloning mean by 61% with clipped-action demonstrations and by 112% with bounded-action demonstrations. With 20 trajectories, the advantage of DRIL narrowed; in the bounded-action regime, Beta behavior cloning remained about 7% above the best DRIL checkpoint. The experiments also show that the informativeness of the disagreement reward changes with the learner representation and training stage. Conclusion: DRIL can substantially improve few-demonstration visual continuous control, while bounded Beta policies provide strong behavior-cloning performance when more demonstrations are available. The results highlight the joint importance of learner support,ensemble response, and checkpoint selection.

Figures & tables

Method	Expert queries	Task reward	Corrective mechanism
BC	No	No	None; supervised fitting on the fixed dataset
DAgger Ross et al. (2011)	Yes	No	Aggregates expert labels on learner-visited states
GAIL Ho and Ermon (2016)	No	No	Adversarial occupancy-measure matching
Hybrid RL–IL Kumar (2024) ; Lu et al. (2023)	Varies	Yes	Refines imitation with environment rewards
Diffusion Meets DAgger Zhang et al. (2024)	No new labels	No	Synthesizes examples near failure states
DRIL Brantley et al. (2020)	No	No	Thresholded disagreement among cloned policies

Table 1 : Qualitative comparison of methods that address imitation-learning distribution shift. “Expert queries” indicates whether additional labels are required after the initial demonstrations. “Task reward” indicates use of the environment reward during policy improvement.

Figure 1 : Two-stage DRIL workflow used in this work. In stage (a), the expert demonstration dataset $\mathcal{D}$ trains the ensemble $\Pi$ by behavior cloning; in the present experiments, $\Pi$ contains five Gaussian policies trained on bootstrap samples. In stage (b), a BC-initialized learner interacts with the environment. Ensemble disagreement provides the binary reward for reinforcement-learning updates, while behavior-cloning updates continue to use the original demonstrations.

Panel of Figure 2 — Figure 2 : Visual actor–critic architectures used in this work. Both learners use the same image preprocessing and convolutional feature extractor. The actor head changes from Gaussian outputs in (a) to Beta-distribution parameters in (b), while the critic estimates a scalar value function. The disagreement ensemble remains Gaussian in all experiments.

Figure 3 : Expert-action generation. The Beta expert produces bounded samples; the Gaussian expert produces many samples outside the action range; the clipped Gaussian actions are those executed in the environment and used in the clipped-action demonstration dataset.

Factor	Levels
Expert demonstration actions	Clipped Gaussian; intrinsically bounded Beta
Number of expert trajectories	1; 20
Learner policy distribution	Gaussian; Beta
Retained training stage	BC; DRIL-peak; DRIL-final
Evaluation mode	Deterministic mean; stochastic sampling

Table 2 : Experimental factors in the main CarRacing study. Every learner is evaluated in deterministic mode, using the distribution mean, and stochastic mode, sampling from the distribution. BC denotes the validation-selected behavior-cloning checkpoint; DRIL-peak is selected by the highest 10-episode training-score mean; DRIL-final is the last checkpoint.

Parameter	Value	Parameter	Value
BC learning rate	$2.5\times 10^{-4}$	BC mini-batch	32
BC data split	80/20	BC patience	20 epochs
Ensemble size	5	Ensemble training	2000 epochs
Uncertainty quantile	0.98	PPO learning rate	$3\times 10^{-4}$ , annealed
PPO rollout	2048 steps	PPO epochs	10
PPO mini-batches	32	Discount $\gamma$	0.99

Table 3 : Main CarRacing training settings. Ensemble members are Gaussian policies trained for 2000 epochs on bootstrap samples. The learner is first trained by BC and then updated by interleaved PPO and BC steps during DRIL.

Expert	Method	1 trajectory		20 trajectories
		Deterministic	Stochastic	Deterministic	Stochastic
Clipped	BC	125 $\pm$ 113	30 $\pm$ 67	171 $\pm$ 124	437 $\pm$ 115
	DRIL-peak	166 $\pm$ 131	322 $\pm$ 208	423 $\pm$ 211	802 $\pm$ 197
	DRIL-final	41 $\pm$ 80	39 $\pm$ 79	229 $\pm$ 107	218 $\pm$ 108
Bounded	BC	194 $\pm$ 113	75 $\pm$ 47	617 $\pm$ 260	137 $\pm$ 70
	DRIL-peak	235 $\pm$ 123	341 $\pm$ 246	656 $\pm$ 266	720 $\pm$ 289

Table 4 : Gaussian learner evaluation on CarRacing. Values are mean

\pm

standard deviation over 100 evaluation episodes. Bold values mark the highest mean scores within each evaluation mode and dataset regime.

Panel of Figure 4 — Figure 4 : Training dynamics of the Gaussian learner using clipped-action expert demonstrations. Panels (a) and (b) correspond to one and 20 demonstrations, respectively.

Panel of Figure 5 — Figure 5 : Stochastic 100-episode evaluation of the Gaussian learner using clipped-action expert demonstrations. Panels (a) and (b) correspond to one and 20 demonstrations, respectively, and compare BC, DRIL-peak, and DRIL-final. Deterministic results are reported in Table 4 .

Panel of Figure 6 — Figure 6 : Training dynamics of the Gaussian learner using bounded-action expert demonstrations. Panels (a) and (b) correspond to one and 20 demonstrations, respectively.

Panel of Figure 7 — Figure 7 : Stochastic 100-episode evaluation of the Gaussian learner using bounded-action expert demonstrations. Panels (a) and (b) correspond to one and 20 demonstrations, respectively, and compare BC, DRIL-peak, and DRIL-final. Deterministic results are reported in Table 4 .

Expert	Method	1 trajectory		20 trajectories
		Deterministic	Stochastic	Deterministic	Stochastic
Clipped	BC	163 $\pm$ 67	216 $\pm$ 108	252 $\pm$ 143	758 $\pm$ 237
	DRIL-peak	229 $\pm$ 142	348 $\pm$ 230	210 $\pm$ 115	572 $\pm$ 280
	DRIL-final	122 $\pm$ 116	143 $\pm$ 111	265 $\pm$ 165	241 $\pm$ 140
Bounded	BC	147 $\pm$ 105	161 $\pm$ 156	567 $\pm$ 239	794 $\pm$ 227
	DRIL-peak	195 $\pm$ 88	200 $\pm$ 184	398 $\pm$ 225	738 $\pm$ 277

Table 5 : Beta learner evaluation on CarRacing with the fixed Gaussian disagreement ensemble. Values are mean

\pm

standard deviation over 100 evaluation episodes. Bold values mark the highest mean scores within each evaluation mode and dataset regime.

Panel of Figure 8 — Figure 8 : Training dynamics of the Beta learner using clipped-action expert demonstrations and the fixed Gaussian disagreement ensemble. Panels (a) and (b) correspond to one and 20 demonstrations, respectively.

Panel of Figure 9 — Figure 9 : Stochastic 100-episode evaluation of the Beta learner using clipped-action expert demonstrations and the fixed Gaussian disagreement ensemble. Panels (a) and (b) correspond to one and 20 demonstrations, respectively, and compare BC, DRIL-peak, and DRIL-final. Deterministic results are reported in Table 5 .

Panel of Figure 10 — Figure 10 : Training dynamics of the Beta learner using bounded-action expert demonstrations and the fixed Gaussian disagreement ensemble. Panels (a) and (b) correspond to one and 20 demonstrations, respectively.

Panel of Figure 11 — Figure 11 : Stochastic 100-episode evaluation of the Beta learner using bounded-action expert demonstrations and the fixed Gaussian disagreement ensemble. Panels (a) and (b) correspond to one and 20 demonstrations, respectively, and compare BC, DRIL-peak, and DRIL-final. Deterministic results are reported in Table 5 .

Expert	Trajectories	Best BC	Best DRIL-peak	Change of DRIL peak
Clipped	1	Beta: 216 $\pm$ 108	Beta: 348 $\pm$ 230	+61.1%
Clipped	20	Beta: 758 $\pm$ 237	Gaussian: 802 $\pm$ 197	+5.8%
Bounded	1	Beta: 161 $\pm$ 156	Gaussian: 341 $\pm$ 246	+111.8%
Bounded	20	Beta: 794 $\pm$ 227	Beta: 738 $\pm$ 277	$-7.1$ %

Table 6 : Best stochastic BC and score-selected DRIL checkpoint in each demonstration regime, allowing the learner distribution to differ. Percent change is computed from the reported means.

Figure 12 : Disagreement signal during the first 1000 interaction steps of the Gaussian learner. The left column uses one expert trajectory and the right column uses 20 trajectories. The top rows show steering and brake/throttle samples with the valid action interval shaded; the middle rows show scalar ensemble uncertainty and the 98th-percentile threshold; the bottom rows show the resulting binary reward. Sampled Gaussian actions frequently extend beyond the valid interval before clipping, and occasional uncertainty values cross the threshold.

Figure 13 : Disagreement signal during the first 1000 interaction steps of the Beta learner using the same fixed Gaussian ensemble. The left column uses one expert trajectory and the right column uses 20 trajectories. The top rows show bounded steering and brake/throttle actions; the middle rows show scalar ensemble uncertainty and the 98th-percentile threshold; the bottom rows show the binary reward. Threshold crossings are infrequent, so the reward remains positive over most of the rollout.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Demonstrations	Expert threshold	BC	DRIL
Breakout	1 trajectory	300	5	355
Breakout	3 trajectories	300	7	339
LunarLanderContinuous	1 trajectory	200	194	257
LunarLanderContinuous	20 trajectories	200	232	228

Appendix

Table 7 : Implementation checks in the two regimes considered by the original DRIL study. Each entry is the mean score over 100 evaluation episodes. The expert threshold is the task score used to define successful or expert-level performance in the corresponding experiment.

Panel of Figure 14 — Figure 14 : Implementation-validation results. The figures show the qualitative behavior of DRIL in the original discrete-visual and continuous-low-dimensional regimes before the image-based continuous-control study.

Explore similar work

Difference-Aware Retrieval Policies for Imitation Learning
Jun 8, 2026Quinn Pfeifer, Ethan Pronovost, Paarth Shah +3Imitation Learning Behavior Cloning
1Paul G. Allen School of Computer Science & Engineering, University of Washington · 2Toyota Research Institute · 3Google DeepMind +1
BlenDAgger: Blended Shared Control for Interactive Imitation Learning
Sep 29, 2026Cailyn Smith, Geoffrey Sun, Henny Admoni +1Robotic Manipulation Policies Robot Policies
The Robotics Institute, School of Computer Science Carnegie Mellon University Pittsburgh, USA · School of Computer Science Carnegie Mellon University Pittsburgh, USA
Language-Critique Imitation Learning from Suboptimal Demonstrations
Jul 1, 2026Chih-Han Yang, Dai-Jie Wu, Yun-Ping Huang +3Temporally Coherent Imitation Learning Strong Imitation Learning
Graduate Institute of Communication Engineering, National Taiwan University (NTU) · University of Utah · National Yang Ming Chiao Tung University +1

Jun 8, 2026, cs.ROJ/K move · Enter open · S save

Difference-Aware Retrieval Policies for Imitation Learning

Quinn Pfeifer, Ethan Pronovost, Paarth Shah, Khimya Khetarpal+2

1Paul G. Allen School of Computer Science & Engineering, University of Washington · 2Toyota Research Institute · 3Google DeepMind+1

Imitation Learning Behavior Cloning Imitation

Open paper AskPDF arXiv

Parametric imitation learning via behavior cloning can suffer from poor generalization to out-of-distribution states due to compounding errors during deployment. We show that reusing the training data during inference via a semi-parametric retrieval-based imitation learning approach can alleviate this challenge. We present Difference-Aware Retrieval Policies for Imitation Learning (DARP), a semi-parametric retrieval-based imitation learning approach that addresses this limitation by reparameterizing the imitation learning problem in terms of local neighborhood structure rather than direct state-to-action mappings. Instead of learning a global policy, DARP trains a model to predict actions based on

k

-nearest neighbors from expert demonstrations, their corresponding actions, and the relative distance vectors between neighbor states and query states. DARP requires no additional assumptions beyond those made for standard behavior cloning -- it does not require additional data collection, online expert feedback, or task-specific knowledge. We demonstrate consistent performance improvements of 15-46% over standard behavior cloning across diverse domains, including continuous control and robotic manipulation, and across different representations, including high-dimensional visual features. Code and demos are available at https://weirdlabuw.github.io/darp-site/.

Abstract

Explore similar work

Difference-Aware Retrieval Policies for Imitation Learning

BlenDAgger: Blended Shared Control for Interactive Imitation Learning

Language-Critique Imitation Learning from Suboptimal Demonstrations