DexDICE: Reconciling Safety and Performance via Dual-Expert Offline Imitation Learning
Authors: Seokin Seo, Sung Kuk Shyn, Minji Seo, Yisoo Lee, Jongmin Lee, Hongseok Yang, Kee-Eung Kim
Organizations: Korea Institute of Science and Technology (KIST) · Korea Advanced Institute of Science and Technology (KAIST) · Hyundai Motor Company · Yonsei University · Korea Institute for Advanced Study (KIAS)
Ensuring high returns while providing strong safety guarantees in offline imitation learning (IL) is a fundamental challenge: existing approaches either rely on explicitly specified cost functions, which are difficult to design for complex real-world safety constraints, or require expert demonstrations that are both safe and high-performing, which are often too costly to collect. We introduce dual-expert imitation learning, which leverages two complementary data sources: (i) safety-compliant demonstrations that faithfully follow safety guidelines but may be suboptimal, and (ii) performance-oriented demonstrations that achieve high returns but may violate safety guidelines. To address this problem, we propose DexDICE (Dual-expert offline imitation learning via stationary DIstribution Correction Estimation). Instead of conventional distribution constraints, DexDICE combines DICE-style offline learning with safe support constraints, enabling agents to exploit high-return behaviors while remaining within safety-compliant regions. We prove that the ideal solution satisfying this constraint exactly has zero unsafe-region visitation with high probability. We evaluate DexDICE on a re-curated DSRL benchmark and validate it on a real-world mobile-robot navigation task trained directly from human teleoperation demonstrations, where it substantially outperforms baselines and remains robust to data scarcity and quality degradation.
Figures & tables
Figure 1: (left) Taxonomy of safe sequential decision-making problems by available performance and safety signals. (middle) Dual-expert IL instance: DP prioritizes fast goal reaching but is possibly unsafe, while DS stays safe but possibly suboptimal. (right) Prior offline IL relies on divergence regularization (e.g., behavior regularization), biasing the policy toward DS ’s suboptimal behaviors ( red path); DexDICE instead imposes a support constraint from DS ( blue path), recovering DP ’s near-optimal route within the safe support.
Figure 2: Data efficiency (a) and sample quality (b) on the safe demonstration set DS across methods, mirroring counterpart plots on the performant demonstration set DP (c), (d) respectively. Results are averaged over 6 tasks. Each subplot examines: (a) varying NDS=2k , k∈{1,3,5,7,10} , with NDP=10 fixed; (b) varying quality(DS)∈{5%,25%,50%,75%,95%} with NDP=NDS=20 and quality(DP)=50% fixed; (c), (d) the counterparts on DP with the roles of DS and DP exchanged. The green shaded region denotes the cost constraint-satisfying area.
Figure 3: Real-world mobile-robot navigation. (a) Arena: start (green) to goal (red) avoiding two obstacles. (b) Per-method results over 10 trials. Success : reached goal without collision. Time-to-goal : on successful trials. Violation rate : fraction of timesteps within an unsafe distance to obstacles.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A : An illustration on tabular experiment. (a) The agent must reach the goal while avoiding unsafe zones. (b) Each point shows the mean return and cost (averaged over 50 seeds) for a given demonstration quality (indicated by labels). The support constraint remains in the high return, low cost region across all settings, while the distribution constraint degrades when dS is suboptimal.
Task
Obs. Dim
Act. Dim
Cost limit
Safe Navigation Tasks
SafetyPointCircle1
28
2
25
SafetyPointCircle2
28
2
25
Safe Velocity Tasks
SafetyHopperVelocity
11
3
25
SafetyWalker2dVelocity
17
6
25
Appendix
Table G: Descriptions of Tasks in Safety Gymnasium.
Figure B : Overall data efficiency averaged over six tasks of each method by jointly varying k , where NDP=NDS=2k .
Figure C : Per-task analysis on data efficiency with jointly varying k where NDP=NDS=2k .
Figure D : Per-task analysis on data efficiency of safe demonstrations DS .
Figure E : Per-task analysis on data efficiency of performant demonstrations DP .
Figure F : Per-task performance for dependencies on quality(DS) .
Figure G : Per-task performance analysis for dependencies on quality(DP) .
Figure H : Sensitivity analysis of DexDICE with respect to the target FNR κ used as the threshold for \mathds1dS . Experiments are conducted on controlled datasets of the PointCircle1 task with NDP=NDS=10 , over 5 random seeds.
Figure I : Stability under corruption of safe demonstrations. We evaluate on the controlled dataset of the PointCircle1 task, using a sufficient number of demonstrations ( NDP=200 , NDS=200 ). The x-axis indicates the corruption rate of DS , i.e., the proportion of safe demonstrations replaced with unsafe demonstrations that exceed the episode cost upper bound. Both DexDICE and DemoDICE-S maintain relatively high episode return and low episode cost even under moderate corruption, while other methods degrade significantly as DS becomes increasingly corrupted.
Figure J : Visualization of per-trial trajectories for the five evaluated methods across 10 deployment trials each. Each colored polyline represents one rollout; obstacle boxes are shown in gray.
Recent imitation learning (IL) algorithms such as flow-matching and diffusion policies demonstrate remarkable performance in learning complex manipulation tasks. However, these policies often fail even when operating within their training distribution due to extreme sensitivity to initial conditions and irreducible approximation errors that lead to compounding drift. This makes it unsafe to deploy IL policies in the field where out-of-distribution scenarios are prevalent. A prerequisite for safe deployment is enabling the policy to determine whether it can execute a task the way it was learned from demonstrations. This paper presents TAIL-Safe, a principled approach to identify, for a trained IL policy, a safe set from where the policy empirically succeeds in completing the learned task. We propose a Lipschitz-continuous Q-value function that maps state-action pairs to a long-term safety score based on three short-term task-agnostic criteria: visibility, recognizability, and graspability. The zero-superlevel set of this function characterizes an empirical control invariant set over state-action pairs. When the nominal policy proposes an action outside this set, we apply a recovery mechanism inspired by Nagumo's theorem that uses gradient ascent to the Q-function to steer the policy back to safety. To learn this Q-function, we construct a high-fidelity digital twin using Gaussian Splatting that enables systematic collection of failure data without risk to physical hardware. Experiments with a Franka Emika robot demonstrate that flow-matching policies, which fail under run-time perturbations, achieve consistent task success when guided by the proposed TAIL-Safe.
Dexterous manipulation presents substantial challenges for imitation learning due to its high-dimensional action space and complex contact-rich dynamics. Policies trained purely from demonstrations often suffer from compounding errors during deployment and require large amounts of expert data to achieve reliable performance. To move beyond the limitations of demonstration data, in this work, we propose DexPIE, a post-training framework for dexterous policy improvement from experience collected through real-world deployment. First, DexPIE enables effective exploration coverage through a dexterous-hand-adapted intervention system and multi-stage DAgger-style data collection across initial and intermediate task stages. Meanwhile, we enhance consistency between training and inference to reduce the distribution shift between rollouts and demonstration data, better aligning rollout behavior with demonstrations, allowing the critic to learn a value function induced by a more consistent underlying policy. Together, these components provide reliable supervision for policy evaluation. Finally, DexPIE improves the policy through conditioning on a continuous optimality indicator, allowing the policy to leverage the quality of data in a more fine-grained manner. Across three challenging real-world dexterous manipulation tasks, DexPIE achieves a 37.3% improvement in success rate over the demonstration-based reference policy, outperforming all baseline methods and demonstrating stronger robustness. The source code and dataset will be made publicly available.
A robot that can be taught a new task from a handful of demonstrations has to work out for itself what it still cannot do, and then ask for exactly that. Interactive imitation learning takes a step in that direction by letting a policy practice on its own and calling an expert when it goes wrong. Existing methods decide when to interrupt the learner. A further 2 decisions are left to whichever episode happened to trigger the interruption: which failure to correct, and where the demonstration should start. This paper is a first attempt at making both of them deliberately. DISEIL (Demonstration dIstillation for Sample-Efficient Imitation Learning) marks each failed episode at the step where the policy first becomes unreliable, represents that moment with a geometric descriptor, and groups the failures into recurring failure modes. A vision-language model and a language model read the selected mode and write a request for the next demonstration, and a store of task constraints checks that the request can be carried out before any expert time is spent. No model produces a robot action. Across 5 simulated tasks under state and image observations, changing only what the expert is asked for gives the highest mean held-out success rate in all 10 settings, with a tie in 1, and the margin is widest at the smallest budget we tested. The scope is narrow: a single round of practice at a time, in simulation, with experts that are mostly scripted. The longer-term aim is a learner that also tracks what its demonstration set already covers, and that asks a human teacher for the missing behavior in proportion to the effort each request costs them.
Suyog Khanal, Arun Kumar A, Santu Rana
Deakin Applied Artificial Intelligence Initiative Geelong, Australia