Organizations: National Key Laboratory for Novel Software Technology and School of Artificial Intelligence, Nanjing University, China · The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data
Adversarial imitation learning (AIL) achieves high-quality imitation by mitigating compounding errors inherent to behavioral cloning (BC), yet its adversarial optimization frequently leads to training instability. A class of non-adversarial Q-based imitation learning (IL) methods, exemplified by IQ-Learn, has emerged to address this instability and is widely believed to outperform BC by leveraging online environment interactions. In this paper, we revisit IQ-Learn and prove that it in fact reduces to BC: it admits an imitation gap lower bound with quadratic dependence on the horizon and therefore remains susceptible to compounding errors. Our theoretical analysis reveals why online interactions fail to help: IQ-Learn uniformly suppresses Q-values for all actions at states not covered by demonstrations, preventing generalization beyond demonstrations. To address this fundamental limitation, we introduce Dual Q-DM, a new Q-based IL method built on Bellman constraints. Crucially, Bellman constraints drive value flow: Q-values propagate from demonstrated to unvisited states through environment dynamics, enabling generalization beyond demonstrations. We prove that Dual Q-DM is equivalent to AIL and can recover expert actions at unvisited states, thereby mitigating compounding errors. To the best of our knowledge, Dual Q-DM is the first non-adversarial IL method that is theoretically guaranteed to eliminate compounding errors. Experimental results further corroborate our theoretical findings.
Figures & tables
Method
Non-Adversarial Optimization
Environment Interactions
RL-related Design
Generalization Beyond Demos
Resolves Compounding Errors
BC
✓
✗
✗
✗
✗
IQ-Learn
✓
✓
✗
✗
✗
AIL
✗
✓
✓
✓
✓
Dual Q-DM (Ours)
✓
✓
✓
✓
✓
Table 1: Comparison of representative classes of IL methods. Here “RL-related Design” refers to whether the method involves an RL-related algorithmic component to solve sequential distribution matching. AIL directly employs a standard RL procedure, and Dual Q-DM involves Bellman constraints, which can be viewed as an analogue of Bellman equations in standard RL.
Method
Imitation Principle
Explicit Reward
RL-related Mechanism
AIL
Distribution Matching
✓
Direct RL (e.g., Bellman equations)
Dual Q-DM
Distribution Matching
✗
Bellman constraints
Table 2: Comparison between AIL and Dual Q-DM.
Figure 1: Illustration of the learned Q-functions on a Gridworld task. (a): The Gridworld environment with wall states (gray), goal state (cyan), and one demonstration trajectory (yellow). (b): IQ-Learn’s Q-function: visited states acquire high V-values (dark blue), but unvisited states exhibit equal Q-values across all actions (red equal signs), confirming that IQ-Learn cannot identify expert actions beyond demonstrations ( Theorem 1 ). (c): Dual Q-DM’s Q-function: Bellman constraints enable value flow from demonstrated to unvisited states, yielding a structured Q-function whose greedy action (yellow-outlined triangle) at unvisited states correctly points toward the expert trajectory, demonstrating generalization beyond demonstrations ( Proposition 1 ).
Figure 3: Final performance regarding different number of expert trajectories on 5 MuJoCo tasks.
Figure 4: Imitation gaps across different horizons.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: An Example of TD MDP. Here arrows denote the corresponding transitions.
Task
Expert Trajectories
Ant-v2
1
HalfCheetah-v2
1
Hopper-v2
5
Humanoid-v2
5
Walker2d-v2
1
Appendix
Table 3: Number and subsample ratio of expert trajectories for each task.
Parameter
Value
discount factor
0.99
replay buffer size
1⋅106
batch size
256
optimizer
Adam
Actor
learning rate
3⋅10−5
Appendix
Table 4: Dual Q-DM Hyper-parameters.
Parameter
Value
discount factor
0.99
replay buffer size
1⋅106
batch size
256
optimizer
Adam
Actor
learning rate
3⋅10−5
Appendix
Table 5: IQ-Learn Hyper-parameters.
Parameter
Value
discount factor
0.99
replay buffer size
1⋅106
batch size
256
optimizer
Adam
Actor
learning rate
3⋅10−5
Appendix
Table 6: LS-IQ Hyper-parameters.
Parameter
Value
discount factor
0.98
gradient penalty coefficient
10
replay buffer size
1⋅106
batch size
256
optimizer
Adam
Reward
Appendix
Table 7: HyPE Hyper-parameters.
Parameter
Value
discount factor
0.97
gradient penalty coefficient
1
replay buffer size
1⋅106
batch size
256
optimizer
Adam
Reward
Appendix
Table 8: DAC Hyper-parameters.
Parameter
Value
discount factor
0.99
batch size
256
optimizer
Adam
Actor
learning rate
1⋅10−4
number of hidden layers
2
Appendix
Table 9: ReCOIL Hyper-parameters.
Figure 6: Training curves of BC, IQ-Learn, and Reg IQ-Learn across 5 MuJoCo tasks under the offline setting, where solid lines denote the mean and shaded regions denote the standard deviation over 5 seeds. The x-axis represents the number of gradient update steps and the y-axis represents return. All three methods exhibit similarly poor performance, failing to learn an effective policy, providing direct empirical support for Theorem 1.
Figure 7: Hyperparameter sensitivity analysis of Dual Q-DM on the HalfCheetah and Humanoid MuJoCo tasks. For HalfCheetah, we vary Bound∈[0.5,1.0,2.0,5.0] and β∈[0.1,0.5,1.0,5.0] ; for Humanoid, we vary Bound∈[0.1,0.5,1.0,2.0,5.0] and β∈[0.1,3.0,5.0,7.0,15.0] . The left and right panels show sensitivity to the bound and coefficient, respectively, with the x-axis representing environment interactions and the y-axis representing return. Dual Q-DM maintains strong and robust performance across a wide range of hyperparameter values, with degradation observed only in the Humanoid environment under extremely large bounds or small coefficients. These results confirm that Dual Q-DM is robust to hyperparameter variations.