Inner Product Regularization for Imitation Learning
Authors: Benjamin A. Christie, Tianyu Qiu, Amy H. Choi, David Fridovich-Keil, Dylan P. Losey
Organizations: Virginia Tech, Department of Mechanical Engineering · The University of Texas at Austin, Department of Aerospace Engineering and Engineering Mechanics · Korea Advanced Institute of Science and Technology, Department of Industrial and Systems Engineering
Robots that learn from demonstrations should extrapolate their policy despite state perturbations such as sensor noise, scene variation, and compounding errors. One approach to this robustness is regularization. Prior regularizers consider each state on its own and limit how sensitive the policy is to any perturbation at that state. By contrast, we explore how the policy's output shifts when given the same perturbation at different states. Our insight is that, locally, the robot should respond to state perturbations in a consistent manner: e.g., if changing an irrelevant object position does not impact the policy at one state, that same change should not shift the policy output at a nearby state. Applying this insight, we develop inner product regularization (InneR). We show that the inner product on action space induces an inner product on state space through the policy Jacobian (the Fisher information for stochastic policies), and that the null space of this inner product governs which state perturbations have a first-order effect on the policy. By clustering states whose inner products are aligned, averaging the inner products within each cluster, and minimizing the smallest eigenvalues of that average, InneR shapes the learned policy so that nearby states share a common null space. InneR enforces consistent responses to perturbations without limiting policy expressiveness, and can be appended to existing imitation learning pipelines for deterministic or stochastic policies. In simulated experiments, InneR makes learned policies more robust to irrelevant state dimensions and shifts between training and test distributions, whereas norm penalties often perform worse than behavior cloning. On a real robot arm performing three manipulation tasks, InneR raises the average success rate of a diffusion policy from 18% to 57% without additional data.
Figures & tables
Figure 1: (Left) Robot learning to close a coffee machine. At test time the robot encounters a new state. As compared to the states in its training dataset, this new state has some perturbations that are task-relevant (shifting the pose of the coffee machine, Δ1 ) and some that are irrelevant (shifting the banana, Δ2 ). (Right) Local state perturbations have a first-order effect on the policy outputs. At each state the policy is sensitive to some perturbation directions and ignores others; averaging over nearby states reveals the directions to which the policy is most and least sensitive. Under standard behavior cloning these responses are inconsistent across nearby states, and thus the policy reacts to both Δ1 and Δ2 . We propose InneR, an inner product regularization approach that enforces similar responses to the same perturbation at nearby states: the policy still responds to Δ1 but now ignores Δ2 .
Algorithm 1 InneR for Deterministic Policies
Figure 2: Single-integrator simulation from Section 7.1 . Plots show the error between the robot and the goal as state dimension n increases (lower is better). The far left of each plot corresponds to the minimum state dimension for that setting, and each added state element is irrelevant noise that is sampled uniformly at random in both the demonstrations and the test states. Ideally the robot is robust to these variations in the irrelevant state dimensions, and so the final state error should be constant in n . InneR is the only method whose error stays nearly flat in every setting besides Diffusion–Dynamic, where it is consistently lower than the alternatives while maintaining a similar positive slope. InneR-C matches it in static , where one global null space is correct, but fails in dynamic . Shaded regions show standard error across 10 seeds.
Figure 3: Single integrator simulation from Section 7.2 on the static task with state dimension n=16 . (Left) Final state error as the number of demonstrations increases for the MLP and Diffusion policies. Here InneR and InneR-C are both effective since the same state entries always contain relevant or irrelevant information. (Right) What each method regularizes, measured for the final MLP policy at 200 training states: the squared Frobenius norm of the Jacobian (log scale) and the shared null space ratio of Equation ( 26 ) ( InneR-C matches InneR , and is not shown for brevity). Standard regularizers like Frobenius and Nuclear are designed to decrease the magnitude of each Jacobian, and by flattening these individual Jacobians they actually make the perturbation responses inconsistent across the dataset: their null space ratio rises above the BC value. By contrast, InneR leaves ∥J(s)∥F2 for each individual state Jacobian at or above its BC value, and locally aligns the Jacobians so that they have a consistent null space. Colors follow Figure 2 .
Figure 4: Single-integrator simulation from Section 7.3 highlighting the dynamic task. At training time the irrelevant state dimensions are correlated with the active goal ( ρ=0.9 ), but at test time this correlation is removed, so a policy that reads the goal from the irrelevant entries fails (i.e., causal confusion de Haan et al. (2019) ). InneR is the only method that improves on BC .
Figure 5: Single-integrator simulation from Section 7.4 highlighting the dynamic task. An invertible nonlinear transformation is applied to the state before it is input to the policy, so that the task-relevant information is dispersed across every entry the policy observes and the dynamics are nonlinear in those coordinates. InneR lowers the error of BC despite this nonlinear dispersion.
Figure 6: Drone landing simulation from Section 8.1 . Plots show the landing success rate as state dimension n increases (higher is better). The state dimension sweep starts at n=19 , and each additional 7 state coordinates come from a distractor drone in both the demonstrations and the test states. InneR largely outperforms BC at all tested dimensions for MLP, and is close to BC for Diffusion. InneR-C performs poorly for MLP but is competitive for Diffusion.
Figure 7: Drone landing simulation from Section 8.2 . Plots show the landing success rate as number of demonstrations increases (higher is better) with state dimension n=22 . (Left) Success rate as the number of demonstrations increases for the MLP and Diffusion policies. InneR is effective for both MLP and Diffusion policies, and InneR-C is only effective for the Diffusion policy. (Right) What each method regularizes (measured for the final MLP policy at 200 training states), i.e. the squared Frobenius norm of the Jacobian (log scale) and the shared null space ratio of Equation ( 26 ). Both Frobenius and nuclear norm baselines reduce the Jacobian norm. Frobenius has a higher null space ratio than BC at 300, 500, and 700 demonstrations, whereas Nuclear is below BC except at 700 demonstrations. InneR consistently reduces the null-space ratio without uniformly reducing the Jacobian norm. Colors follow Figure 2 .
Figure 8: Drone landing simulation from Section 8.3 . In training, irrelevant state dimensions are correlated with the active goal ( ρ=0.9 ), but at test time this correlation is removed, so a policy that reads the goal from the irrelevant entries fails (i.e., causal confusion de Haan et al. (2019) ). InneR improves upon BC for both MLP and Diffusion, while InneR-C only improves upon BC for Diffusion.
Figure 9: Drone landing simulation from Section 8.4 . An invertible nonlinear transformation is applied to the state before it is input to the policy, so that the task-relevant information is dispersed across every entry the policy observes and the dynamics are nonlinear in those coordinates. InneR improves upon BC for MLP and InneR-C improves upon BC for Diffusion despite this nonlinear dispersion.
Figure 10: Our real-world tasks. Each task is shown at the start, middle, and end of a successful InneR rollout with the three distractors (apple, orange and banana). (Left) in close , the robot pulls the handle down to latch the lid of the coffee maker. (Middle) In stack , the robot picks up the black cube and places it on the tan cube (teal and orange segmentations, respectively). (Right) In push , the robot pushes the pot into the target region. Tracked objects are tinted by the segmentation masks that our perception pipeline uses to estimate their poses, and the background is faded for clarity.
Figure 11: Success rates in our real-world experiments for BC , Nuclear , and InneR ; the number above each bar is the success rate in percent. Colors follow Figure 2 . (Left) Close with zero to three distractors. (Middle, Right) Stack and push with zero and three distractors. Error bars show standard error over 20 trials. Averaged across the three tasks, InneR matches BC with no distractors ( 75.0% vs. 73.3% ) and outperforms both baselines with three distractors.
Figure 12: Success rate in close with three distractors as Gaussian noise is added to the object positions that the policy observes. The horizontal axis is the standard deviation of the added noise; 0 mm repeats the three distractor results of Figure 11 . InneR has the highest success rate up to 1.5 mm
Figure 13: The regularization results from our hardware experiments, measured for the three-distractor diffusion policies at training states. (Left) Trace of the Fisher information F (log scale). (Right) Shared null space ratio of Equation ( 26 ), with F in place of the inner product, r=14,k=8 . Nuclear shrinks the Fisher information by more than two orders of magnitude, whereas InneR keeps it close to BC ’s level and attains the lowest null space ratio on every task. Error bars show standard error over 8 batches of 256 states.
Figure 14: Selectivity of the three-distractor policies on states encountered during the real-world trials (higher is better). InneR is the most selective method on every task. Error bars show standard error over 400 states.
BC
Nuclear
InneR
Close
Task objects
2.193
0.636
2.072
Distractors
0.305
0.094
0.247
Stack
Task objects
1.272
0.490
1.378
Distractors
0.249
0.094
0.212
Push
Task objects
1.978
2.292
1.712
Distractors
0.259
0.479
0.211
Table 1: Sensitivity of the three-distractor policies
Regularization-based methods have become a standard approach for training Deep Reinforcement Learning policies against adversarial input perturbations. In this paper, we unify these methods by deriving new upper bounds on the performance gap between the nominal and worst-case policies. Each upper bound is expressed as an existing regularization objective plus a KL-divergence penalty between the nominal and worst-case policies, which further explains why adding a KL penalty improves robustness in practice. Building on these bounds, we formulate robust training as a constrained optimization problem, showing that existing methods correspond to the special case of a fixed Lagrange multiplier. We instead update the multiplier jointly with the policy to automatically tune the regularization weight. Finally, we conduct extensive adversarial evaluations across several continuous control tasks to validate our theoretical analysis.
Amine Andam, Jamal Bentahar, Mustapha Hedabou
Mohammed VI Polytechnic University · Khalifa University · Concordia University
Parametric imitation learning via behavior cloning can suffer from poor generalization to out-of-distribution states due to compounding errors during deployment. We show that reusing the training data during inference via a semi-parametric retrieval-based imitation learning approach can alleviate this challenge. We present Difference-Aware Retrieval Policies for Imitation Learning (DARP), a semi-parametric retrieval-based imitation learning approach that addresses this limitation by reparameterizing the imitation learning problem in terms of local neighborhood structure rather than direct state-to-action mappings. Instead of learning a global policy, DARP trains a model to predict actions based on k-nearest neighbors from expert demonstrations, their corresponding actions, and the relative distance vectors between neighbor states and query states. DARP requires no additional assumptions beyond those made for standard behavior cloning -- it does not require additional data collection, online expert feedback, or task-specific knowledge. We demonstrate consistent performance improvements of 15-46% over standard behavior cloning across diverse domains, including continuous control and robotic manipulation, and across different representations, including high-dimensional visual features. Code and demos are available at https://weirdlabuw.github.io/darp-site/.
Quinn Pfeifer, Ethan Pronovost, Paarth Shah +3
1Paul G. Allen School of Computer Science & Engineering, University of Washington · 2Toyota Research Institute · 3Google DeepMind +1
We study adversarial imitation learning (AIL), in which an agent learns to imitate expert demonstrations by optimizing a policy against an adversarial reward that distinguishes expert and learner behavior. Historically, reward regularization and entropy-based policy regularization are key components of empirically successful methods such as GAIL and LS-IQ, yet their finite-sample benefits remain underexplored. We establish fast rates for jointly regularized AIL in finite-horizon Markov decision processes with general function approximation. Our model-free algorithm, Dually Regularized AIL, combines KL policy regularization with a quadratic reward penalty weighted by expert and learner occupancies. With K online episodes and N expert trajectories, we prove a O(K1+N1) bound on the regularized imitation gap for fixed regularization parameters. Our analysis combines an online mirror descent construction for general convex reward classes to control estimation error from finite expert data and stochastic learner feedback, with a sharp analysis of optimistic KL-regularized policy learning. To the best of our knowledge, Dually Regularized AIL is the first algorithm to simultaneously achieve O(ε1) sample complexity in both expert demonstrations and online interactions for this regularized AIL objective, even with stochastic experts. These results provide a rigorous characterization of the complementary statistical benefits of reward and policy regularization in AIL.