Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty

Zhaoming Xie; Kevin Karol; Jessica Hodgins

cs.ROFeb 20, 2026

Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty

Authors: Zhaoming Xie, Kevin Karol, Jessica Hodgins

Organizations: Robotics AI Institute, Cambridge, Massachusetts, USA

Abstract

Reinforcement learning provides a framework for learning control policies that can reproduce diverse motions for simulated characters. However, such policies often exploit unnatural high-frequency signals that are unachievable by humans or physical robots, making them poor representations of real-world behaviors. Existing work addresses this issue by adding a reward term that penalizes a large change in actions over time. This term often requires substantial tuning efforts. We propose to use the action Jacobian penalty, which penalizes changes in action with respect to the changes in simulated state directly through auto differentiation. This effectively eliminates unrealistic high-frequency control signals without task specific tuning. While effective, the action Jacobian penalty introduces significant computational overhead when used with traditional fully connected neural network architectures. To mitigate this, we introduce a new architecture called a Linear Policy Net (LPN) that significantly reduces the computational burden for calculating the action Jacobian penalty during training. In addition, a LPN requires no parameter tuning, exhibits faster learning convergence compared to baseline methods, and can be more efficiently queried during inference time compared to a fully connected neural network. We demonstrate that a Linear Policy Net, combined with the action Jacobian penalty, is able to learn policies that generate smooth signals while solving a number of motion imitation tasks with different characteristics, including dynamic motions such as a backflip and various challenging parkour skills. Finally, we apply this approach to create policies for dynamic motions on a physical quadrupedal robot equipped with an arm.

Figures & tables

Figure 1. Our system is able to learn smooth time-varying linear feedback policies for a simulated character to perform challenging motion skills.

Figure 2. We introduce Linear Policy Net. A fully connected multilayer perceptrons (MLP) is used to generate a feedback matrix

K_{t}

and a feedforward action

\bm{k}_{t}

from the reference state

\hat{\bm{s}}_{t}

. The final control action is generated by applying the linear feedback matrix on the simulation state plus the feedforward terms.

Metric by Motion Type	LPN	FF+Jac Pen	Lipschitz	No Reg	reward 0.01	reward 0.1	reward 1
Walking
Action Smoothness $\downarrow$	0.0016	0.0014	0.0040	0.0031	0.0032	0.0025	0.0015
High Frequency Ratio $\downarrow$	0.9	2.1	9.5	8.8	9.2	5.5	1.5
Motion Jerk $\downarrow$	115.6	108.5	139.5	134.2	134.3	134.0	106.53
Backflip
Action Smoothness $\downarrow$	0.061	0.042	0.195	0.148	0.068	0.031	N/A

Table 1. Comparison of smoothness of policies trained with different methods, average over three runs with different random seeds. We highlight the best performance. The LPN has top three performance in all the measures except for the motion jerk metric for the backflip task.

Figure 3. Learning curves of various methods, average over

3

runs with different random seeds. The total reward is evaluated only on the motion imitation reward.

Figure 4. Control action of the pitch joint on the left ankle for the backflip and table tennis footwork drill.

Explore similar work

Mar 19, 2026

Articulated-Body Dynamics Network: Dynamics-Grounded Prior for Robot Learning

Recent work in reinforcement learning has shown that incorporating structural priors for articulated robots, such as link connectivity, into policy networks improves learning efficiency. However, dynamics properties, despite their fundamental role in determining how forces and motion propagate through the body, remain largely underexplored as an inductive bias for policy learning. To address this gap, we present the Articulated-Body Dynamics Network (ABD-Net), a novel graph neural network architecture grounded in the computational structure of forward dynamics. Specifically, we adapt the inertia propagation mechanism from the Articulated Body Algorithm, systematically aggregating inertial quantities from child to parent links in a tree-structured manner, while replacing physical quantities with learnable parameters. Embedding ABD-NET into the policy actor enables dynamics-informed representations that capture how actions propagate through the body, leading to efficient and robust policy learning. Through experiments with simulated humanoid, quadruped, and hopper robots, our approach demonstrates increased sample efficiency and generalization to dynamics shifts compared to transformer-based and GNN baselines. We further validate the learned policy on real Unitree G1 and Go2 robots, state-of-the-art humanoid and quadruped platforms, generating dynamic, versatile and robust locomotion behaviors through sim-to-real transfer with real-time inference.

Sangwoo Shin, Kunzhao Ren, Xiaobin Xiong +1

University of Wisconsin–Madison. · Shanghai Innovation Institute (SII).

Robot Policy Learning Robot Learning RL for Legged Locomotion

Ask PDF

Oct 5, 2026

Interleaved Projected Gradient Descent for Safe Imitation Learning

We propose an imitation-learning design for neural-network control policies under state and input constraints. Training alternates a standard imitation gradient step with a block of

k

safety steps that pull the network's actions toward their projection onto the safe set; at run time, the controller is the trained network alone, with no safety filter. We analyze this scheme as inexact projected gradient descent in the space of policy actions. When the projected actions are recomputed at every safety step and each step moves the actions consistently toward the safe set, letting

k

grow logarithmically yields asymptotic constraint satisfaction on the training states and bounds the distance to the constrained optimum of the imitation loss; with the projected actions held fixed, the same holds only if they are exactly representable by the network. On a nonlinear autonomous racing task, we compare our method with adding a weighted constraint-violation penalty to the imitation loss. With a sufficiently large weight, our method matches the lap time of unconstrained imitation while reducing the fraction of violating episodes from

15\%

1\%

, about six times fewer than the penalty approach at its best weight. Its lap times are less sensitive to the weight, which instead sets how quickly violations vanish during training. In racing, the safety corrections are sparse and the conditions of the analysis do not hold; the gain arises instead through the data collected during training. These gains come at the cost of additional training computation.

Shengfan Cao, Francesco Borrelli

University of California, Berkeley

Gradient Descent Imitation Learning

Ask PDF

Jun 5, 2026

Spline Policy: A Structured Representation for Robot Policies

Modern imitation-learning policies for robot manipulation often represent actions as fixed-resolution action chunks, which are simple and effective but expose limited geometric and temporal structure before execution. This paper studies Spline Policy (SP), a structured representation that replaces action chunks with spline parameters while keeping the policy backbone unchanged. The predicted spline can be decoded as a compact continuous trajectory, queried at different temporal resolutions, constrained or edited in parameter space, and passed to downstream controllers. For quadratic spline outputs, the same representation can also be converted into a state-dependent vector field through an analytical distance-field construction. Under the regularity and projection assumptions of this construction, the induced dynamics do not increase the distance to the generated spline, yielding a principled local corrective mechanism around the predicted motion. The spline output further supports uncertainty propagation from observations to spline parameters, trajectories, and flow fields, and can be combined with classical control mechanisms such as null-space collision avoidance without retraining the policy backbone. We instantiate SP with diffusion, flow-matching, transformer-based, and vision-language-action backbones. Experiments in low-dimensional motion learning, simulated manipulation under matched backbones, dexterous manipulation, and real-robot case studies show that SP remains compatible with modern policy learners while exposing useful motion-structure properties, including compact decoding, temporal resampling, local correction around predicted motions, uncertainty evaluation, and controller compatibility.

Mengze Tian, Yiming Li, Sichao Liu +2

Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland · Idiap Research Institute, Martigny, Switzerland

Robot Policy Learning Robot Skill Learning Robot Motion Generation Robotic Manipulation Robot Imitation Learning

Ask PDF