Tune to Learn: How Controller Gains Shape Robot Policy Learning

Antonia Bronars; Younghyo Park; Pulkit Agrawal

cs.ROApr 2, 2026

Tune to Learn: How Controller Gains Shape Robot Policy Learning

Authors: Antonia Bronars, Younghyo Park, Pulkit Agrawal

Organizations: MIT · MIT Improbable AI Lab

Abstract

Position controllers have become the dominant interface for executing learned manipulation policies. Yet a critical design decision remains understudied: how should we choose controller gains for policy learning? The conventional wisdom is to select gains based on desired task compliance or stiffness. However, this logic breaks down when controllers are paired with state-conditioned policies: effective stiffness emerges from the interplay between learned reactions and control dynamics, not from gains alone. We argue that gain selection should instead be guided by learnability: how amenable different gain settings are to the learning algorithm in use. In this work, we systematically investigate how position controller gains affect three core components of modern robot learning pipelines: behavior cloning, reinforcement learning from scratch, and sim-to-real transfer. Through extensive experiments across multiple tasks and robot embodiments, we find that: (1) behavior cloning benefits from compliant and overdamped gain regimes, (2) reinforcement learning can succeed across all gain regimes given compatible hyperparameter tuning, and (3) sim-to-real transfer is harmed by stiff and overdamped gain regimes. These findings reveal that optimal gain selection depends not on the desired task behavior, but on the learning paradigm employed. Project website: https://younghyopark.me/tune-to-learn

Figures & tables

Panel of Figure 1 — Fig. 1 : Different robot learning paradigms prefer different controller gain interfaces. Colored regions indicate gain regimes where each paradigm succeeds. Contrary to conventional wisdom of tuning gains for desired task compliance, optimal gains depend on the learning paradigm. Based on our experimental findings, heatmaps illustrate representative gain preferences for (a) behavior cloning, which favors compliant, overdamped gains, (b) reinforcement learning, which adapts to nearly any setting, and (c) sim-to-real transfer, which is degraded by stiff and overdamped gains.

Panel of Figure 2 — Fig. 2 : Controller gains induce diverse action–response dynamics. We evaluate a broad range of representative gain configurations and their resulting dynamic responses to assess their impact on learnability.

Panel of Figure 3 — Fig. 3 : Tracking response curves from existing robot datasets reveal tight command-following behavior, suggesting stiff controller gains are prevalent in existing data collection pipelines.

Panel of Figure 4 — Fig. 4 : Task-level impedance can be decoupled from low-level controller gains with learned policies. A learned policy can achieve (a) compliant behavior despite stiff low-level gains, and (b) stiff behavior despite compliant gains.

Panel of Figure 5 — Fig. 5 : Behavior cloning prefers compliant and overdamped controller gains. Closed-loop rollout success rates across a grid of proportional ( $\mathbf{K}_{p}$ ) and derivative ( $\mathbf{K}_{d}$ ) gains for diverse manipulation tasks and robot embodiments. Each heatmap reports success averaged over evaluation rollouts. Across tasks, higher success rate (darker red) consistently concentrates in the compliant, overdamped regime (upper-left), while stiff or weakly damped controllers yield degraded performance.

Fig. 6 : Any teleoperation system requires a mapping $\phi$ from user inputs $\mathbf{u}$ to desired position targets $\mathbf{x}_{\text{des}}$ for the controller, which substantially shapes how the robot is perceived under different controller gains.

Panel of Figure 8 — Fig. 8 : Compliant controllers attenuate action errors. (a) Validation MSE loss during training: compliant gains yield higher loss, while stiff gains achieve lower loss. (b) Open-loop success rate under action noise: compliant gains maintain high success while stiff gains completely fail. (c) Compliant gains keep the perturbed trajectory close to the original, while (d) stiff gains cause large deviations that lead to task failure.

Panel of Figure 9 — Fig. 9 : Teleoperation performance under different gain regimes. With optimized input mapping $\phi^{\star}(\mathbf{K})$ (Eq. 7 ), compliant and overdamped controllers (grid top-left) achieve similar or better success rates, user ratings, and shorter completion time to stiffer settings.

Panel of Figure 10 — Fig. 10 : RL training across gain regimes. (a–b) Success rate across the hyperparameter landscape varies among gain settings and tasks; policies with 95%+ success rate (green circles) are found across all conditions. (c–d) Sample efficiency and training stability of PPO is comparable across gain regimes for both tasks.

Fig. 11 : End-to-end BC pipeline still favors compliant and overdamped gain regime.

Task / Platform	Existence Proof
FR3 Joint-Reach	✓
FR3 EE-Reach	✓
FR3 Lift-Cube	✓
FR3 Open-Drawer	✓
G1 Track-Velocity	✓
Allegro In-Hand Cube Manipulation	✓

TABLE I : RL solution existence across gain regimes. For each task, we verify that at least one successful policy can be discovered in every gain configuration given appropriate environment shaping. A checkmark indicates that gain regimes corresponding to four corner extremes in Fig. 2 yield working controllers (99%+ success rate). Videos or live policy rollouts of discovered behaviors for each gain settings are available on our project website .

Panel of Figure 13 — Fig. 12 : Stiff and overdamped gain settings yield lower SysID modeling errors, but exhibit larger closed-loop Sim2Real errors . Policy observations during closed-loop rollout evolve similarly between sim and real (b-left) for compliant, overdamped gains, but very dissimilarly (b-right) for stiff, overdamped gains.

Panel of Figure 14 — Fig. 13 : Stiff and overdamped gain settings reduce sim2real transferability. The Sim2Real trajectory error (Eq. 11 ) is consistently larger (light blue) in the stiff and overdamped regime (a-c). The primary Sim2Real failure mode is high-frequency oscillation (d).

Fig. 14 : Jitter Failures vs.

\Delta t

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 15 : Effective Cartesian stiffness throughout training for the two counterintuitive pairings. Despite $32\times$ lower actuator stiffness, the stiff-behavior policy achieves ${\sim}5\times$ higher effective task-level stiffness than the compliant-behavior policy.

Panel of Figure 17 — Fig. 16 : Six tasks used for behavior cloning.

Fig. 17 : TPR fidelity across gain configurations and decimation rates. Success rate and joint-position MSE remain robust ( ${\geq}90\%$ success, MSE ${<}10^{-3}$ ) up to $25\times$ decimation (20 Hz) for all gain settings.

Task	$\mathcal{G}^{\text{CO}}$	$\mathcal{G}\setminus\mathcal{G}^{\text{CO}}$	$\beta_{\mathbf{K}_{\text{p}}}$	$\beta_{\mathbf{K}_{\text{d}}}$	$p$ -value
	Success Rate		Logistic Reg.		Barnard’s
Dishwasher Opening	70.1%	45.6%	$-0.074$	$+0.277$	$1.6\times 10^{-51}$
Bimanual Handover	35.1%	16.5%	$-0.173$	$+0.225$	$6.2\times 10^{-35}$
Mug Hanging	62.4%	40.9%	$-0.146$	$+0.155$	$1.4\times 10^{-40}$
Dishrack Unloading	12.7%	6.9%	$-0.144$	$+0.159$	$1.2\times 10^{-5}$
Dishrack Loading	28.2%	14.8%	$-0.169$	$+0.217$	$2.3\times 10^{-19}$

Appendix

TABLE II : Statistical analysis of BC results. Success rates for compliant-overdamped (

\mathcal{G}^{\text{CO}}

) vs. other gain regions, logistic regression coefficients on

\log_{2}\mathbf{K}_{\text{p}}

and

\log_{2}\mathbf{K}_{\text{d}}

, and Bonferroni-corrected one-sided Barnard’s exact test

p

-values.

\mathcal{H}_{0}

is rejected in all cases.

Panel of Figure 20 — Fig. 18 : Non-prehensile box manipulation task for the user study. A single trial of the task involves teleoperating the robot from a reset pose to make contact with the box, then pushing the box towards the goal. The task is complete when the green square is completely occluded by the box (b).

Fig. 19 : User study survey. After each trial, users complete the survey to rate their subjective experience teleoperating with a given gain setting.

Panel of Figure 22 — Fig. 20 : Five tasks for online RL solution existence proof. For each task, we trained a successful policy for 8+ gain configurations spanning the range of stiff / compliant, overdamped / underdamped.

Panel of Figure 23 — Fig. 21 : Action representation. The policy output is scaled by a per-joint-group vector $\boldsymbol{\alpha}$ and added to a reference position $\mathbf{q}_{\text{ref}}$ to produce the position target $\mathbf{q}_{\text{des}}$ sent to the PD controller.

Task	$\mathbf{q}_{\text{ref}}(t)$	$\mathcal{G}_{1}$	$\mathcal{G}_{2}$	Gripper
FR3 Joint-Reach	$\mathbf{q}(t)$	$q_{0\text{--}3}$ (elbow)	$q_{4\text{--}6}$ (wrist)	–
FR3 EE-Reach	$\mathbf{q}(t)$	$q_{0\text{--}3}$ (elbow)	$q_{4\text{--}6}$ (wrist)	–
FR3 Lift Cube	$\mathbf{q}(t)$	$q_{0\text{--}3}$ (elbow)	$q_{4\text{--}6}$ (wrist)	binary
FR3 Open Drawer	$\mathbf{q}(t)$	$q_{0\text{--}3}$ (elbow)	$q_{4\text{--}6}$ (wrist)	binary
G1 Locomotion	$\mathbf{q}_{0}$	$q_{0\text{--}13}$ (lower)	$q_{13\text{--}36}$ (upper)	–

Appendix

TABLE III : Action representation across RL tasks.

Task	Criterion	Threshold
FR3 Joint-Reach	$\\|\mathbf{q}-\mathbf{q}_{\text{goal}}\\|<\epsilon$	$\epsilon=0.1$ rad
FR3 EE-Reach	$\\|\mathbf{p}-\mathbf{p}_{\text{goal}}\\|<\epsilon_{p},$	$\epsilon_{p}=0.02$ m
	$\\|\Delta\theta\\|<\epsilon_{r}$	$\epsilon_{r}=0.1$ rad
FR3 Lift Cube	$\\|\mathbf{p}_{\text{obj}}-\mathbf{p}_{\text{goal}}\\|<\epsilon$	$\epsilon=$ 0.05m
FR3 Open Drawer	$d_{\text{drawer}}>d_{\text{min}}$	$d_{\text{min}}=0.2$
G1 Locomotion	$\\|\dot{\mathbf{q}}\\|/\\|\dot{\mathbf{q}}_{\text{goal}}\\|>\rho$	$\rho=0.4$

Appendix

TABLE IV : Success criteria for each RL task.

Hyperparameter	Value
Algorithm	PPO (SKRL)
Discount factor $\gamma$	$0.99$
GAE $\lambda$	$0.95$
Learning epochs	$5$
Clip range (ratio)	$0.2$
Clip range (value)	$0.2$

Appendix

TABLE V : PPO hyperparameters shared across all tasks.

Hyperparameter	Reach	Lift	Drawer	G1
Network layers	[64,64]	[256,128,64]	[256,128,64]	[256,128,128]
Learning rate	$1\text{e-}3$	$1\text{e-}3$	$5\text{e-}4$	$1\text{e-}3$
KL threshold	$0.01$	$0.01$	$0.008$	$0.01$
Rollouts	$24$	$24$	$96$	$24$
Mini-batches	$4$	$4$	$96$	$4$
Entropy coeff.	$0.01$	$0.01$	$0.001$	$0.008$

Appendix

TABLE VI : PPO hyperparameters that vary across tasks.

Parameter	Lower	Upper
Stiffness $K_{p}$	$1$	$1024$
Damping $K_{d}$	$1$	$1024$
Armature	$0$	$0.5$
Static friction	$0.01$	$1.0$
Dynamic friction ratio	$0$	$1.0$
Viscous friction	$0$	$1.0$

Appendix

TABLE VII : System identification parameter bounds. Parameters are optimized per-actuator.

Panel of Figure 29 — Fig. 22 : System identification result for sample gain settings in each gain regime. We show commanded positions (green), real-world achieved positions (orange), and simulation positions (blue) achieved with the optimal actuator parameters.

Panel of Figure 30 — Fig. 23 : Behavior cloning performance across dataset size. Success rate per gain setting for the Block Stack task. The preference for compliant and overdamped gain settings is maintained across dataset sizes (a-c).

Panel of Figure 31 — Fig. 24 : Behavior cloning performance across policy architectures. Success rate per gain setting for the Block Stack task. The preference for compliant and overdamped gain settings is maintained across policy architectures (a-c).

Panel of Figure 32 — Fig. 25 : Behavior cloning performance across action chunk size. Success rate per gain setting for the Block Stack task. The preference for compliant and overdamped gain settings is observed when predicting both single actions (a) and action chunks (b).

Panel of Figure 33 — Fig. 26 : Behavior cloning performance across action representations. Success rate per gain setting for the Block Stack task. The preference for compliant and overdamped gain settings is observed when predicting both absolute (a) and relative (b) joint position actions.

Panel of Figure 34 — Fig. 27 : Behavior cloning performance across control frequencies. Success rate per gain setting for the Block Stack task. The preference for compliant and overdamped gain settings is observed when predicting actions at 10Hz (a) and 50Hz (b).

Panel of Figure 35 — Fig. 28 : Offline imitation learning scales more favorably under compliant and overdamped gains. Success rate as a function of dataset size across tasks and robot embodiments. Policies trained with low stiffness and high damping achieve higher success with fewer demonstrations, while stiff or weakly damped controllers exhibit poorer data scaling.

Panel of Figure 36 — Fig. 29 : Stiff and overdamped gain settings reduce sim2real transferability. The Sim2Real trajectory error (Eq. 11 ) is consistently larger (light blue) in the stiff and overdamped regime (a-c).

Panel of Figure 37 — Fig. 30 : Stiff and overdamped gains increase Sim2Real NN error. The Sim2Real NN error (Eq. 31 ) is consistently larger (light blue) in the stiff and overdamped regime (a-c).

Condition	$\mathcal{G}^{\text{SO}}$	$\mathcal{G}\setminus\mathcal{G}^{\text{SO}}$	$\beta_{\mathbf{K}_{\text{p}}}$	$\beta_{\mathbf{K}_{\text{d}}}$	$p$ -value
	Mean Error		OLS Reg.		Mann-Whitney
Joint-Reach (no DR)	0.043	0.010	$+0.298$	$+0.087$	$1.9\times 10^{-36}$
Joint-Reach (with DR)	0.060	0.018	$+0.354$	$+0.215$	$1.7\times 10^{-20}$
EE-Reach (no DR)	0.198	0.123	$+0.063$	$+0.127$	$3.5\times 10^{-36}$

Appendix

TABLE VIII : Statistical analysis of Sim2Real results. Mean trajectory error for stiff-overdamped (

\mathcal{G}^{\text{SO}}

) vs. other gain regions, OLS regression coefficients on

\log_{2}\mathbf{K}_{\text{p}}

and

\log_{2}\mathbf{K}_{\text{d}}

, and Bonferroni-corrected one-sided Mann-Whitney U test

p

-values.

\mathcal{H}_{0}

is rejected in all cases.

Explore similar work

May 30, 2026

Too Much of a Good Thing: When sim2real Efforts Impede Policy Learning (And What to Do About It)

While sim2real efforts are necessary for effective policy transfer to hardware, there is such a thing as too much of a good thing. We argue that sim2real efforts have led to misaligned incentives with policy learning, resulting in simulator lock in and poor policy exploration due to the unreasonable constraints imposed by the real world. We offer a diagnosis and explanation of the current status of the problem, and propose a potential solution via a sim2sim2real paradigm that leverages the robot's kinematics as the sole design constraint.

Kyle Morgenstein, Bharath Masetty, Stephen Welch +1

Apptronik · University of Texas at Austin

Robot Policy Learning Robotics Simulation Sim-to-Real Transfer Transfer Learning

Ask PDF

Oct 12, 2025

Gain Tuning Is Not What You Need: Reward Gain Adaptation for Constrained Locomotion Learning

Existing robot locomotion learning techniques rely heavily on the offline selection of proper reward weighting gains and cannot guarantee constraint satisfaction (i.e., constraint violation) during training. Thus, this work aims to address both issues by proposing Reward-Oriented Gains via Embodied Regulation (ROGER), which adapts reward-weighting gains online based on penalties received throughout the embodied interaction process. The ratio between the positive reward (primary reward) and negative reward (penalty) gains is automatically reduced as the learning approaches the constraint thresholds to avoid violation. Conversely, the ratio is increased when learning is in safe states to prioritize performance. With a 60-kg quadruped robot, ROGER achieved near-zero constraint violation throughout multiple learning trials. It also achieved up to 50% more primary reward than the equivalent state-of-the-art techniques. In MuJoCo continuous locomotion benchmarks, including a single-leg hopper, ROGER exhibited comparable or up to 100% higher performance and 60% less torque usage and orientation deviation compared to those trained with the default reward function. Finally, real-world locomotion learning of a physical quadruped robot was achieved from scratch within one hour without any falls. Therefore, this work contributes to constraint-satisfying real-world continual robot locomotion learning and simplifies reward weighting gain tuning, potentially facilitating the development of physical robots and those that learn in the real world.

Arthicha Srisuchinnawong, Poramate Manoonpong

Vidyasirimedhi Institute of Science and Technology, Rayong, Thailand. · The University of Southern Denmark, Odense, Denmark.

Constrained RL RL for Legged Locomotion Reward Design for RL Quadruped Locomotion

Ask PDF

Aug 23, 2026

Enhancing Sim2Real Transfer for Torque-Controlled Robots through Real2Sim Dynamics Estimation and Reinforcement Learning

Transferring reinforcement learning policies from simulation to Real-World robots remains a major challenge, particularly when dealing with low-level torque control, where even small modelling inaccuracies can lead to unstable or unsafe behaviours. In this work, we propose a Real2Sim2Real pipeline that improves Sim2Real transfer for torque-controlled robotic arms by combining trajectory matching, parameter optimization via genetic algorithms, and domain randomization. Using the 7-DOF Franka Emika Panda robot, we first identify friction, inertia, and gravity compensation parameters by minimizing the error between real and simulated joint trajectories. These calibrated dynamics are then used to train a TQC-based reinforcement learning agent in simulation. The trained policy is evaluated in both Gazebo and MuJoCo environments, and finally deployed on the real robot. Our results demonstrate a significant improvement in tracking accuracy and policy robustness after parameter tuning, with smooth policy transfer from simulation to the Real-World across multiple target-reaching tasks. This work highlights the effectiveness of accurate physical modelling in enabling stable and generalizable torque-based reinforcement learning policies.

Davide Bargellini, Alex Pasquali, Andrea Govoni +2

Department of Electrical, Electronic, and Information Engineering (DEI), University of Bologna, 40136 Bologna, Italy · Robotics and Mechatronics (RaM) group, University of Twente, Enschede, The Netherlands

Sim-to-Real Transfer Robotic RL RL Control

Ask PDF