PAKT: Physically-Aligned Kinesthetic Teaching for Reinforcement Learning

Lars Johannsmeier; Yashraj Narang

cs.ROSep 22, 2026

PAKT: Physically-Aligned Kinesthetic Teaching for Reinforcement Learning

Authors: Lars Johannsmeier, Yashraj Narang

Organizations: NVIDIA

Abstract

Real-world reinforcement learning (RL) systems still struggle with the demands of contact-rich industrial manipulation, including micrometer-level precision, success rates above 99%, and human-level cycle times. Although off-policy algorithms can improve performance by leveraging demonstrations and interventions, a key bottleneck is the lack of an intuitive interface for collecting such guidance while complying with constraints of the physical system and the policy. We propose PAKT, a framework for kinesthetic teaching in RL. As opposed to teleoperation approaches, PAKT relies on kinesthetic guidance, which is widely used in industry. However, a critical weakness of kinesthetic guidance is the possibility for the operator to move the robot along trajectories (e.g., velocities, accelerations, jerk) that the robot and/or policy cannot physically reproduce. Using PAKT, operators guide the robot through admittance control, which maps human-applied forces to motion. The downstream reference generator applies the same kinematic limits used during policy execution, keeping the collected trajectories within these limits. To support this teaching interface with an appropriate execution layer, PAKT adds a high-performance control stack that maps low-frequency RL actions to high-frequency torque commands. It consists of a reference generator and subsequent impedance controller, where the reference generator preserves the tracking performance of the impedance controller while improving contact handling and producing smoother policy actions. Across the reported runs on four insertion and industrial assembly benchmarks, including a data center compute tray, the end-to-end system reduces cycle time by 23%-48% and cumulative intervention count by 62%-86% relative to the HIL-SERL baseline. Project website: https://pakt-website.github.io/pakt-website}{https://pakt-website.github.io/pakt-website

Figures & tables

Figure 1: Overview of PAKT. In policy mode, policy actions are sent directly to the reference generator, whereas during guiding mode, human-applied forces first pass through an admittance controller. The reference generator outputs a filtered, limit-compliant command for the impedance controller, which computes the final torque command. The policy runs at $10$ Hz and the downstream controller stack at $1000$ Hz.

ID	Equation	Comment
C1	Eq. 10	Ours
C2	$\boldsymbol{f}_{\text{imp}}=-\mathbf{D}_{x}(\Lambda,\mathbf{K}_{x},\mathbf{\zeta})\dot{\mathbf{x}}+\mathbf{K}_{x}\tilde{\mathbf{x}}$	No inertia compensation or desired velocity
C3	$\boldsymbol{f}_{\text{imp}}=-\mathbf{D}_{x}\dot{\mathbf{x}}+\mathbf{K}_{x}\tilde{\mathbf{x}}$	No inertia compensation or desired velocity, and no damping design
C4	$\boldsymbol{f}_{\text{imp}}=-\mathbf{D}_{x}\dot{\mathbf{x}}+\mathbf{K}_{x}\tilde{\mathbf{x}}+\int k_{i}\tilde{\mathbf{x}}dt$	No inertia compensation or desired velocity, and no damping design, but with integrator
C5	$\boldsymbol{f}_{\text{imp}}=-\mathbf{D}_{x}\dot{\mathbf{x}}+\mathbf{K}_{x}\mathrm{clip}(\tilde{\mathbf{x}},\mathbf{e}_{\mathrm{lim}})$	No inertia compensation or desired velocity, and no damping design, but with error clipping

Table 1: Variations of controllers used in RL. For C5,

\mathbf{e}_{\mathrm{lim}}

denotes the clipped error. C5 is the controller used in our baseline [ 22 ]

Figure 2: Tracking error for the tested control stacks in the x, y, and z dimensions. Tracking errors are low-pass filtered at $7$ Hz for visibility. We truncate C5 to maintain visibility for the other time series, since error clipping makes C5 lag behind the reference trajectory and therefore produce much larger absolute tracking errors than the other stacks. C1 and C2 overlap in the tracking error plots, as do C3 and C4.

Controller	Average tracking error [mm]	Contact force [N]
C1 (PAKT)	$\mathbf{\left[16.8,14.1,9.6\right]}$	$\mathbf{22.5}$
C2	$\mathbf{\left[16.6,14.1,9.6\right]}$	$33.9$
C3	$\left[24.8,22.5,16.2\right]$	$46.5$
C4	$\left[24.8,22.4,16.0\right]$	$45.0$
C5 (HIL-SERL)	$\left[100.5,37.2,16.8\right]$	$28.2$

Table 2: Average tracking error and contact force for the tested controller stacks.

Figure 3: The four insertion tasks used in the RL experiments with outlines indicating the insertion location: (a) FMB peg insertion, (b) RAM insertion, (c) limit fixture assembly, and (d) busbar assembly. We show a more detailed overview in Appendix A and in the supplementary video.

Figure 4: Reward, cycle time, and intervention rate over training for the baseline, hybrid, and PAKT. RAM curves show the mean and standard deviation across five random seeds. The other tasks are single runs. Top row: The hybrid and PAKT reach higher observed success rates than the baseline. Middle row: Their similar cycle times isolate the primary effect of the controller stack. Bottom row: The lower intervention rate of PAKT relative to the hybrid isolates the effect of the kinesthetic interface.

Task	Initial demonstration time [s]			Success rate			Average cycle time [s]			Cumulative interventions
	B	H	P	B	H	P	B	H	P	B	H	P
Peg	$216$	$199$	$\mathbf{183}$	$\mathbf{1.0}$	$\mathbf{1.0}$	$\mathbf{1.0}$	$3.88$	$\mathbf{2.25}$	$\mathbf{2.17}$	$6276$	$2690$	$\mathbf{980}$
RAM insertion	$243$	$219$	$\mathbf{182}$	$0.88$	$\mathbf{0.99}$	$\mathbf{0.98}$	$5.2$	$\mathbf{4.07}$	$\mathbf{4.0}$	$8780$	$5205$	$\mathbf{3325}$
Limit fixture	$209$	$194$	$\mathbf{177}$	$\mathbf{0.99}$	$\mathbf{0.98}$	$\mathbf{0.99}$	$3.01$	$\mathbf{1.87}$	$\mathbf{1.92}$	$5937$	$2026$	$\mathbf{1342}$
Busbar	$221$	$187$	$\mathbf{146}$	$0.85$	$\mathbf{0.98}$	$\mathbf{0.99}$	$3.91$	$\mathbf{2.09}$	$\mathbf{2.03}$	$9286$	$2920$	$\mathbf{1327}$

Table 3: Results on the four tasks for the HIL-SERL baseline (B), hybrid (H), and PAKT (P). Cumulative intervention count is the discrete area under the intervention-rate curve over training steps. Average cycle time is computed over successful evaluation trials only. Failed and timed-out trials are excluded. Every entry in the RAM row is the mean across five seeds. The other rows are single runs.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Experimental setup used for all real-world tasks.

Appendix

Figure 6: The four insertion tasks used in the RL experiments: (a) FMB peg insertion, (b) RAM insertion, (c) limit fixture assembly, and (d) busbar assembly.

Parameter	Value
Robot platform	Franka Emika Robot with Franka Hand
Control compute	NVIDIA Jetson Orin AGX with L4T and PREEMPT_RT kernel
Policy / classifier compute	Desktop PC with Ubuntu 22.04 and NVIDIA RTX 4090
Policy command rate	$10$ Hz
Cameras	Two Intel RealSense D405 wrist cameras
Observation space	Two RGB wrist-camera images, Cartesian pose (zyx Euler convention), Cartesian velocity

Appendix

Table 4: Shared hardware and runtime details used across all tasks.

Task	Episode length [steps]	Action scale (translation [m], rotation [rad])	$x,y$ reset range [m]	Yaw reset range [rad]
Peg insertion	$150$	$(0.01,0.02)$	$\pm 0.02$	$\pm 0.05$
RAM insertion	$200$	$(0.005,0.02)$	$\pm 0.02$	$\pm 0.05$
Limit fixture	$150$	$(0.01,0.02)$	$\pm 0.02$	$\pm 0.05$
Busbar	$150$	$(0.01,0.02)$	$\pm 0.02$	$\pm 0.05$

Appendix

Table 5: Task-specific configuration for the four RL tasks in the paper.

Parameter	Value
Learning algorithm	RLPD with SAC-style pixel agent
Batch size	$256$ total = $128$ demo + $128$ online RL
Critic-to-actor ratio	$2$
Discount factor $\gamma$	$0.97$
Maximum training steps	$1{,}000{,}000$ gradient steps
Replay buffer capacity	$200{,}000$ transitions

Appendix

Table 6: Shared policy and training details.

Parameter	Value
Policy hidden dimensions	$[256,256]$
Policy activation	tanh
Policy output distribution	Gaussian with $\tanh$ squashing
Standard-deviation parameterization	exp
Standard-deviation range	$[10^{-5},5]$
Policy layer normalization	Enabled

Appendix

Table 7: Actor, critic, and optimization details.

Parameter	Value
Encoder type	Pretrained frozen ResNet-10
Image pretraining	ImageNet
Pre-pooling	Enabled
Pooling method	Spatial learned embeddings
Number of spatial blocks	$8$
Bottleneck dimension	$256$

Appendix

Table 8: Vision encoder and data augmentation details.

Parameter	Value
Classifier inputs	Task-relevant wrist-camera images
Backbone	Pretrained frozen ResNet-10
Pooling	Spatial learned embeddings
Bottleneck dimension	$256$
Classification head	Two-layer MLP
Loss	Cross-entropy

Appendix

Table 9: Success-classifier architecture and optimization details.

Parameter	Baseline	Ours
Cartesian translational stiffness	$\left[3000,3000,3000\right]$ N/m	$\left[3000,3000,3000\right]$ N/m
Cartesian rotational stiffness	$\left[300,300,300\right]$ N m/rad	$\left[300,300,300\right]$ N m/rad
Cartesian translational damping	$\left[89,89,89\right]$ N s/m	N/A
Cartesian rotational damping	$\left[7,7,7\right]$ N m s/rad	N/A
Cartesian translational damping factors	N/A	$\left[0.7,0.7,0.7\right]$
Cartesian rotational damping factors	N/A	$\left[0.7,0.7,0.7\right]$

Appendix

Table 10: Baseline-versus-ours comparison of the controller parameterization.

Task	Trans. vel.	Trans. acc.	Rot. vel.	Rot. acc.	Trans. jerk	Rot. jerk
	$[\min,\max]$	$[\min,\max]$	$[\min,\max]$	$[\min,\max]$	$\max$	$\max$
	[m/s]	[m/s 2 ]	[rad/s]	[rad/s 2 ]	[m/s 3 ]	[rad/s 3 ]
Peg insertion	$[-0.09,0.10]$	$[-0.90,1.00]$	$[-0.18,0.20]$	$[-1.8,2.0]$	$10000.0$	$10000.0$
RAM insertion	$[-0.045,0.05]$	$[-0.45,0.50]$	$[-0.18,0.20]$	$[-1.8,2.0]$	$10000.0$	$10000.0$
Limit fixture	$[-0.09,0.10]$	$[-0.90,1.00]$	$[-0.18,0.20]$	$[-1.8,2.0]$	$10000.0$	$10000.0$
Busbar	$[-0.09,0.10]$	$[-0.90,1.00]$	$[-0.18,0.20]$	$[-1.8,2.0]$	$10000.0$	$10000.0$

Appendix

Table 11: Task-specific reference-generator limits of our stack derived from the action scales in Tab. 5 . The

[\min,\max]

entries are signed directional bounds.

Explore similar work

Jul 10, 2026cs.RO

PAC-ACT: Post-training Actor-Critic for Action Chunking Transformers

Precision industrial contact manipulation requires reliable robot policies under pose perturbations and contact-force constraints. Vision-language-action models offer broad generalization but often introduce high inference latency and GPU-memory cost, while vision-action chunking policies are more suitable for real-time industrial control. However, these policies are usually trained by behavior cloning and suffer from distribution shift in contact-rich tasks. This paper proposes PAC-ACT, a reinforcement-learning post-training framework for pretrained Action Chunking Transformer policies. PAC-ACT reformulates policy optimization at the chunk level, constructs an ACT-transferred actor-critic architecture, and introduces a hybrid behavior-prior constraint to preserve the pretrained action distribution during online fine-tuning. Experiments on industrial precision-contact benchmarks show that PAC-ACT improves task success, contact stability, and force safety while retaining low latency and low GPU-memory usage. On the Contour task, PAC-ACT significantly reduces peak contact force and decreases the proportion of force readings above 60 N by 46 times. Sparse-reward ablations further show that the proposed behavior-prior constraint enables effective exploration under randomized initial poses.

Yujie Pang, Zudong Li

Action Chunks Soft Actor-Critic Robust Manipulation Reinforcement Learning Post-Training Transformer Encoder

Ask PDF

Jun 2, 2026cs.RO

Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation

Human-in-the-loop reinforcement learning (HIL-RL) improves sample efficiency in real-robot manipulation through online human intervention. However, successful trajectories may include suboptimal actions that deviate from the desired task-execution path and force human intervention. Existing HIL-RL methods typically apply the consistent credit assignment principle to all transitions, uniformly propagating discounted terminal rewards through suboptimal segments, ignoring the actual contribution of each transition to task success. This overestimates Q-values for critic learning and indirectly misguides actor updates toward suboptimal behavior patterns. To this end, we propose PACT, a Preference-calibrated Actor-Critic Training framework that leverages the implicit preference signals induced by intervention to perform credit reassignment on identified suboptimal segments while directly guiding policy training for unbiased critic-actor learning. Specifically, we first design a progress model that learns from human demonstration and identifies suboptimal segments for credit correction. Then, from the human action and resampled policy action at the intervention state, we build preference pairs to define a counterfactual advantage that penalizes Bellman targets of the identified suboptimal segment, enabling directional credit calibration. Moreover, we directly align the policy with human corrective actions in the bounded mean space, providing an additional signal beyond critic-guided updates. Across five real-robot manipulation tasks, PACT improves the average success rate by 24.5% and achieves 1.3 times faster convergence, thereby improving both RL sample efficiency and performance. Code is available at https://anonymous.4open.science/r/HILRL-A1X-BC05.

Zeyi Liu, Guangyao Liu, Yinuo Qu +6

Reinforcement Learning From Human Feedback Human-In-The-Loop Credit Assignment Optimal Actions

Ask PDF

Jun 7, 2026cs.RO

Real-IKEA: Physical Fidelity is the Prerequisite for Robust Manipulation

Robotic manipulation robustness often founders on the physics gap between simplified simulations and the resistance-laden real world. In this work, we emphasize that physical realism in articulated interaction is an important ingredient for robust policy learning. We present Real-IKEA, a dataset and simulation framework designed with physical accuracy as a first-class goal. Real-IKEA provides 1,079 articulated asset configurations, derived from 83 authentic IKEA handles and knobs processed through a meticulous six-step physical workflow. For contact-geometry accuracy, we introduce a bidirectional surface-deviation metric to quantify collision meshes. For dynamics realism, we establish resistance-calibrated configurations that vary damping and friction. Crucially, we demonstrate through a Reinforcement Learning (RL) policy that high-fidelity assets enable the discovery of robust "hooking" and "levering" strategies that prioritize mechanical advantage over fragile friction-pulling. Together, these results position Real-IKEA as a critical benchmark for developing manipulation policies capable of human-level robustness in articulated object tasks.

Kunqi Xu, Zhenhao Huang, Siyuan Luo +2

Generalizable Robotic Manipulations Robust Manipulation Manipulation Articulated Objects Character Fidelity

Ask PDF