Task-Space Imitation Guidance for Efficient Reinforcement Learning

Salar Asayesh; Hossein Darani; Todd Cao; Evgeny Andriash; Mani Ranjbar

cs.ROOct 5, 2026

Task-Space Imitation Guidance for Efficient Reinforcement Learning

Authors: Salar Asayesh, Hossein Darani, Todd Cao, Evgeny Andriash, Mani Ranjbar

Organizations: Sanctuary AI, Canada

Abstract

We introduce Task-Space Imitation Guidance for Efficient Reinforcement Learning (TIGER), a reward-construction and pretraining framework for sparse-reward tabletop robotic manipulation. TIGER treats an action-chunked imitation policy not as an executable controller or action prior, but as a local task-space progress estimator: predicted action chunks are converted, using controller-aware action-to-motion mapping, into short-horizon end-effector references, and the RL agent receives dense progress rewards toward these references while the sparse environment reward remains the dominant objective. During pretraining, TIGER uses imitation-guided look-ahead signals to relax conservative value penalties for actions predicted to make task-space progress, reducing off-manifold exploration during early online RL. Across simulation and real-robot experiments, TIGER improves early sample efficiency and reduces measured safety violations while matching or improving final success rates relative to prior RL and IL-RL baselines on the evaluated tasks.

Figures & tables

	Robomimic			MetaWorld			Humanoid
Method	Can	Square	ToolHang	Assembly	Box Close	Stick Pull	Shape Sorting
BC	–/–/0.7	–/–/0.6	–/–/0.5	–/–/0.7	–/–/0.38	–/–/0.25	–/–/0.1
LaNE	–/–/0.25	–/–/0.025	–/–/0.0	–/–/0.08	–/–/0.06	–/–/0.6	–/–/0.0
RLPD	55/–/0.71	120/–/0.79	–/–/0.0	40/50/0.95	35/45/0.94	–/–/0.18	–/–/0.0
IBRL	35/75/0.91	80/160/0.91	470/–/0.0	25/30/0.96	25/40/0.94	35/–/0.72	115/–/0.78
DSRL	5 /–/0.64	–/–/0.37	–/–/0.0	5 /–/0.73	–/–/0.3	–/–/0.12	–/–/0.0

Table 1: Simulation benchmark summary. Each cell reports

S_{50}\downarrow/S_{90}\downarrow/F\uparrow

, where

S_{50}

and

S_{90}

are environment steps, in thousands, required to reach 50% and 90% success, and

F

is final success. “–” denotes that the threshold was not reached.

Figure 1: Final episode length achieved by each method.

Task	ReWiND	Robometer	TIGER
Assembly	25/30/ .95	25/30/.95	25/30/.95
Box Close	30/60/.87	25/50/.92	25/45/.93
Stick Pull	50/60/.84	55/–/.60	40/50/.92
Square	130/–/.74	120/–/.71	90/160/.87

Table 2: Additional simulation comparisons. Entries report

S_{50}\!\downarrow/S_{90}\!\downarrow/F\!\uparrow

; environment steps are in thousands and results are averaged over five seeds. Panel (a) uses the same demonstrations and RLPD backend. Panel (b) reports task averages over all ten LIBERO-Spatial tasks; BC has no online-learning thresholds. A dash otherwise indicates that a threshold was not reached.

Panel of Figure 2 — Figure 2: Top: Visualization of each task. Bottom: Training curves for each task. The x-axis represents environment steps in intervals of 1000. For block picking and drawer opening, the y-axis indicates the average task success rate over each 1000-step interval. For rotary insertion, the y-axis denotes the curriculum stage. Task completion requires reaching stage #9.

Panel of Figure 3 — Figure 3: Safety violations on the two real-robot tasks (block picking and drawer opening). Bars show the mean number of episode resets across two seeds with min–max ranges.

	Stationary	Rotating
	Random Location	Location
BC	60%	10%
TIGER	100%	90%
IBRL	90%	77.5%
DSRL	60%	20%

Table 3: Rotary insertion evaluation.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Performance on MetaWorld benchmark

Panel of Figure 8 — Figure 8: Performance on Humanoid Shape Sorting task

Panel of Figure 9 — Figure 9: Multi-task LIBERO-Spatial learning curves.

Figure 10: Effect of imitation-guidance design.

Panel of Figure 11 — Figure 11: Effect of demonstration quality on learning performance

Figure 12: Effect of the reward-scaling factor

k

on success rate.

Figure 13: Pretraining objective ablation on Robomimic Can.

Figure 14: Transferability of the controller-aware mapping.

Figure 15: Sensitivity of controller-aware mapping accuracy to the number of training demonstrations. Validation MSE decreases rapidly and saturates after approximately five demonstrations, indicating that the 50-demonstration fits used in production are not data-limited.

Symbol	Description	Value
$\gamma$	Discount factor	0.99
$\sigma$	Exploration Std	0.1
$U$	Update frequency	2
$c$	Noise clip	0.3
$LR$	Adam learning rate	$1\text{\times}{10}^{-4}$
$N_{\text{batch}}$	Mini-batch size	256

Appendix

Table 4: Core algorithm hyper-parameters.

Parameter	Description	Value
$\lambda_{p}$	position distance scaling factor (eq. ( 2 ))	5.0
$\lambda_{q}$	rotation distance scaling factor (eq. ( 2 ))	10.0
$\alpha$	imitation guidance strength (eq. ( 9 ))	1.0
$\lambda_{\text{CQL}}$	CQL pretraining loss scale (eq. ( 7 ))	1.0

Appendix

Table 5: TIGER common hyper-parameters.

Parameter	Real Franka	MetaWorld	Robomimic	Humanoid
$\lambda_{\textit{TIGER}}$	1.0	2.0	3.0	1.0
$\lambda_{\textit{env}}$	10.0	10.0	30.0	20

Appendix

Table 6: TIGER environment-specific hyper-parameters.

Parameter	Description	Value
Backbone	Visual encoder backbone	ResNet-18
BatchNorm	Backbone normalization	FrozenBatchNorm2d
Hidden dim	Per-camera feature dimension	256
Fused dim	Conditioning feature dimension	128
$T_{a}$	Predicted action chunk length	20
Time dim	Flow-time embedding dimension	64

Appendix

Table 7: Flow-matching imitation-policy hyper-parameters.

Explore similar work

Temporal Self-Imitation Learning
Jun 18, 2026Yinsen Jia, Boyuan ChenRL for Robotics Long-Horizon Robotic Manipulation
Duke University
ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstration
Jun 8, 2026Yuying Zhang, Francesco Verdoja, Wenyan Yang +1Robot Skill Learning Robotic RL
School of Electrical Engineering Aalto University, Finland
SpeedTuning: Speeding Up Policy Execution with Lightweight Reinforcement Learning
Aug 10, 2026David D. Yuan, Tony Z. Zhao, Kaylee Burns +1Robot Policy Learning Robot Skill Learning
Stanford University

Jun 18, 2026, cs.ROJ/K move · Enter open · S save

Temporal Self-Imitation Learning

Yinsen Jia, Boyuan Chen

Duke University

RL for Robotics Long-Horizon Robotic Manipulation Imitation Learning

Open paper AskPDF arXiv

Long-horizon policies trained with reinforcement learning can still achieve high return through inefficient interactions, while rare efficient behaviors discovered during training may be forgotten. We argue that temporal efficiency itself provides a source of self-supervision for reinforcement learning. We introduce Temporal Self-Imitation Learning (TSIL), a reinforcement learning framework that mines temporally efficient successful trajectories generated during learning and converts them into reusable supervision for future policy improvement. TSIL progressively refines learning using configuration-conditioned adaptive temporal targets derived from fast successful trajectories, while preserving and replaying efficient behaviors through efficiency-weighted self-imitation learning. Across 30 long-horizon tasks spanning robot manipulation and interactive navigation, TSIL consistently improves task success rates, learning efficiency, behavioral efficiency, and robustness to unstable training conditions.

Abstract

Explore similar work

Temporal Self-Imitation Learning

ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstration

SpeedTuning: Speeding Up Policy Execution with Lightweight Reinforcement Learning