Grasp-TAGS: Task-Aware Grasp Selection via Critic Distillation for Prehensile Loco-Manipulation

Xun Tu; Minghao Zou; Karthik Desingh

cs.RONov 6, 2026

Grasp-TAGS: Task-Aware Grasp Selection via Critic Distillation for Prehensile Loco-Manipulation

Authors: Xun Tu, Minghao Zou, Karthik Desingh

Organizations: Department of Computer Science & Engineering at the University of Minnesota, Twin Cities

Abstract

Loco-manipulation, which requires mobile manipulators to coordinate locomotion and manipulation over long horizons, is a key capability for large-object rearrangement in applications such as service robotics and warehouse automation. Despite recent progress, reliably executing these tasks remains challenging because performance depends on complex robot-object and object-environment interaction dynamics. Existing approaches address this challenge either through model-based methods, which often require accurate dynamics models, or through learning-based methods, which reduce this requirement but typically assume a fixed or predefined initial grasp. However, different geometrically stable grasps can lead to substantially different downstream manipulation performance. Therefore, we propose Grasp-TAGS, a task-aware grasp selection framework for goal-conditioned large-object rearrangement. We train a unified PPO-based reinforcement learning policy across multiple initial grasp configurations and distill the learned critic into a lightweight grasp utility network that ranks candidate grasps using only the candidate grasp configuration and goal information. At inference time, we select the grasp with the highest predicted downstream utility for the downstream policy to complete the task, before even making a contact. Experiments demonstrate that Grasp-TAGS improves rearrangement success over both fixed single-grasp policies and random grasp selection, achieving an average success rate of 89.84% compared with 79.87% for the best-performing single-grasp policies, an improvement of 9.97 percentage points. Grasp-TAGS also reduces final orientation errors across all evaluated objects. For more qualitative results, refer to the supplementary webpage: https://rpm-lab-umn.github.io/Grasp-TAGS-web-page/

Figures & tables

Fig. 1 : Qualitative example of grasp-dependent rearrangement behavior. For the illustrated target pose, the backrest grasp ( $g_{1}$ ) yields better final alignment with the target pose than the two armrest grasps ( $g_{2}$ and $g_{3}$ ).

Fig. 2 : Overview of the proposed pipeline Grasp-TAGS. (1) A hierarchical policy trained via PPO performs loco-manipulation using a pretrained low-level controller. (2) After PPO training, the learned critic is distilled into a grasp utility network. (3) At inference time, the distilled network selects the grasp $g^{*}$ with the highest predicted downstream utility before policy execution.

Reward Terms	Formula	Weight	Description
Task completion	$\mathbb{1}[\\|p_{o}-p_{o}^{*}\}<\varepsilon_{p}$	10.0	Positive reward when the object reaches the target pose
Keypoint pose matching	$\exp\left(-\frac{\\|\mathbf{K}_{goal}-\mathbf{K}_{obj}\\|_{2}}{\sigma^{2}}\right)$	8.0	Reduce the distance between the current and target object keypoints
Goal-directed velocity	$\exp\left(\frac{\hat{\mathbf{v}}_{o}\cdot\hat{\mathbf{d}}_{goal}}{\sigma^{2}}-1\right)$	1.0	Encourage alignment between the object velocity and the direction toward the goal
Alive	$\mathbb{1}[episode\ is\ alive]$	5.0	Sparse reward for maintaining a valid and active episode
Linear velocity smoothness	$\\|v_{o}\\|^{2}$	-2.0	Penalty on changes in the object’s linear velocity
Angular velocity smoothness	$\\|\omega_{o}\\|^{2}$	-2.0	Penalty on changes in the object’s angular velocity

TABLE I : Reward terms to train the high-level controller

Obs. Terms	Dim	Description
$q_{1:128}$	18	Position of all the joints (both arm & legs)
$\dot{q}_{1:128}$	18	Velocity of all the joints
$s^{b}_{t}$	2	Orientation of the base (no yaw)
$\dot{s}^{b}_{t}$	3	Angular velocity of the base
$a^{high}_{t-1}$	9	Last high-level action
$p^{b}_{ee,t}$	7	SE(3) pose of the end-effector in base frame

TABLE II : Observation terms to the high-level controller. It consists of proprioceptive data from the robot and the real-time observations of the object.

Fig. 3 : Objects and candidate grasp poses used in the experiments. The dataset includes three chairs, two carts, and three buckets, with predefined candidate grasps highlighted in red. Candidate grasps used for our experiments are shown in red circle.

	Chair-1			Chair2
	$PPO_{i}$	$PPO_{g}$ + Random	Grasp-TAGS	$PPO_{i}$	$PPO_{g}$ + Random	Grasp-TAGS
Success rate [ $\%$ ] ( $\uparrow$ )	70.09/75.46/78.20	75.88	93.65	63.79/71.51/70.00	60.28	84.72
Planar Positional Error [m] ( $\downarrow$ )	0.0292/0.0250/0.0366	0.0281	0.0162	0.034/ 0.0161 /0.0142	0.045	0.0239
Angular Error [rad] ( $\downarrow$ )	0.3571/0.2850/0.2802	0.2407	0.0712	0.5268/0.3421/0.3732	0.4382	0.1308
Keypoint Angular Error [deg] ( $\downarrow$ )	20.4597/16.3271/16.0567	13.7906	4.0793	30.1858/19.5889/21.3821	25.1091	7.4923
	Chair-3			Cart-1

TABLE III : Quantitative results on simulated large-object rearrangement. We compare

PPO_{i}

PPO_{g}+

Random, and Grasp-TAGS using Success rate, Planar Positional Error, Angular Error and Keypoint Angular Error across multiple objects (

\downarrow

: smaller numbers indicate better results;

\uparrow

: larger numbers indicate better results).

Fig. 4 : Success rates for the best

PPO_{i}

among the policies trained individually on each grasp pose,

PPO_{g}+

random grasp selection, and Grasp-TAGS. Grasp-TAGS achieves the highest success rate (

\uparrow

) across all evaluated objects in all 4096 parallel environments.

Fig. 5 : Illustrative example of grasp utility predictions for a target object pose (green mesh) . The colors visualize the relative utility estimated by Grasp-TAGS, where green denotes higher predicted downstream utility and red denotes lower predicted utility.

Robotic Loco-Manipulation

See all 41 papers

Grasp-Oriented Non-Prehensile Manipulation via Learning a Graspability Field
Jun 29, 2026Licheng Zhong, Gim Hee LeeRL for Robotics Robotic Grasping
Department of Computer Science, National University of Singapore, Singapore
Learning Dexterous Grasping from Sparse Taxonomy Guidance
Apr 5, 2026Juhan Park, Taerim Yoon, Seungmin Kim +10RL for Robotics Robotic Grasping
Department of Artificial Intelligence, Korea University, Seoul, Korea · Korea University of Technology and Education, Cheonan, Korea · Naver AI Lab, Korea +4
GraspTune: Tactile-Driven Execution Refinement for Robust Grasping
Sep 21, 2026Juntao Li, Xingke Xia, Sichao Liu +1Robotic Grasping Generalization in Robotic Manipulation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · KTH Royal Institute of Technology, Stockholm, Sweden

Jun 29, 2026J/K move · Enter open · S save

Grasp-Oriented Non-Prehensile Manipulation via Learning a Graspability Field

Licheng Zhong, Gim Hee Lee

Department of Computer Science, National University of Singapore, Singapore

RL for Robotics Robotic Grasping Non-Prehensile Manipulation Non-Prehensile Robotic Manipulation

Open paper AskPDF arXiv

Non-prehensile manipulation is often used as a preparatory step for robotic grasping, yet existing approaches typically require a predefined target object pose. In practice, however, objects admit multiple graspable configurations and the desired pose is not known in advance. We reformulate non-prehensile manipulation for grasping as optimizing an object centric graspability objective rather than reaching a specific pose. We construct a graspable set from synthesized grasps and define a graspability field that measures how suitable an object configuration is for successful grasp execution. The scalar measure provides a dense learning signal for reinforcement learning and determines when to terminate manipulation. This yields a closed-loop manipulation-to-grasp pipeline driven by a single policy. Experiments in simulation and on a real robot show that the policy reliably reconfigures objects into graspable states and transitions to grasping without external planners or manually specified stopping conditions. The predicted graspability distance correlates with real world grasp success, which indicates that the learned representation captures grasp feasibility of object configurations.

Abstract

Robotic Loco-Manipulation

Explore similar work

Grasp-Oriented Non-Prehensile Manipulation via Learning a Graspability Field

Learning Dexterous Grasping from Sparse Taxonomy Guidance

GraspTune: Tactile-Driven Execution Refinement for Robust Grasping