Grasp-TAGS: Task-Aware Grasp Selection via Critic Distillation for Prehensile Loco-Manipulation

Xun Tu; Minghao Zou; Karthik Desingh

cs.RONov 6, 2026

Grasp-TAGS: Task-Aware Grasp Selection via Critic Distillation for Prehensile Loco-Manipulation

Authors: Xun Tu, Minghao Zou, Karthik Desingh

Organizations: Department of Computer Science & Engineering at the University of Minnesota, Twin Cities

Abstract

Loco-manipulation, which requires mobile manipulators to coordinate locomotion and manipulation over long horizons, is a key capability for large-object rearrangement in applications such as service robotics and warehouse automation. Despite recent progress, reliably executing these tasks remains challenging because performance depends on complex robot-object and object-environment interaction dynamics. Existing approaches address this challenge either through model-based methods, which often require accurate dynamics models, or through learning-based methods, which reduce this requirement but typically assume a fixed or predefined initial grasp. However, different geometrically stable grasps can lead to substantially different downstream manipulation performance. Therefore, we propose Grasp-TAGS, a task-aware grasp selection framework for goal-conditioned large-object rearrangement. We train a unified PPO-based reinforcement learning policy across multiple initial grasp configurations and distill the learned critic into a lightweight grasp utility network that ranks candidate grasps using only the candidate grasp configuration and goal information. At inference time, we select the grasp with the highest predicted downstream utility for the downstream policy to complete the task, before even making a contact. Experiments demonstrate that Grasp-TAGS improves rearrangement success over both fixed single-grasp policies and random grasp selection, achieving an average success rate of 89.84% compared with 79.87% for the best-performing single-grasp policies, an improvement of 9.97 percentage points. Grasp-TAGS also reduces final orientation errors across all evaluated objects. For more qualitative results, refer to the supplementary webpage: https://rpm-lab-umn.github.io/Grasp-TAGS-web-page/

Figures & tables

Fig. 1 : Qualitative example of grasp-dependent rearrangement behavior. For the illustrated target pose, the backrest grasp ( $g_{1}$ ) yields better final alignment with the target pose than the two armrest grasps ( $g_{2}$ and $g_{3}$ ).

Fig. 2 : Overview of the proposed pipeline Grasp-TAGS. (1) A hierarchical policy trained via PPO performs loco-manipulation using a pretrained low-level controller. (2) After PPO training, the learned critic is distilled into a grasp utility network. (3) At inference time, the distilled network selects the grasp $g^{*}$ with the highest predicted downstream utility before policy execution.

Reward Terms	Formula	Weight	Description
Task completion	$\mathbb{1}[\\|p_{o}-p_{o}^{*}\}<\varepsilon_{p}$	10.0	Positive reward when the object reaches the target pose
Keypoint pose matching	$\exp\left(-\frac{\\|\mathbf{K}_{goal}-\mathbf{K}_{obj}\\|_{2}}{\sigma^{2}}\right)$	8.0	Reduce the distance between the current and target object keypoints
Goal-directed velocity	$\exp\left(\frac{\hat{\mathbf{v}}_{o}\cdot\hat{\mathbf{d}}_{goal}}{\sigma^{2}}-1\right)$	1.0	Encourage alignment between the object velocity and the direction toward the goal
Alive	$\mathbb{1}[episode\ is\ alive]$	5.0	Sparse reward for maintaining a valid and active episode
Linear velocity smoothness	$\\|v_{o}\\|^{2}$	-2.0	Penalty on changes in the object’s linear velocity
Angular velocity smoothness	$\\|\omega_{o}\\|^{2}$	-2.0	Penalty on changes in the object’s angular velocity

TABLE I : Reward terms to train the high-level controller

Obs. Terms	Dim	Description
$q_{1:128}$	18	Position of all the joints (both arm & legs)
$\dot{q}_{1:128}$	18	Velocity of all the joints
$s^{b}_{t}$	2	Orientation of the base (no yaw)
$\dot{s}^{b}_{t}$	3	Angular velocity of the base
$a^{high}_{t-1}$	9	Last high-level action
$p^{b}_{ee,t}$	7	SE(3) pose of the end-effector in base frame

TABLE II : Observation terms to the high-level controller. It consists of proprioceptive data from the robot and the real-time observations of the object.

Fig. 3 : Objects and candidate grasp poses used in the experiments. The dataset includes three chairs, two carts, and three buckets, with predefined candidate grasps highlighted in red. Candidate grasps used for our experiments are shown in red circle.

	Chair-1			Chair2
	$PPO_{i}$	$PPO_{g}$ + Random	Grasp-TAGS	$PPO_{i}$	$PPO_{g}$ + Random	Grasp-TAGS
Success rate [ $\%$ ] ( $\uparrow$ )	70.09/75.46/78.20	75.88	93.65	63.79/71.51/70.00	60.28	84.72
Planar Positional Error [m] ( $\downarrow$ )	0.0292/0.0250/0.0366	0.0281	0.0162	0.034/ 0.0161 /0.0142	0.045	0.0239
Angular Error [rad] ( $\downarrow$ )	0.3571/0.2850/0.2802	0.2407	0.0712	0.5268/0.3421/0.3732	0.4382	0.1308
Keypoint Angular Error [deg] ( $\downarrow$ )	20.4597/16.3271/16.0567	13.7906	4.0793	30.1858/19.5889/21.3821	25.1091	7.4923
	Chair-3			Cart-1

TABLE III : Quantitative results on simulated large-object rearrangement. We compare

PPO_{i}

PPO_{g}+

Random, and Grasp-TAGS using Success rate, Planar Positional Error, Angular Error and Keypoint Angular Error across multiple objects (

\downarrow

: smaller numbers indicate better results;

\uparrow

: larger numbers indicate better results).

Fig. 4 : Success rates for the best

PPO_{i}

among the policies trained individually on each grasp pose,

PPO_{g}+

random grasp selection, and Grasp-TAGS. Grasp-TAGS achieves the highest success rate (

\uparrow

) across all evaluated objects in all 4096 parallel environments.

Fig. 5 : Illustrative example of grasp utility predictions for a target object pose (green mesh) . The colors visualize the relative utility estimated by Grasp-TAGS, where green denotes higher predicted downstream utility and red denotes lower predicted utility.

Robotic Loco-Manipulation

See all 41 papers

Jun 29, 2026

Grasp-Oriented Non-Prehensile Manipulation via Learning a Graspability Field

Non-prehensile manipulation is often used as a preparatory step for robotic grasping, yet existing approaches typically require a predefined target object pose. In practice, however, objects admit multiple graspable configurations and the desired pose is not known in advance. We reformulate non-prehensile manipulation for grasping as optimizing an object centric graspability objective rather than reaching a specific pose. We construct a graspable set from synthesized grasps and define a graspability field that measures how suitable an object configuration is for successful grasp execution. The scalar measure provides a dense learning signal for reinforcement learning and determines when to terminate manipulation. This yields a closed-loop manipulation-to-grasp pipeline driven by a single policy. Experiments in simulation and on a real robot show that the policy reliably reconfigures objects into graspable states and transitions to grasping without external planners or manually specified stopping conditions. The predicted graspability distance correlates with real world grasp success, which indicates that the learned representation captures grasp feasibility of object configurations.

Licheng Zhong, Gim Hee Lee

Department of Computer Science, National University of Singapore, Singapore

RL for Robotics Robotic Grasping Non-Prehensile Manipulation Non-Prehensile Robotic Manipulation

Ask PDF

Apr 5, 2026

Learning Dexterous Grasping from Sparse Taxonomy Guidance

Dexterous manipulation requires planning a grasp configuration suited to the object and task, which is then executed through coordinated multi-finger control. However, specifying grasp plans with dense pose or contact targets for every object and task is impractical. Meanwhile, end-to-end reinforcement learning from task rewards alone lacks controllability, making it difficult for users to intervene when failures occur. To this end, we present GRIT, a two-stage framework that learns dexterous control from sparse taxonomy guidance. GRIT first predicts a taxonomy-based grasp specification from the scene and task context. Conditioned on this sparse command, a policy generates continuous finger motions that accomplish the task while preserving the intended grasp structure. Our result shows that certain grasp taxonomies are more effective for specific object geometries. By leveraging this relationship, GRIT improves generalization to novel objects over baselines and achieves an overall success rate of 87.9%. Moreover, real-world experiments demonstrate controllability, enabling grasp strategies to be adjusted through high-level taxonomy selection based on object geometry and task intent.

Juhan Park, Taerim Yoon, Seungmin Kim +10

Department of Artificial Intelligence, Korea University, Seoul, Korea · Korea University of Technology and Education, Cheonan, Korea · Naver AI Lab, Korea +4

RL for Robotics Robotic Grasping Dexterous Manipulation

Ask PDF

Sep 21, 2026

GraspTune: Tactile-Driven Execution Refinement for Robust Grasping

Visual grasp proposal generation has advanced rapidly, yet converting a selected proposal into a stable physical grasp remains a central execution-stage challenge. This paper introduces GraspTune, a tactile-driven execution-stage refinement framework that starts from a nominal proposal and applies bounded residual TCP motions during approach, contact formation, and final grasp execution. GraspTune learns control-facing contact semantics from local depth, tactile signals, state, and history using state-conditioned expert contact queries and multi-task supervision for contact change, contact risk, and post-close readiness. The representation conditions a diffusion-pretrained residual policy and is aligned with PPO for closed-loop execution. Across more than 60,000 simulated executions over 20 object categories, GraspTune establishes an execution-layer benefit across four proposal generators, raising stable grasp success by +19.22, +9.55, +12.45, and +20.70 percentage points for GraspNet, Contact-GraspNet, AnyGrasp, and VGN. A four-fold held-out category study raises unseen-object execution from 54.58% to 70.33%, showing category-disjoint generalization of contact correction. Across more than 1,000 real-robot trials on a UR5e setup with Xense fingertip sensors, GraspTune raises GraspNet execution from 71.0% to 84.3%, validating direct transfer without realworld policy fine-tuning. Together, these results turn visually plausible proposals into stable physical grasps for downstream contact-rich manipulation. A supplementary video is available at https://youtu.be/kcq7fSLNtzU.

Juntao Li, Xingke Xia, Sichao Liu +1

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · KTH Royal Institute of Technology, Stockholm, Sweden

Robotic Grasping Generalization in Robotic Manipulation Tactile Sensing for Robotic Manipulation

Ask PDF

Abstract

Robotic Loco-Manipulation

Explore similar work

Grasp-Oriented Non-Prehensile Manipulation via Learning a Graspability Field

Learning Dexterous Grasping from Sparse Taxonomy Guidance

GraspTune: Tactile-Driven Execution Refinement for Robust Grasping