AFFORD2ACT: How Far Can Sparse Keypoints Go for Robot Manipulation?

Anukriti Singh; Kasra Torshizi; Khuzema Habib; Kelin Yu; Ruohan Gao; Pratap Tokekar

cs.ROOct 1, 2025

AFFORD2ACT: How Far Can Sparse Keypoints Go for Robot Manipulation?

Authors: Anukriti Singh, Kasra Torshizi, Khuzema Habib, Kelin Yu, Ruohan Gao, Pratap Tokekar

Organizations: Department of Computer Science, University of Maryland, College Park, MD, USA.

Abstract

Vision-based manipulation policies are commonly learned from rich visual observations, typically using dense visual features extracted from full resolution images. It remains unclear how much of this visual information is really needed for effective policy learning. In this paper, we investigate the opposite end of the spectrum: how effective can manipulation policies be if they use as input only a compact visual representation. Sparse 2D keypoints provide a deliberately compact representation, giving the policy spatial information while removing direct access to appearance, texture, and background. We design AFFORD2ACT (Affordance-Guided Keypoints to Act) to study how much manipulation capability can be retained under this representation constraint. An off-the-shelf language-conditioned affordance model identifies the task-relevant region, from which up to 15 object keypoints are selected, together with 4 end-effector keypoints. The policy input is therefore reduced to only 19 2D keypoints (38 coordinates). A lightweight gating further decides which keypoints to pay attention to during execution. Across six real-robot manipulation tasks, this compact state outperforms the evaluated RGB, RGB-D, and dense-affordance behavior cloning baselines, degrades less as the number of demonstrations is reduced, and achieves 82% average success in held-out settings. Replacing learned gating with uniform weights reduces success from 92% to 52%. At the same time, the representation depends on reliable affordance localization and keypoint tracking, and remains vulnerable to incorrect masks, occlusion, tracking drift, and distractors.

Figures & tables

Figure 1: Overview. (a) An RGB observation is reduced, using (b) off-the-shelf affordance, correspondence, and tracking models, to (c) a compact state of 19 keypoints: up to 15 on the object’s actionable region and 4 on the robot end-effector. (d) The policy encodes the keypoints with a transformer, weights each keypoint with a learned gate (weights shown schematically), and predicts end-effector motion and the gripper command. (e) The same policy on held-out instances.

Figure 2: Experimental setup and policy network. (a) UR3e arm with a side-mounted RealSense D435i camera, and objects used in the held-out (left column) and seen (right column) scenarios. (b) Each tracked keypoint is embedded by $\psi_{\theta}$ and contextualized by a transformer encoder $\mathcal{T}_{\theta}$ ; a gating network $g_{\theta}$ produces softmax weights $\hat{w}_{t}$ that aggregate the tokens into $h_{t}$ , from which an MLP head predicts the end-effector delta action and the gripper command.

Panel of Figure 3 — Figure 3: Sparse keypoints versus dense visual inputs. (a) Total average success rate across input modalities; for each modality we run 20 trials per task and average across tasks, reporting seen and held-out performance separately. (b) Success as the number of demonstrations is reduced; standard deviation bars show variability across tasks.

Subgoal (%)	Afford2Act	UAD-2D	RGB	RGB-D
Aligned with object	94	80	52	78
Grasped object	88	75	24	42
Completed final motion	86	65	32	28
Robot collided	2	70	20	12

Table 1: Semantic subgoal analysis on seen instances. Percentage of annotated rollouts that reach each intermediate stage of the task, rather than only final task success.

Figure 6: Rollouts on held-out instances. Rows show Pour (top), Cut (middle), and Stir (bottom), with each sequence progressing left-to-right from approach $\rightarrow$ interaction $\rightarrow$ completion. Green dots denote affordance-selected keypoints, and translucent ring size indicates gating weight over time. Keypoints remain anchored to task-relevant parts, while the gating weight shifts with phase (e.g., cut: blade tip $\rightarrow$ handle; pour: edge/spout $\rightarrow$ main handle; stir: handle $\rightarrow$ bowl).

Panel of Figure 7 — Figure 7: Training scenes versus held-out test scenes. Each column pairs the scene used for training (top) with a held-out test scene (bottom) for one generalization setting, with the task and its success rate under that setting. Test scenes introduce an object category not seen in training (cut with a different blade), a novel object instance (teapot instead of mug), a new background (green tabletop), and distractors (clutter and a person moving in the workspace). The average success rate on seen objects is $92\%$ .

Figure 8: Qualitative generalization on two tasks. Rows show rollouts for Pour (top) and Kick (bottom). Left: seen training objects. Right: held-out settings with new backgrounds and, for Pour , novel object instances. Each panel shows three key frames (setup, approach, completion). No test-time adaptation is used.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Affordance prompt	Training demonstrations
Grasp/Lift	hold	40
Cut	cut_with	40
Brush/Sweep	hold , sweep	40
Pour	pour	40
Stir	hold , stir	40
Kick	kick	40

Appendix

Table 2: Tasks. Each task is trained as an independent policy from 40 teleoperated demonstrations.

Figure 9: Failure from weak affordance masking. On the prompt “pick up and stir” , incorrect keypoints lead the gripper to collide with the spatula.

Failure Type	What Happens	Likely Cause	Potential Mitigation
Affordance mask error	Keypoints are initialized on the wrong region or miss the true contact region.	The affordance model predicts an incomplete or incorrect actionable region.	Confidence-aware keypoint filtering, mask-noise augmentation, or online re-anchoring.
Tracking drift	Keypoints move away from the intended object part during execution.	Occlusion, contact-induced motion, or viewpoint ambiguity from the single external camera.	Periodic re-detection, confidence-based tracking checks, or multi-view sensing.
Correspondence error	A keypoint index corresponds to an inconsistent functional part across instances.	DINO similarity can be ambiguous when visually similar regions exist.	Stronger ROI filtering, multiple candidate matches, or descriptor confidence thresholds.
Execution/contact failure	The robot reaches the relevant region but fails the final interaction.	Grasp slippage, insufficient contact, or small control errors.	Improved gripper/contact control or additional demonstrations near failure cases.
Distractor interference	The policy attends to or tracks visually irrelevant objects or motion.	Distractors affect affordance localization or point tracking.	Distractor augmentation, confidence filtering, or re-anchoring during execution.

Appendix

Table 3: Failure taxonomy observed during real-world evaluation.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,671 papers

ComManip: Overfitting Manipulation Policies to Comfortable Regions
Sep 29, 2026Yidan Lai, Xinyi Chen, Shuquan Man +1Visuomotor Policy Learning Robotic Manipulation
South China University of Technology, Guangzhou, China · Nanyang Technological University, Singapore
AffordGen: Generating Diverse Demonstrations for Generalizable Object Manipulation with Afford Correspondence
Apr 12, 2026Jiawei Zhang, Kaizhe Hu, Yingqian Huang +3Affordance-Based Robotic Manipulation Robotic Manipulation
Shanghai Qi Zhi Institute · Tsinghua University · Fudan University +1
KeyGen: Unsupervised Keypoint based Object-Centric Representations for Category-Level Policy Generalization
Sep 23, 2026Shuxin Cao, Liquan Wang, Masoud Moghani +2Point Cloud Learning Generalization in Robotic Manipulation
Georgia Institute of Technology · University of Toronto

Sep 29, 2026J/K move · Enter open · S save

ComManip: Overfitting Manipulation Policies to Comfortable Regions

Yidan Lai, Xinyi Chen, Shuquan Man, Huiping Zhuang

South China University of Technology, Guangzhou, China · Nanyang Technological University, Singapore

Visuomotor Policy Learning Robotic Manipulation Data-Efficient Robot Learning Robot Policy Generalization

Open paper AskPDF arXiv

Training robot manipulation policies relies on costly robot demonstrations, making large-scale data collection impractical. Meanwhile, to improve policy generalization, existing approaches seek greater diversity in visual observations by varying object placements, viewpoints, and robot configurations during data collection. However, under a limited demonstration budget, this strategy forces the policy to model diverse visual observations, providing insufficient supervision to learn reliable observation-action correspondences under similar local conditions. Our study reveals that policies trained under this strategy achieve lower task success rates than those trained within a compact, visually and kinematically stable region. We refer to these stable regions as comfortable manipulation regions. To exploit this finding, we propose ComManip, a learning paradigm that specializes manipulation policies to comfortable manipulation regions. During inference, ComManip repositions the mobile base until the detected target center enters the familiar image-space range estimated from comfortable-region demonstrations. It then executes the same manipulation policy, enabling effective manipulation across diverse target locations. We conduct extensive experiments across multiple manipulation tasks, demonstration budgets, and policy families including ACT,

π_{0.5}

, RDT, OpenVLA-OFT, and SmolVLA. The results demonstrate that ComManip improves task success by roughly 20 percentage points or more across different policy architectures in large workspaces under limited demonstration budgets, suggesting that specializing manipulation policies to comfortable regions provides a more data-efficient learning paradigm for manipulation.

Abstract

Robotic Manipulation

Explore similar work

ComManip: Overfitting Manipulation Policies to Comfortable Regions

AffordGen: Generating Diverse Demonstrations for Generalizable Object Manipulation with Afford Correspondence

KeyGen: Unsupervised Keypoint based Object-Centric Representations for Category-Level Policy Generalization