Robot manipulation has traditionally relied on end-effector-centric interactions. In contrast, humans routinely exploit whole-arm contacts to improve efficiency, robustness, and adaptability, especially when the hands are occupied or when precise grasping is unnecessary. Enabling similar capabilities in robots remains challenging: the robot must select where on its arm to make contact, generate effective non-prehensile interactions, and satisfy task-specific pose or safety constraints on the end-effector. We propose Whole-Arm Contact Learning (WACL), a hierarchical framework that describes each whole-arm contact task with three robot-agnostic representations and hands them to a quasi-static surface-contact optimizer: semantic keypoints that summarize the scene, a reference contact trajectory that specifies where and how the arm presses on the object, and safety-aware path constraints that a vision-language model generates from the task description to protect the end-effector's task. The robot's kinematic chain enters only in this optimizer, which grounds these representations onto the arm by jointly searching over which link makes contact, where on that link, and the joint configuration that realizes it. This decomposition lets WACL learn the contact trajectory from 10 bare-hand human demonstrations per task collected without teleoperation, while new end-effector constraints change only the optimizer's input rather than any trained component. Real-world experiments on a 7-DoF arm across four whole-arm tasks show that WACL sustains contact under end-effector constraints where a strong link-pose baseline fails (78% vs. 33% success) and generalizes to new constraints without retraining. Website: https://wacl-robot.github.io
Figures & tables
Figure 1: WACL overview. WACL produces three robot-agnostic representations and grounds them onto the robot in a single downstream optimizer. Contact trajectory from hand demonstrations. A segmentation model and 4D hand tracker fuse bare-hand multi-view RGB-D demonstrations into a reference contact trajectory of contact positions and surface normals. Keypoint-conditioned contact learning. Semantic keypoints distilled by KALM are matched to the scene, conditioning a trajectory model that generates the contact trajectory. Safety-aware path constraint generation. A VLM grounds a fixed vocabulary of primitives into path constraints that preserve the primary end-effector task. Surface-contact trajectory optimization. The optimizer searches each link’s reachable anchors with surface-contact inverse kinematics and beam search under the constraints.
Figure 2: Evaluation tasks. Each task pairs a secondary whole-arm contact action with a primary end-effector task that must hold throughout: cup upright, plate level, gripper above the sink, gripper tethered by a rope.
Figure 3: Contact links. Collision meshes of the xArm 7 forearm (link 4) , wrist yaw (link 5) , and wrist pitch (link 6) form the contact surface M .
Success
Variant
What changes
Task
Variant
Full WACL
+ second constraint
cup-upright AxisCone added
CloseFaucet
7/10
7/10
w/o sliding anchor
anchor fixed at first waypoint
OpenCoffee
3/10
7/10
w/o link search
link fixed to link 5
PressButton
6/10
9/10
link fixed to link 6
PressButton
7/10
9/10
geometric primitive
hand-specified arc as trajectory
OpenCoffee
4/10
7/10
Table 2: Generalization, ablations, and controls. Each variant runs 10 rollouts on one constrained task, on the same placements as the full model. A second end-effector constraint leaves success unchanged, while removing an optimizer component or replacing the learned trajectory lowers it.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Scene-decomposition prompt. First VLM call of Appendix A.4 . The model maps the task description to task-relevant entities, adding a bounding surface only when the task requires one, and a SAM3 prompt ensemble per entity, and names the contact target. Task-specific examples are omitted; braces mark variables.
Figure 5: Constraint-instantiation prompt. Second VLM call of Appendix A.4 . Given the task, the tool-center-point (TCP) state from forward kinematics, and the scene oriented bounding boxes, the model derives the constrained axis from the grasp geometry, grounds regions with a margin, and emits the minimal set of typed constraint primitives as a YAML list. Task-specific examples are omitted; braces mark variables.
Figure 6: Per-inference runtime breakdown. Wall-clock per stage for a representative inference with a single candidate plan; Appendix A.7 gives the numbers. Shading groups the rows by track: constraint generation (top three rows), the keypoint policy (next two), and the optimizer (bottom four), whose anchor preparation runs concurrently with the other two tracks. The optimizer’s remaining stages start at the dashed barrier once all three tracks finish, with the beam search and the free-space approach overlapped; the right marker is the total.
Figure 7: Real-robot setups. ( 7(a) ) the 7-DoF xArm 7 used in the main experiments; ( 7(b) ) the 7-DoF Franka FR3 with a Robotiq gripper used for the cross-embodiment transfer of Appendix B.6 .
Figure 8: Success rates with Wilson 95% intervals. The xArm results of Table 1 , 10 rollouts per task and 40 in aggregate. The aggregate constrained intervals are disjoint; the unconstrained ones overlap, consistent with the paired tests in the text; p is the exact McNemar test on the 40 placement pairs.
Whole-arm manipulation involves direct contact with the environment while the robot completes a task by distributing contact across multiple links as contacts form, slide, and break. This setting breaks common implicit assumptions in many learning-based manipulation pipelines: arm configuration tightly couples motion and contact forces, contact state is partially observed under occlusion, and purely learned rollouts can become physically inconsistent under distribution shift because many multi-link contact configurations are sparsely represented in the data. To address this, we propose TACTIC (Tactile and Vision Conditioned Contact-Centric Control), a receding-horizon controller for whole-arm manipulation. TACTIC uses a contact-centric hybrid predictive model that combines RGB-D, distributed tactile sensing, and a compact 2D proximity representation. The model couples a learned, action-conditioned latent dynamics model with analytical kinematics through contact Jacobians, enabling rollouts of future contact configurations and interaction forces. TACTIC integrates these rollouts into a sampling-based MPC planner with contact-aware action sampling: contact Jacobian-based projections steer sampled action sequences toward force-modulating directions, and objectives defined over predicted proximity and interaction forces trade task progress against whole-arm force regulation. We evaluate TACTIC in simulation against state-of-the-art model-based and model-free methods, and perform ablations that isolate the contribution of each design choice. TACTIC consistently outperforms other methods. We further demonstrate real-world performance on a robot with distributed tactile sensing across three whole-arm manipulation tasks that require multi-contact trajectories: turning over and repositioning a manikin, and goal-reaching in a 3D dynamic maze. Website: https://emprise.cs.cornell.edu/tactic
Rishabh Madan, Angchen Xie, Samantha Saak +7
Cornell University · Carnegie Mellon University · Toyota Research Institute
Humanoid robots have many degrees of freedom and contact surfaces, yet most loco-manipulation research restricts manipulation to the hands and avoids other body contacts with the environment. To extend manipulation beyond the hands, we present PanTact, a system that learns humanoid whole-body dexterity, the capability to manipulate with any suitable body surface, directly from robot-free human demonstrations. Our key idea is to use contact as manipulation-centric information that transfers across embodiments. Contact locations and timing specify the interactions needed to accomplish a task, while the robot chooses how to coordinate its body to realize them within its own physical constraints. Specifically, PanTact captures, learns, and executes whole-body contact via: 1) a wearable, robot-free data collection interface that augments egocentric vision and bimanual UMI grippers with whole-body modular contact patches to record body contact locations and states; 2) a visuomotor policy that mitigates the egocentric observation gap and predicts whole-body contact goals; and 3) contact-aware whole-body control formulations that realize active contact goals using suitable robot surfaces. We deploy PanTact on a Unitree G1 humanoid without additional body contact sensors, demonstrating whole-body dexterity involving body support, pushing, sweeping, and load-sharing. Website: https://pantact.github.io
Xiaomeng Xu, Han Zhang, Zhian Ruan +6
Stanford University · Amazon FAR · Carnegie Mellon University
Learning humanoid-object interaction requires coordinating whole-body balance, locomotion, and dexterous hand contact to control both robot and object motion. Human demonstrations provide examples of coordinated interaction, but transferring these behaviors to humanoid robots requires learning how to establish and maintain effective contacts under different embodiments and dynamics. We present Weave, a unified framework for learning whole-body dexterous humanoid-object interaction from captured human demonstrations. Weave first converts captured human-object interactions into executable robot-object references through contact-aware retargeting and approach-motion completion. At its core is a contact- and geometry-aware policy that jointly commands 29 body joints and 12 actuated finger joints across multiple objects and interaction sequences. Evaluation across nine objects yields a 92.5% success rate on trained interactions and, without any additional training, 65.0% on sequences never seen during training. We additionally release ~9,000 physically executed rollouts spanning ~23 hours, providing robot-object trajectories with contact annotations for downstream interaction-policy learning and physically consistent HOI motion generation. Project website: https://xiaohu-art.github.io/Weave/
Liu Cao, Xingze Wu, Jingzhi Cui +4
1Tsinghua IIIS · 3Dalian University of Technology · 4The Chinese University of Hong Kong +1