COWBOY: A Scalable Sim-to-Real Framework for Learning Contextual Whole-Body Manipulation

Jiahui Yang; Jason Jingzhou Liu; Ray Muxin Liu; Andrew E. Wang; Kenneth Shaw; Deepak Pathak; Ruslan Salakhutdinov

cs.ROOct 20, 2026

COWBOY: A Scalable Sim-to-Real Framework for Learning Contextual Whole-Body Manipulation

Authors: Jiahui Yang, Jason Jingzhou Liu, Ray Muxin Liu, Andrew E. Wang, Kenneth Shaw, Deepak Pathak, Ruslan Salakhutdinov

Organizations: Carnegie Mellon University

Abstract

Mobile manipulators must adapt not only to objects, but also to the spatial context in which they appear: where to position the base, how to manipulate, and where to look while navigating surrounding scene geometry. To address this, we present COWBOY, a scalable sim-to-real framework for learning such contextual whole-body manipulation. On whole-body dexterous grasping, COWBOY transfers zero-shot to hundreds of objects and diverse in-the-wild scenes, spanning cluttered tabletops, shelves, drawers, and other unstructured settings. Rather than learning scene adaptation directly with RL in simulation, a vectorized environment-aware whole-body controller (WBC) handles reaching in thousands of scenes by coordinating base-arm motion, collision avoidance, and active vision. Then, local RL experts are trained in local scenes for object interaction and can be reused across thousands of full-scene layouts. We distill these composed WBC-RL teachers into a single vision policy that controls the mobile base, arm, dexterous hand, and actuated camera from point-cloud observations. The resulting policy exhibits contextual whole-body behavior, automatically adapting base placement, camera viewpoint, arm trajectory, and grasp strategy to observed scene and object geometry. Videos are available at https://cowboy-manip.com

Figures & tables

Figure 1: COWBOY enables learning whole-body manipulation in simulation across thousands of scenes and objects. We train a single vision-based policy that transfers zero-shot in diverse real-world scenarios across hundreds of objects. The policy adapts its grasping modes, base positioning, camera viewing directions solely depending on the scene context without explicit conditioning.

Figure 2: Left: Robot System. Our platform consists of a holonomic TidyBot++ [ 44 ] base, a Franka arm, a LEAP Hand [ 36 ] , and an ARX X5 arm carrying a RealSense D435 RGB-D camera for active vision, totaling 32 degrees of freedom. The base is additionally equipped with a Unitree L2 3D LiDAR. Right: Vectorized WBC. The vectorized fabrics WBC coordinates the base and arm to reach target end-effector poses, controls the active camera to track a look-at point, and performs self- and environment-collision avoidance.

Figure 3: For a local scene, shown as the red-shaded volume, we train a local RL policy to manipulate the target object. We then embed the same local scene within diverse full scenes, where the WBC first brings the end effector near the object while avoiding collisions to match the local policy’s initial state distribution. The local policy can then complete the task, enabling a policy trained in one local scene to generate expert motion across diverse full-scenes.

Figure 4: We use a point-cloud conditioned transformer for the student policy, architecture inspired by [ 51 ] .

Figure 5: COWBOY achieves 91% grasp success across 24 in-the-wild scenes and 100 objects. Bottom-right numbers show per-scene success over 25 objects with 3 trials each.

Figure 6: Both sets of objects not seen at training. Left: 100 random objects. Right: 20 fixed objects.

Table 7

Table 8

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Implementation	Controller Family	Collision Avoidance	Horizon Based	Native Batch API
Pink [ 4 ]	Differential IK / QP IK	Partial	No	No
Mink [ 54 ]	Differential IK / QP IK	Yes	No	No
PlaCo [ 10 ]	QP-based whole-body IK/dynamics	Yes	No	No
OpenSoT [ 17 ]	Hierarchical QP WBC	Yes	No	No
ARC-OPT WBC [ 30 ]	Optimization-based WBC	Yes	No	No
OCS2 [ 11 ]	MPC / optimal control	Yes	Yes	No

Appendix

Table 5: Representative public controller implementations compared along dimensions relevant to large-scale robot learning. The table is not a runtime benchmark; it motivates our use of a fixed-graph geometric-fabric WBC for scalable RL.

Figure 7: Left: Elongated objects for training the side-grasp expert and constrained side-grasp expert. Right: Objects for training the top-down and constrained side-grasp experts.

Figure 8: Visualization of the top down, side grasp, and constrained side grasp local expert policies.

State Term	Dimension
LEAP hand joint positions	16
Relative fingertip positions	12
Scene-box bottom-center position	3
Scene-box dimensions	3
Scene-box rotation	6
Object position	3

Appendix

Table 6: Actor observation dimensions for local RL experts. Translational quantities are expressed in the current end-effector frame, and rotations use the continuous 6D representation.

State Term	Dimension
Full robot joint positions	23
Full robot joint velocities	23
World-frame end-effector position	3
World-frame end-effector rotation	6
End-effector linear and angular velocity	6
Relative fingertip positions	12

Appendix

Table 7: Critic-state dimensions for local RL experts. The critic receives privileged robot proprioception and world-frame end-effector state in addition to the actor observation.

Parameter	Value
Algorithm	PPO continuous control
Policy model	Continuous actor with learned log standard deviation
Actor-critic network	Shared trunk, separate=False
MLP hidden units	$[512,256,128,64]$
Activation	ELU
Input normalization	Enabled

Appendix

Table 8: PPO hyperparameters used to train the local RL experts.

Parameter	Initial Setting	Terminal Setting
Observation white noise, top/shelf	$\mathcal{N}(0,0)$	$\mathcal{N}(0,0.02)$
Observation white noise, side-table	$\mathcal{N}(0,0)$	$\mathcal{N}(0,0.002)$
Observation correlated noise	$\mathcal{N}(0,0)$	$\mathcal{N}(0,0.001)$
Action white noise, top/shelf	$\mathcal{N}(0,0)$	$\mathcal{N}(0,0.10)$
Action white noise, side-table	$\mathcal{N}(0,0)$	$\mathcal{N}(0,0.05)$
Action correlated noise	$\mathcal{N}(0,0)$	$\mathcal{N}(0,0.015)$

Appendix

Table 9: Domain randomization parameters for local expert training. Gaussian entries report zero-mean perturbations with the listed terminal standard deviation. Friction randomization uses

250

buckets for the robot, object, and table.

Parameter	Initial Setting	Terminal Setting
Object wrench force	$\mathcal{U}(0,0.5)\,\mathrm{N}$	$\mathcal{U}(0,5.0)\,\mathrm{N}$
Wrench resampling interval	$1.0\,\mathrm{s}$	$1.0\,\mathrm{s}$
Torsional radius	$0.01\,\mathrm{m}$	$0.01\,\mathrm{m}$

Appendix

Table 10: Random wrench disturbance used during expert training.

Figure 9: Visualization of full-scene simulation environments. Procedural generation produces diverse scene types, including drawers, cluttered tabletops, boxes, and shelves. Each scene is paired with a local RL expert based on both the scene geometry and object geometry: elongated tabletop objects use the side-grasp expert, more compact tabletop objects use the top-down expert, and shelf scenes use the constrained side-grasp expert.

Figure 10: Pipeline for generating point cloud observations for the student policy. a) Each scene starts with primitive meshes in simulation. b) We sample points on the surface of the meshes to generate a ground-truth full point cloud. c) Given the full point cloud, we then use the vectorized depth camera and LiDAR renderer to produce a simulated point cloud for each sensor. The green point cloud is the simulated LiDAR points. The red point cloud is the simulated depth camera points. d) We finally downsample and concatenate both the simulated LiDAR and depth camera point clouds into a final point cloud that we feed as observation to the student policy.

Observation Term	Dimension
Observed point cloud in Franka-base frame	$3072\times 3$
Franka manipulation-arm joint positions	7
ARX vision-arm joint positions	6
LEAP hand joint positions	16
LEAP hand control-target delta	16
Target-object position estimate	3

Appendix

Table 11: Student observation terms. Joint states are normalized by joint limits. The point cloud is encoded as tokens by the PointNet encoder rather than flattened into a vector.

Whole-Body Robotic Manipulation

Whole-Body Robotic Manipulation coordinates a robot’s arms, hands, body, and mobility to manipulate objects while maintaining balance and contact.

See all 88 papers

Whole-Body Dexterity
Oct 15, 2026Conor Mc Gartoll, An Dang, Pranay Thangeda +5Whole-Body Robotic Manipulation Long-Horizon Robotic Manipulation
Amazon · University of California, San Diego · University of Michigan
FastGrasp: Learning-based Whole-Body Control Method for Fast Dexterous Grasping with Mobile Manipulators
Apr 14, 2026Heng Tao, Yiming Zhong, Zemin Yang +1RL for Robotics Robotic Grasping
ShanghaiTech University
Co ${}^{2}$ Skill: Whole-Body Control via Skill Composition for Long-Horizon Human-Environment Interaction
Oct 7, 2026Jeonghwan Kim, Hyeonwoo Kim, Hanbyul JooRobotic Loco-Manipulation Robot Skill Learning
Department of Computer Science, Seoul National University

Oct 15, 2026J/K move · Enter open · S save

Whole-Body Dexterity

Conor Mc Gartoll, An Dang, Pranay Thangeda, Arjun Gupta+4

Amazon · University of California, San Diego · University of Michigan

Whole-Body Robotic Manipulation Long-Horizon Robotic Manipulation RL for Robotics

Open paper AskPDF arXiv

Current humanoid dexterous manipulation remains largely confined to end-effectors and simple grasps, falling short of humans' ability to engage the entire body by bracing and sliding bulky objects against the arms and torso. We introduce Whole-Body Dexterity (WBD), a general framework that uses the robot's body and appendages as contact surfaces to grasp and manipulate objects. Our sim-to-real pipeline first generates a dataset of feasible whole-body grasps, then trains reinforcement-learning experts on a general any-to-any repositioning objective by sampling initial and goal grasps from the dataset, and distills these experts into a single policy that uses proprioception and object geometry. We show that the distilled policy transfers zero-shot to a real Unitree G1, where it repositions a basketball and a cardboard box to regions around the torso and completes a long-horizon task that stows a box under one arm, carries it from a shelf to a table, and places it. The controller operates interactively, steered by coarse motion references from a human teleoperator. Across real-world whole-body manipulation tasks, the distilled controller completes 90% of trials, compared with 18% for direct retargeting and 4% for a generalist whole-body controller.

Abstract

Whole-Body Robotic Manipulation

Explore similar work

Whole-Body Dexterity

FastGrasp: Learning-based Whole-Body Control Method for Fast Dexterous Grasping with Mobile Manipulators

Co2{}^{2}2Skill: Whole-Body Control via Skill Composition for Long-Horizon Human-Environment Interaction

Co ${}^{2}$ Skill: Whole-Body Control via Skill Composition for Long-Horizon Human-Environment Interaction