Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera

Seongyong Kim; Junhyeon Cho; Kang-Won Lee; Soo-Chul Lim

cs.ROFeb 26, 2026

Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera

Authors: Seongyong Kim, Junhyeon Cho, Kang-Won Lee, Soo-Chul Lim

Organizations: Dongguk University

Abstract

To catch a thrown object, a robot must be able to perceive the object's motion and generate control actions in a timely manner. Rather than explicitly estimating the object's 3D position, this work focuses on a novel approach that recognizes object motion using pixel-level visual information extracted from a single RGB image. Such visual cues capture changes in the object's position and scale, allowing the policy to reason about the object's motion. Furthermore, to achieve stable learning in a high-DoF system composed of a robot arm equipped with a multi-fingered hand, we design a heterogeneous multi-agent reinforcement learning framework that defines the arm and hand as independent agents with distinct roles. Each agent is trained cooperatively using role-specific observations and rewards, and the learned policies are successfully transferred from simulation to the real world.

Figures & tables

Fig. 2: Pipeline of the system and experimental setup. Each policy ( $\pi_{arm},\pi_{hand}$ ) operates on selected observations from two consecutive timesteps. Privileged information is used only during value network training. A single RGB camera is mounted 0.5 m behind and 2.2 m above the robot. The arm and hand are controlled by separate policies that are trained collaboratively to catch a thrown object.

Fig. 3: (a) Objects used for training (top), validation (middle) in simulation, and real-world experiments (bottom). (b) Random object trajectories generated in simulation, shown without robot motion to highlight object dynamics.

Fig. 4: Visualization of pixel-level features in simulation and real-world environments. A bounding box is generated around the object in the RGB image, from which the corner and center coordinates, as well as width and height, are extracted. The final input features include these values and their temporal differences.

	Parameter	Type	Distribution	Range
Robot Arm	Joint stiffness	Scaling	Uniform	$[0.8,\,1.2]$
	Joint damping	Scaling	Uniform	$[0.8,\,1.2]$
	Action noise	Additive	Gaussian	$\mu=0.0,\ \sigma=0.03$
	Observation noise	Additive	Gaussian	$\mu=0.0,\ \sigma=0.005$
	Initial joint pos	Additive	Uniform	$[-0.125,\,0.125]$
Robot Hand	Joint stiffness	Scaling	Uniform	$[0.7,\,1.3]$

TABLE I: Domain randomization parameters applied in training.

Fig. 5: Tracking and success rates over training. Results are averaged over 3 seeds, and the shaded regions indicate the standard deviation.

Metric	Method	Objects
Metric	Method	Seen Objects	Unseen Objects
T.R. (%)	w/o-PF	12.13 $\,{\scriptstyle\pm\,1.27}$	12.11 $\,{\scriptstyle\pm\,1.9}$
	only-WH	8.03 $\,{\scriptstyle\pm\,0.6}$	6.11 $\,{\scriptstyle\pm\,0.35}$
	only-Center	87.07 $\,{\scriptstyle\pm\,1.62}$	87.5 $\,{\scriptstyle\pm\,1.53}$
	S-A RL	78.2 $\,{\scriptstyle\pm\,1.32}$	75.83 $\,{\scriptstyle\pm\,2.00}$
	Proposed	89.97 $\,{\scriptstyle\pm\,0.21}$	89.28 $\,{\scriptstyle\pm\,0.79}$

TABLE II: Performance of the baselines in simulation. The results are averaged over 3 seeds.

Fig. 6: Visualization of the results of deploying a trained policy in real-world experiments. This figure presents real-world catching sequences for objects with different geometries (see Fig. 3 (a)). The “mono-cam view” shows the scene captured by the installed camera, where objects are segmented using SAM2, and pixel-level features extracted from the segmentation results are provided to the policy as input.

Metric	Method	Objects
Metric	Method	Cube	L-block	Triangle
T.R. (%)	only-WH	0	0	0
	only-Center	57	54	54
	S-A RL	50	43	47
	Proposed	73	73	70
S.R. (%)	only-WH	0	0	0

TABLE III: Performance of the baselines in the real-world. Results are averaged over 30 trials for each object.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,672 papers

MAAP: Multi-Agent Active Perception for Collaborative Manipulation
Sep 18, 2026Bruno N. Y. Chen, Li Kang, Heng Zhou +4Active Vision for Robot Manipulation Robotic Manipulation
Carnegie Mellon University · Shanghai Jiao Tong University · University of Science and Technology of China +3
Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video
Jun 7, 2026Yunhai Han, Jianuo Qiu, Linhao Bai +14Robot Skill Learning Sim-to-Real Transfer
1Georgia Institute of Technology · University of Pennsylvania · 3Toyota Research Institute +1
Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration
Oct 8, 2026Jusuk Lee, Sungha Kim, Yeonsoo Park +8Robotic Manipulation Learning from Demonstration
Seoul National University · University of Maryland, College Park · All Purpose AI +1

Sep 18, 2026J/K move · Enter open · S save

MAAP: Multi-Agent Active Perception for Collaborative Manipulation

Bruno N. Y. Chen, Li Kang, Heng Zhou, Xiufeng Song+3

Carnegie Mellon University · Shanghai Jiao Tong University · University of Science and Technology of China+3

Active Vision for Robot Manipulation Robotic Manipulation Active Vision Multi-Agent Systems Imitation Learning Robot Imitation Learning+1

Open paper AskPDF arXiv

Multi-agent manipulation naturally produces multiple task-driven viewpoints: every arm carries a wrist camera and moves through the scene while acting. Yet these observations are typically underutilized, and active perception in manipulation is still often treated as requiring a dedicated sensing agent. We introduce MAAP (Multi-Agent Active Perception), in which every arm is dual-purpose: it executes manipulation actions and, through the wrist camera it carries, simultaneously serves as a moving viewpoint for the team. We pair this with RAIL (Role-Aware Imitation Learning), a controller that predicts each arm's current role alongside its action chunk and conditions action generation on it, representing role-dependent actions within one network. Across four simulated tasks, widening the perception regime lifts average success from 56.5% with a fixed camera to 62.5% with one active wrist view and 70.0% with all of them, while MAAP+RAIL reaches 79.2%. RAIL's additional gain is concentrated on the three-arm Microwave task, where success rises from 47% to 82% on identical multi-wrist inputs. On a dual-arm platform, MAAP+RAIL succeeds in 14 of 20 placement trials compared with 0 of 20 for fixed-view ACT. Collaborative manipulation can thus serve as an active perception mechanism in its own right.

Abstract

Robotic Manipulation

Explore similar work

MAAP: Multi-Agent Active Perception for Collaborative Manipulation

Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration