CoEnv: Driving Embodied Multi-Agent Collaboration via Compositional Environment
Authors: Li Kang, Yutao Fan, Rui Li, Heng Zhou, Yiran Qin, Zhemeng Zhang, Songtao Huang, Xiufeng Song, Zaibin Zhang, Bruno N. Y. Chen, Zhenfei Yin, Dongzhan Zhou, Wangmeng Zuo, Lei Bai
Organizations: Shanghai Jiao Tong University · Shanghai AI Laboratory · Harbin Institute of Technology · University of Science and Technology of China · CUHK-Shenzhen · Fudan University · Dalian University of Technology · Carnegie Mellon University · University of Oxford
Multi-agent embodied systems hold promise for complex collaborative manipulation, yet face critical challenges in spatial coordination, temporal reasoning, and shared workspace awareness. Inspired by human collaboration where cognitive planning occurs separately from physical execution, we introduce the concept of compositional environment -- a synergistic integration of real-world and simulation components that enables multiple robotic agents to perceive intentions and operate within a unified decision-making space. Building on this concept, we present CoEnv, a framework that leverages simulation for safe strategy exploration while ensuring reliable real-world deployment. CoEnv operates through three stages: real-to-sim scene reconstruction that digitizes physical workspaces, VLM-driven action synthesis supporting both real-time planning with high-level interfaces and iterative planning with code-based trajectory generation, and validated sim-to-real transfer with collision detection for safe deployment. Extensive experiments on challenging multi-arm manipulation benchmarks demonstrate CoEnv's effectiveness in achieving high task success rates and execution efficiency, establishing a new paradigm for multi-agent embodied AI.
Figures & tables
Figure 1: Motivation of CoEnv. Physical-world execution offers high fidelity but risks costly collisions, while the digital world enables cost-effective and safe testing. CoEnv composes both worlds through pose and action alignment, forming a compositional environment ( left ) that supports real-to-sim reconstruction, simulation-conditioned action synthesis, and safe real-world deployment ( right ).
Figure 2: Overview of the CoEnv framework. Top-left: Real-to-Sim Scene Reconstruction converts multi-view RGBD observations into a simulator-ready scene via 3D asset generation, object localization (Grounded SAM 2 + GPT-5 + FoundationPose), and iterative camera calibration. Top-right: Simulation-Conditioned Action Synthesis performs hierarchical task planning followed by grounded execution in either interactive mode (closed-loop VLM feedback) or iterative mode (code agent with iterative refinement). Bottom: Sim-to-Real Transfer deploys validated trajectories via trajectory interpolation with collision volume verification for safe multi-agent execution.
Figure 3: Task demonstrations. We evaluate CoEnv on five multi-agent manipulation tasks with increasing coordination complexity. Top three rows: two-agent tasks (Franka × 2) including Cube Stacking, Ball Pickup, and Transfer Cylinder. Bottom two rows: three-agent tasks (Franka + AgileX Piper) including Place Cucumber and Brush Box. Each row shows keyframes from a successful real-world execution.
Table 1: Real-world multi-agent task evaluation. Each task is evaluated over 10 trials under two execution modes: Interactive (closed-loop VLM feedback) and Iterative (code-based trajectory generation). Si : i -th subtask milestone completion ( x /10). SR: overall task success ( x /10). Overall: combined success rate (%) across both modes.
w/o Adaptive Camera
w/o Checkpoint Verif.
Task
S1
S2
S3
SR
S1
S2
S3
SR
Cube Stacking
6/10
5/10
–
5/10
4/10
4/10
–
4/10
Ball Pickup
10/10
6/10
–
6/10
10/10
2/10
–
2/10
Transfer Cylinder
6/10
2/10
0/10
0/10
6/10
0/10
0/10
0/10
Place Cucumber
9/10
6/10
4/10
4/10
6/10
8/10
4/10
4/10
Brush Box
10/10
8/10
0/10
0/10
9/10
8/10
0/10
0/10
Table 2: Ablation study on Interactive mode. We separately remove the adaptive camera control (w/o Camera) and checkpoint verification (w/o Ckpt.) mechanisms. SR: task success rate ( x /10).
Figure 5: Scalable data collection pipeline. CoEnv first synthesizes and validates manipulation strategies in simulation, then transfers them to real robots for physical execution, collecting real-world multi-agent trajectory data that can be used to train generalist policies, providing an alternative to manual teleoperation.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Description
Two-Agent (Franka × 2)
Cube Stacking
Two arms stack a blue cube and a red cube separately.
Ball Pickup
Bimanual coordination to lift a soccer ball with balanced contact.
Transfer Cylinder
Pick up a cylinder, perform a bimanual handover and place it at the target location.
Three-Agent (Franka + AgileX Piper × 2)
Place Cucumber
Pick up the pot lid and place the cucumbers into the pot.
The cylinder is grasped and lifted by the first arm.
S2
The second arm successfully receives the cylinder.
Appendix
Table 5: Subtask milestones and success criteria. The last milestone of each task defines the overall success condition.
Aspect
Interactive Mode
Iterative Mode
Foundation model
GPT-5 (VLM)
Claude Code (code agent)
Decision granularity
One action per VLM call
Full program per iteration
Feedback signal
Real-time images + state
Post-hoc checkpoints + state
API calls / task
∼ 30+ VLM calls
∼ 1–5 code generations
Error correction
Adjust next action online
Rewrite code offline
Planning
Multi-round visual analysis
Structured system prompt
Appendix
Table 6: Comparison of Interactive and Iterative execution modes.
Figure 6: Representative failure cases. From left to right: (1) Cube Stacking : the stacked cube slips off due to sim-to-real positional offset at placement; (2) Transfer Cylinder : the receiving arm fails to align with the handing arm’s gripper during handover, causing the cylinder to drop; (3) Place Cucumber : the pot lid is grasped at an unstable point (red circle), causing it to fall during the holding phase; (4) Brush Box : planning error causes the brush to be dropped (red rectangle) during moving phase.
Category
Content
Gripper specifications
Maximum opening width and finger length for each robot type (Franka, Piper).
Orientation semantics
Delta-based rotation convention; mapping from pitch/yaw values to physical gripper directions.
Grasp strategy guidelines
Shape-conditioned heuristics (top-down for flat objects, horizontal for upright cylinders, cooperative bimanual for oversized objects); edge-grasp safety margins; fingertip positioning rules (overshoot for horizontal grasps, low- z target for flat objects).
Workspace limits
Per-robot reachable workspace ranges.
Collision avoidance
Awareness that arm links sweep a volume beyond the TCP; minimum clearance thresholds; retreat-toward-base principle for clearing shared workspace.
Appendix
Table 7: Shared domain knowledge embedded in the system prompts of both execution modes. All entries are generic robot/physics specifications, not task-specific solutions.