Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination

Suyu Ye; Zheyuan Zhang; Vaishnav Tadiparthi; Hossein Nourkhiz Mahjoub; Ehsan Moradi Pari; Tianmin Shu; Homanga Bharadhwaj; Nakul Agarwal

cs.ROOct 1, 2026

Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination

Authors: Suyu Ye, Zheyuan Zhang, Vaishnav Tadiparthi, Hossein Nourkhiz Mahjoub, Ehsan Moradi Pari, Tianmin Shu, Homanga Bharadhwaj, Nakul Agarwal

Organizations: Honda Research Institute USA · Johns Hopkins University

Abstract

Robots operating in the physical world will increasingly need to coordinate with other robots, particularly in manipulation tasks where an object may be too large or heavy for a single robot to carry alone. Physical limitations caused by hardware degradation or actuator faults can restrict the actions a robot can reliably execute, yet these limitations may be unknown to its partner. We study whether a helper can infer a robot partner's physical constraints from observing it coordinate with another robot, then use the inferred capability to coordinate with the same partner on a new task. This is difficult because a demonstration shows what the constrained robot did, but not what it could have done. In physically coupled tasks, the other robot may also compensate for its limitations, making those limitations difficult to identify from the constrained robot's behavior alone. Our key insight is that these constraints shape the joint behavior of the team, making the actions of both robots informative about the constrained partner's capability. We introduce Watch, Infer, Coordinate, a benchmark spanning three physically coupled manipulation settings, together with an inference approach that scores candidate constraints using observed joint behavior. Across all three settings, our method substantially improves constraint inference and zero-shot coordination, approaching an oracle with access to the true constraints.

Figures & tables

Figure 1: Overview of our problem setting. A helper must coordinate with a constrained partner whose physical capabilities are initially unknown. Rather than interacting with the partner directly, the helper first watches it coordinate with another robot and uses this demonstration to infer constraints on its low-level actions, such as joint position, joint velocity, or base-direction limits. The helper then replaces the demonstrator and must coordinate with the same constrained partner on a new task. Although the required coordination changes between tasks, the partner’s physical constraints remain the same, allowing the helper to transfer what it learned about the partner’s capabilities to the new task.

Work	Prior-Trajectory Inference	Multi-Agent Trajectories	Low-Level Partner Constraints	Continuous Control	Physically Coupled Manipulation	Transfer to New Task
CHORUS ( Doshi et al., 2026 )	✗	✗	✗	✓	✓	✗
Watch-and-Help ( Puig et al., 2021 )	✓	✗	✗	✗	✗	✗
TICC-POMDP ( Lee et al., 2020 )	✗	✗	✗	✗	✗	✓
Smart Help ( Cao et al., 2024 )	✗	✗	✗	✗	✗	✗
CHAIC ( Du et al., 2024 )	✗	✗	✗	✗	✓	✗
CE-CM-Div ( Tisnikar et al., 2026 )	✓	✓	✗	✗	✗	✓

Table 1: Comparison with related work. Prior-Trajectory Inference indicates that episode-specific information is inferred from trajectories observed before the downstream task. Multi-Agent Trajectories indicates that the trajectories used for inference contain behavior from multiple interacting agents. Low-Level Partner Constraints indicates that the inferred quantity specifies low-level constraints such as limits on the partner’s joint positions, joint velocities, or allowable motion directions, rather than task-level capabilities such as reachable height or liftable weight. Continuous Control indicates continuous-valued physical control rather than symbolic or macro actions. Physically Coupled Manipulation indicates that multiple agents jointly manipulate the same object such that their actions mechanically combine to determine its motion. Transfer to New Task indicates that the inferred information is subsequently used on a distinct task or goal requiring different behavior.

Figure 2: Overview of our physically coupled partner-adaptation benchmark. (A) During the observation stage, the helper watches a demonstrator coordinate with the constrained agent on an obstacle-free task. The labels show physical constraints on the action space of the constrained agent, which are dependent on the setting. (B) At test time, the helper replaces the demonstrator and coordinates zero-shot with the same constrained agent on a new task with obstacles. The benchmark includes three settings of increasing physical complexity: 2D rod carrying, fixed-base dual-UR5 carrying, and mobile dual-UR5 carrying. Across all settings, the constrained agent’s physical constraints remain fixed while the task and helper change.

$N$	CE-CM-Div	Ours, partner only	Ours	CE-CM-Div	Ours, partner only	Ours	CE-CM-Div	Ours, partner only	Ours
	2D Rod			Fixed-Base Dual-UR5			Mobile Dual-UR5
1	$0.304\pm 0.038$	$0.104\pm 0.015$	0.058 $\pm$ 0.017	$0.074\pm 0.000$	$0.068\pm 0.006$	0.031 $\pm$ 0.012	$0.306\pm 0.028$	$0.201\pm 0.045$	0.190 $\pm$ 0.060
2	$0.340\pm 0.041$	$0.031\pm 0.009$	0.003 $\pm$ 0.002	$0.074\pm 0.000$	$0.037\pm 0.000$	0.019 $\pm$ 0.000	$0.278\pm 0.028$	$0.148\pm 0.042$	0.130 $\pm$ 0.060
4	$0.348\pm 0.031$	0.002 $\pm$ 0.002	0.002 $\pm$ 0.002	$0.074\pm 0.000$	$0.037\pm 0.000$	0.019 $\pm$ 0.000	$0.250\pm 0.000$	$0.090\pm 0.006$	0.069 $\pm$ 0.005
8	$0.302\pm 0.011$	0.000 $\pm$ 0.000	$0.002\pm 0.002$	$0.074\pm 0.000$	$0.043\pm 0.006$	0.019 $\pm$ 0.011	$0.250\pm 0.000$	$0.042\pm 0.000$	0.009 $\pm$ 0.000

Table 2: Constraint-inference accuracy at

N\in\{1,2,4,8\}

demonstrations. We report normalized Hamming distance between the inferred and true constraints. “Ours, partner only” uses only the constrained agent’s behavior for inference and excludes the demonstrator’s behavior. Values are mean

\pm

standard error; lower is better. Bold indicates the best result within each robot setting and row, including ties.

		2D Rod		Fixed-Base Dual-UR5		Mobile Dual-UR5
Task	Method	Success $\uparrow$	SNA $\uparrow$	Success $\uparrow$	SNA $\uparrow$	Success $\uparrow$	SNA $\uparrow$
Obstacle-free control	Capacity-blind + CEM	100.0 $\pm$ 0.0	$91.4\pm 0.9$	$0.0\pm 0.0$	$0.0\pm 0.0$	$66.7\pm 0.0$	$55.2\pm 0.0$
	CE-CM-Div + CEM	100.0 $\pm$ 0.0	$90.4\pm 1.6$	$55.6\pm 11.1$	$30.5\pm 8.3$	$66.7\pm 0.0$	$54.3\pm 0.0$
	Behavioral cloning	$66.7\pm 0.0$	$52.5\pm 6.1$	$22.2\pm 11.1$	$7.5\pm 3.8$	$66.7\pm 0.0$	$31.1\pm 0.0$
	Oracle + CEM	$100.0\pm 0.0$	$91.4\pm 0.2$	$100.0\pm 0.0$	$44.8\pm 1.8$	$100.0\pm 0.0$	$80.2\pm 0.0$
	Ours + CEM	100.0 $\pm$ 0.0	92.4 $\pm$ 0.4	100.0 $\pm$ 0.0	43.9 $\pm$ 0.8	100.0 $\pm$ 0.0	80.2 $\pm$ 0.0

Table 3: Planning performance on the obstacle-free control and new tasks with obstacles. Our method, CE-CM-Div, and behavioral cloning use

N=8

demonstrations. Capacity-blind CEM uses no demonstrations, while Oracle CEM receives the true constraints. Success and SNA are percentages reported as mean

\pm

standard error over three seeds; higher is better. Bold indicates the best non-oracle result for each task and metric, including ties.

Figure 3: Qualitative examples of zero-shot coordination using our method. (A) Representative successful trajectories for 2D rod carrying, fixed-base dual-UR5 carrying, and mobile dual-UR5 carrying, shown from left to right over time. (B) Example of how an inferred physical constraint changes test-time planning. Ignoring the constrained agent’s limitation leads to a pose it cannot reach, while incorporating the inferred constraint causes the planner to select a feasible alternative.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Setting	Horizon	Population	Elites	Iterations
2D Rod	32	64	8	3
Fixed-Base Dual-UR5	4	16	2	2
Mobile Dual-UR5	4	4	1	1

Appendix

Table 4: CEM-MPC hyperparameters used for zero-shot coordination.

Explore similar work

IMBench: A Benchmark for Intuitive Robotic Manipulation
Jul 17, 2026Anurag Maurya, Sukhvansh Jain, Prajwal Avhad +9Robotic Manipulation Robot Policies
Manav Robotics, Bengaluru, India
WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation
Jun 24, 2026Baiqi Li, Ce Zhang, Yu Fang +4Robotic Manipulation Reproducibility
University of North Carolina at Chapel Hill
Structured LLM Reasoning for Zero-Shot Human--Robot Coordination Under Hidden Goals
Aug 5, 2026Dong Hae Mangalindan, Anand Gokhale, Francesco Bullo +1Human-Robot Collaboration Theory-Of-Mind Reasoning
Department of Electrical and Computer Engineering, Michigan State University, East Lansing, MI 48824 · Center for Control, Dynamical Systems, and Computation, UC Santa Barbara, Santa Barbara, CA 93106 USA

Jul 17, 2026, cs.ROJ/K move · Enter open · S save

IMBench: A Benchmark for Intuitive Robotic Manipulation

Anurag Maurya, Sukhvansh Jain, Prajwal Avhad, Gautham Balachandran+8

Manav Robotics, Bengaluru, India

Robotic Manipulation Robot Policies Intuition Task-Relevant Information

Open paper AskPDF arXiv

Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints. They build an understanding of the physical world that helps them convert reasoning into actions and quickly adapt to new scenes, tasks, and rules. We refer to this capability as intuitive manipulation. Existing benchmarks fail to capture this integration: they evaluate physical reasoning in isolation from execution, or measure policy performance without requiring explicit reasoning. We introduce IMBENCH, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution. Our tasks require models to infer task-relevant physical structure and generate feasible action sequences under explicit constraints, including contact-rich manipulation, tool use, and multi-stage dependencies. We introduce a benchmark of 35 tasks, 14K filtered trajectories, and scalable tools for generating diverse scenarios. Experiments reveal a consistent gap: vision language models show partial physical reasoning ability but fail to produce executable plans, while state-of-the-art vision-language-action models struggle to satisfy task constraints and generalize across scenarios. These results identify intuitive manipulation as a missing axis in current foundation models and generalist robot policies, and position IMBENCH as a step toward evaluating and enabling more integrated, adaptive physical intelligence.

Abstract

Explore similar work

IMBench: A Benchmark for Intuitive Robotic Manipulation

WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation

Structured LLM Reasoning for Zero-Shot Human--Robot Coordination Under Hidden Goals