Leave It Better Than Found: Value-Guided Cooperation in Sequential Tasks

Mario Gomez Andreu; Michael Amir; Antonio Marino; Tianxu An; Marco Hutter; Amanda Prorok

cs.RONov 5, 2026

Leave It Better Than Found: Value-Guided Cooperation in Sequential Tasks

Authors: Mario Gomez Andreu, Michael Amir, Antonio Marino, Tianxu An, Marco Hutter, Amanda Prorok

Organizations: Prorok Lab, Department of Computer Science and Technology, University of Cambridge, Cambridge, United Kingdom · Robotic Systems Lab, Department of Mechanical and Process Engineering, ETH Zürich, Zürich, Switzerland

Abstract

We consider the problem of sequential task continuation, where one robot acts in a workspace that has been affected by a previous robot. Current approaches in multi-robot cooperation rely heavily on co-training or co-optimization, assumingthat the robots have access to collective interaction data during the policy training or planning process. However, as increasingly varied robotic embodiments and policies are deployed and operate in the same real-world environment, learning policies for cooperation becomes infeasible due to the combinatorial explosion of possible robot combinations that need to be accounted for. To address this challenge, this paper considers the problem of ad hoc sequential cooperation between robots that have never before operated together in a shared environment. We demonstrate that robots can explicitly communicate their environmental preferences through sharing their value functions, enabling coordination without the need for prior joint training. To ground this exchange and prevent overconfidence in out-of-distribution states, we further introduce a k-nearest neighbors (KNN)- based uncertainty filter. Our method yields an increase of up to 48.5 pp compared with naive or even selfish agents, in task success rates across simulated and real-world experiments, demonstrating that value function transmission between agents is useful not merely as an internal learning tool, but as an effective basis for ad hoc coordination. Website: https://proroklab.github.io/PAVE/

Figures & tables

Panel of Figure 1 — Figure 1: Two agents perform tasks sequentially in a shared environment: a leader pushes the table aside to clean beneath it; a follower must then traverse the room. Naive (top), the leader is unaware of the follower’s needs and leaves the table blocking the path, preventing traversal. PAVE (Ours) (bottom), the follower shares its value function over environment states, and the leader adapts by placing the table in a high-value region (overlaid with heatmap) for the follower, clearing the path, and enabling traversal.

Scenario	Setting	Leader Task	Follower Task
Mobile Arm	Heterogeneous embodiment: a robotic manipulator mounted on a mobile base.	Control the mobile base to navigate to a target object.	Control the manipulator to grasp the object; reach is restricted to a cone in front of the base.
Obstacle Arena	A pusher and a navigator share a room with a bottleneck.	Push an obstacle out of the center of the room toward the nearest wall.	Traverse the bottleneck and reach the opposite wall.
Warehouse	Two agents in a warehouse with three transfer areas (left, center, right).	Pick up a randomly placed object and transport it to one of the transfer areas.	Retrieve the object from the transfer area and place it at a designated shelf location.
Warehouse (Pickup-blocked)	Variation in which the follower, trained on the left/right transfer areas, cannot pick up from the center.	Same as Warehouse ; placing at the center transfer area is infeasible for the follower.	Pick up only from the left or right transfer area; central pick-up is impossible.
Warehouse (Pickup-unseen)	Variation in which the follower, trained on the left/right transfer areas, can in principle pick up from any of the three.	Same as Warehouse ; all three drop-offs are feasible.	Pick up from any transfer area; the central one was unseen during training.
Household	Two robots operate in the same domestic environment.	Clean up dust under the table. The table has to be moved to reach it.	Traverse the narrow pathway and reach the opposite side.

Table 1: Overview of the evaluation scenarios. The Warehouse family shares the same physical environment but differs in the follower’s training distribution and capability. Household is a hardware-only scenario.

Panel of Figure 2 — Figure 2: The three simulated scenarios. In each, a leader (first agent) reshapes the shared environment before a follower (second agent) must complete its own task from the resulting handoff state.

Figure 3: Distribution Plots of Leader and Follower Return under baselines and our method. Leader behavior either supports or hinders the follower, making the followers’ return distribution inherently multi-modal. Our approach successfully shifts this distribution toward the mode of follower success.

Scenario		Naive	PAVE (w/o OOD)	PAVE	PAVE (w/o $\pi_{L}$ )	Leader-only ( $\alpha=0$ )	FT
mobile arm	$S_{L}=$	92.89 $\pm$ 5.79	95.44 $\pm$ 2.92	99.72 $\pm$ 0.36	12.89 $\pm$ 0.57	99.94 $\pm$ 0.12	95.22 $\pm$ 2.73
	$S_{F}=$	32.67 $\pm$ 17.84	93.83 $\pm$ 2.60	81.17 $\pm$ 5.53	13.39 $\pm$ 1.24	34.44 $\pm$ 4.78	99.50 $\pm$ 0.42
	$S_{L\wedge F}=$	30.28 $\pm$ 15.81	93.61 $\pm$ 2.81	81.00 $\pm$ 5.31	10.39 $\pm$ 0.68	34.44 $\pm$ 4.78	95.11 $\pm$ 2.73
obstacle arena	$S_{L}=$	100.00 $\pm$ 0.00	100.00 $\pm$ 0.00	100.00 $\pm$ 0.00	94.22 $\pm$ 4.72	100.00 $\pm$ 0.00	100.00 $\pm$ 0.00
	$S_{F}=$	45.78 $\pm$ 11.71	84.67 $\pm$ 6.43	83.61 $\pm$ 6.64	74.17 $\pm$ 7.40	36.39 $\pm$ 11.12	54.83 $\pm$ 12.62
	$S_{L\wedge F}=$	45.78 $\pm$ 11.71	84.67 $\pm$ 6.43	83.61 $\pm$ 6.64	73.22 $\pm$ 6.72	36.39 $\pm$ 11.12	54.83 $\pm$ 12.62

Table 2: Leader / follower / joint success rates (%) per scenario across modes, each as mean

\pm

std across

6

seeds.

S_{L}

S_{F}

= leader/follower success rate.

S_{L}\wedge S_{F}

is computed per seed (counting the rollouts in which both leader and follower succeeded) and then aggregated across seeds. Bold marks the maximum between methods without access to privileged information. FT serves as a privileged comparison; bold when it outperforms PAVE . Household is hardware-only and reports the success rates of

20

leader-follower deployments with randomized initial conditions.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
Agents & Policies
$\pi_{L}$	Leader’s policy
$\pi_{F}$	Follower’s policy (fixed, independently trained)
$r_{L},r_{F}$	Leader’s and follower’s reward functions
States, Distributions, and Dynamics
$\mathbf{w}$	Shared environment state

Appendix

Table 6

Parameter	Value
step_penalty	0.01
progress_weight	1
goal_reward	25
gaussian_sigma	0.3

Appendix

Table 3: mobile_arm environment parameters.

Parameter	Value
goal_reward	25

Appendix

Table 4: obstacle_arena environment parameters.

Parameter	Value
step_penalty	0.01
step_penalty_carrying	0.05
pickup_reward	1
goal_reward	25
progress_weight	1

Appendix

Table 5: warehouse environment parameters.

Parameter	Value
step_penalty	0.01
progress_weight	3
wall_collision_penalty	0.1
pickup_reward	25

Appendix

Table 6: household environment parameters.

Hyperparameter	Value
Learning rate	0.0003
Rollout length $n_{\text{steps}}$ (per env)	2048
Minibatch size	64
Epochs per update	10
Discount $\gamma$	0.99
GAE $\lambda$	0.95

Appendix

Table 7: PPO (Stable-Baselines3, MlpPolicy ) algorithm hyperparameters.

Scenario	Follower steps	Leader steps
mobile_arm	2 M	1 M
obstacle_arena	500 k	400 k
warehouse	1.5 M	500 k
warehouse_pickup_blocked	1.5 M	500 k
warehouse_pickup_unseen	1.5 M	500 k
household	1.5 M	400 k

Appendix

Table 8: Per-agent training budget: Follower/leader total environment steps

Figure 4: Hardware setup: Two RoboMaster robots and the table (box) in the Household arena, tracked by OptiTrack. The dirt is virtual and co-located with the initial table position.

Scenario	Horizon	Samples	Iters	Elite frac.	$\sigma_{\text{noise}}$	Min. std	Replan	Dist.
mobile_arm	30	200	4	0.2	0.5	0.1	5	gaussian
obstacle_arena	40	300	10	0.3	0.5	0.208	3	gaussian
warehouse	40	500	10	0.2	1.5	0.1	5	gaussian
warehouse_pickup_blocked	40	500	10	0.2	1.5	0.1	5	gaussian
warehouse_pickup_unseen	40	500	10	0.2	1.5	0.1	5	gaussian
household	20	100	5	0.3	0.5	0.2	5	gaussian

Appendix

Table 9: Cross-entropy-method (CEM) planner hyperparameters per scenario.

Figure 5: Comparison of trajectories PAVE (w/o OOD) (dashed) and PAVE (solid) in one example of the Warehouse Pickup Blocked scenario. Left : Leader/Follower Trajectories. Right: Value function and uncertainty-weighted value function evaluated along the middle of the arena. Areas in which a drop is possible are marked in grey, blocked areas are marked in red.

Multi-Robot Coordination

Multi-robot coordination studies how teams of robots plan, communicate, allocate tasks, and act together safely and efficiently.

See all 335 papers

Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss
Jul 20, 2026Kemal Devrim Kafadar, Eren Özaltun, Mahmud Efnan Şanlı +4Multi-Agent Coordination Multi-Agent Communication
Department of Computer Engineering, Istanbul Technical University, Istanbul, Türkiye. · Department of Computer Science, University of Stuttgart, Stuttgart, Germany. · Istanbul Technical University Artificial Intelligence and Data Science Application and Research Center, Istanbul, Türkiye. +1
Multi-Agent Decision-Focused Learning via Value-Aware Sequential Communication
Apr 10, 2026Benjamin Amoh, Geoffrey Parker, Wesley MarreroMulti-Agent Coordination Multi-Agent Communication
Thayer School of Engineering, Dartmouth College
Acting on the Unseen: Communication-Free Collaborative Filtering for Decentralized Multi-Robot Task Allocation
May 25, 2026Alexander Apartsin, Yigal Meshulam, Yehudit ApersteinRobust Matrix Completion Multi-Robot Task Allocation
Holon Institute of Technology, Holon, Israel. · Afeka Tel Aviv Academic College of Engineering, Tel Aviv, Israel

Jul 20, 2026J/K move · Enter open · S save

Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss

Kemal Devrim Kafadar, Eren Özaltun, Mahmud Efnan Şanlı, Feyza Orak+3

Department of Computer Engineering, Istanbul Technical University, Istanbul, Türkiye. · Department of Computer Science, University of Stuttgart, Stuttgart, Germany. · Istanbul Technical University Artificial Intelligence and Data Science Application and Research Center, Istanbul, Türkiye.+1

Multi-Agent Coordination Multi-Agent Communication Cooperative MARL

Open paper AskPDF arXiv

Robust multi-agent coordination relies heavily on inter-agent communication, which is frequently disrupted by physical and environmental constraints in real-world deployments. To maintain operation during these intermittent communication failures, agents can employ internal prediction models to estimate missing shared state information. However, predictors trained with standard reconstruction objectives treat all transitions equally. In a Reinforcement Learning context, this forces the model to waste capacity learning stochastic exploration noise and the outdated dynamics of suboptimal policies. In this paper, we propose a value-aware extension of Multi-Agent Observation Sharing under Communication Dropout (MARO) to patch communication gaps; we refer to this method as Value-Aware MARO. By dynamically weighting the predictor's loss function using advantage estimates derived from the underlying actor-critic architecture, our objective explicitly couples the predictor's learning process to the policy's evolution. This formulation focuses the model's capacity on the intentional, high-return dynamics actively reinforced by the agents. We evaluate our framework on several tasks within the Multi-Agent Particle Environment under varying communication reliability levels. Experimental results demonstrate that our approach maintains performance under declining communication reliability, particularly below 40%. While our method performs comparably in tasks where the baseline already maintains high coordination, our value-aware weighting effectively prevents the performance collapse observed in the standard predictor during high-attrition scenarios. In these environments, our method achieves an average improvement in mean returns of more than 20% and reduces performance variance by a mean of 64.7% compared to the standard unweighted baseline.

Abstract

Multi-Robot Coordination

Explore similar work

Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss

Multi-Agent Decision-Focused Learning via Value-Aware Sequential Communication

Acting on the Unseen: Communication-Free Collaborative Filtering for Decentralized Multi-Robot Task Allocation