Observe, Share, or Predict? Disentangling Partner Information in Embodied Multi-Agent Collaboration

Adrián Szlatincsán; Filippo Ziliotto; Bruno Lepri; Luciano Serafini; Massimiliano Luca; Tommaso Campari

cs.AIOct 12, 2026

Observe, Share, or Predict? Disentangling Partner Information in Embodied Multi-Agent Collaboration

Authors: Adrián Szlatincsán, Filippo Ziliotto, Bruno Lepri, Luciano Serafini, Massimiliano Luca, Tommaso Campari

Organizations: Fondazione Bruno Kessler · University of Trento

Abstract

LLMs are used as high-level controllers for embodied agents, but it remains unclear whether they support collaborative reasoning, such as anticipating a collaborator's actions in real-time teamwork without dialogue. We ask if observing the partner's state, exchanging object knowledge, and predicting the partner's next action, shape task outcomes and coordination when agents can share percepts but never intentions, plans, or task-level messages, and whether the two agents occupy symmetric roles. Using the Habitat-based PARTNR benchmark, we introduce a Decoupled Partial Observability regime that enforces strictly individual perception, and we augment decentralized zero-shot ReAct planners with Other Agent's State (OAS) observation, object Knowledge Exchange (KE) between co-located agents, and Other Agent's Next Action (OANA) prediction, evaluated on the full 2 x 2 x 2 grid with four planners on 1,000 validation tasks. OAS observation improves task quality, reducing extraneous effort for every planner, whereas KE primarily improves execution efficiency, cutting simulation steps by up to 25% with little change in success. The two are largely redundant for strong planners but complementary for the weakest, for which KE without the OAS is the worst configuration. OANA prediction lowers the success rate in 15 of 16 model x context pairs while raising explicit partner mentions to 55-100%, and its accuracy gains lie mainly in the coarse action type, partly because agents recognize the action the partner is already performing. The robot retrieves objects sooner and completes its tasks earlier than the simulated human. Shared context reduces total task time by shortening the remaining duration that the human works alone. Also, coordination benefits arise from responding to shared context, especially the OAS, rather than explicit anticipation, yielding shallow, planner-dependent partner modeling.

Figures & tables

Figure 1: Observation settings and OANA prediction. In the object-only baseline (KE and OAS both off) of Decoupled Partial Observability, agents observe only objects and their positions (including whether an agent holds them). With object KE, co-located agents synchronize their accumulated object knowledge. With OAS observation, agents also observe the partner’s state, but only when co-located. With OANA prediction, each replanning step predicts the partner’s next action before the agent selects its own. Text in the figure is simplified relative to the prompts; the background image is AI-generated.

Figure 2: Decoupled architecture with Other Agent’s State (OAS) observation, object Knowledge Exchange (KE), and Other Agent’s Next Action (OANA) prediction . Both agents build a world graph using their own environmental object observations and their actions. The graph’s nodes represent the house, rooms, furniture, receptacles, objects, and agents. The edges represent the type of relation between them. If allowed, the agents also observe the OAS and exchange their knowledge about the objects when they are in the same room. Their high-level LLM planners call a High-Level Action (at replanning) and the corresponding low-level skills (action execution in the simulator), and predict the OANA. The predicted OANA is then compared with the other agent’s current and next actions (union target, Section 3.3 ), retrieved from its list of all called high-level actions.

			Llama-3.1-8B				Llama-3.1-70B (4-bit)				Gemini 3.1 Flash Lite				DeepSeek V3.2
OAS	KE	OANA	SR $\uparrow$	PC $\uparrow$	Steps $\downarrow$	EE $\downarrow$	SR $\uparrow$	PC $\uparrow$	Steps $\downarrow$	EE $\downarrow$	SR $\uparrow$	PC $\uparrow$	Steps $\downarrow$	EE $\downarrow$	SR $\uparrow$	PC $\uparrow$	Steps $\downarrow$	EE $\downarrow$
✗	✗	✗	60.9	77.1	4214	62.3	76.0	86.9	3056	39.4	80.4	89.8	4118	44.3	82.3	91.0	3857	38.2
✗	✗	✓	55.0	73.0	4243	62.0	72.9	85.6	3648	43.5	76.0	88.6	4304	43.5	75.2	87.1	5486	35.0
✗	✓	✗	54.3	72.9	4914	69.6	76.9	87.6	3041	43.0	79.9	89.4	3081	38.2	82.2	90.6	2951	34.8
✗	✓	✓	56.2	72.9	4121	62.7	72.1	84.6	3125	42.1	75.8	88.3	3416	35.1	75.6	87.4	4249	31.9
✓	✗	✗	58.8	74.7	4022	58.5	76.2	86.5	3425	36.5	81.2	89.8	3193	33.0	83.6	91.5	3112	28.1

Table 1: Task outcomes and redundant work under Decoupled Partial Observability for every combination of Other Agent’s State (OAS) observation, object Knowledge Exchange (KE) on co-location, and Other Agent’s Next Action (OANA) prediction. SR: Success Rate (%), PC: Percent Complete (%), Steps: Simulation Steps, EE: Extraneous Effort (%). The best value per model is bolded. All coordination metrics (Planning Cycles, Other-Agent Mention Ratio, Exploration Efficiency, Task Offloading, Extra Planning, Action Success), reported per agent, are in Table 3 (Appendix A.3 ).

Figure 3: Success Rate against episode length (Simulation Steps) for every configuration in Table 1 . Marker shape and color give the shared context (none, KE, OAS, OAS + KE); filled markers are runs without OANA prediction, and each arrow points to the same context with prediction enabled. For Gemini and DeepSeek, KE and OAS both shorten episodes, and OAS also raises Success Rate; all arrows but one point down, since prediction lowers Success Rate in fifteen of sixteen model

\times

context pairs (the exception is the Llama 8B with KE only). The y-axis ranges differ per panel.

Figure 4: Completion lag between the agents: human minus robot Extra Planning Ratio (percentage points) for each shared context, without (blue) and with (orange) OANA prediction. Positive values mean the human keeps replanning after the robot has finished. Without prediction, the human trails in every configuration, and the full shared context (OAS + KE) shrinks the lag in every model; prediction widens it for the Llama 8B and reverses it for DeepSeek, for Gemini in two contexts, and marginally for the Llama 70B with KE only (

-0.5

points).

Figure 5: Partner-prediction accuracy by shared context, for the robot predicting the human (top) and the human predicting the robot (bottom): Action Type accuracy (blue) and all-parameter accuracy given a correct action type (orange), under the union target (solid, filled markers; Table 2 ) and the strict next-action target (dashed, hollow markers; Table 8 ). Each point is a table entry; markers are offset horizontally so that close values remain visible. From None to OAS + KE, Action Type accuracy rises more than the conditional term for every planner and agent (+14.7/+14.6 against +4.8/+1.8 points for robot/human, averaged over planners, union target). The strict target lowers Action Type accuracy by 36.5% on average (27–48% across cells), while the conditional term is unchanged on average (52.2% vs. 52.4%) but planner-dependent: lower in every cell for the Llama 8B and higher in most cells for the Llama 70B and DeepSeek.

Model	OAS	KE	Action Type (R/H) $\uparrow$	Principal Par. (R/H) $\uparrow$	Action Type+Princ. Par. (R/H) $\uparrow$	All Params (R/H) $\uparrow$	Action Type +All Params (R/H) $\uparrow$	Principal Par. if Type (R/H) $\uparrow$	All Params if Type (R/H) $\uparrow$
Llama-3.1-8B-Instruct	✗	✗	31.8 / 29.5	24.6 / 21.7	17.2 / 15.8	17.8 / 15.4	15.7 / 14.1	54.2 / 53.5	49.4 / 47.8
	✗	✓	33.3 / 31.1	26.1 / 24.2	18.4 / 17.8	18.8 / 17.2	16.6 / 15.4	55.3 / 57.2	49.8 / 49.4
	✓	✗	36.1 / 33.8	22.5 / 21.5	17.8 / 17.9	18.1 / 17.0	16.4 / 15.8	49.2 / 53.1	45.5 / 46.9
	✓	✓	37.9 / 34.2	23.2 / 21.6	18.9 / 17.2	18.5 / 16.8	16.9 / 15.5	49.9 / 50.4	44.5 / 45.2
Llama-3.1-70B-Instruct 4-bit	✗	✗	38.5 / 41.8	34.2 / 37.7	24.1 / 26.5	26.7 / 30.3	22.1 / 24.3	62.5 / 63.3	57.3 / 58.1
	✗	✓	43.6 / 49.6	39.1 / 41.2	29.5 / 32.0	30.8 / 34.0	26.5 / 28.7	67.6 / 64.5	60.7 / 57.9

Table 2: Other Agent’s Next Action (OANA) prediction accuracy (%) across action abstraction levels for robot (R) and human (H) agents. The target action is the union of the current and next actions of the other agent. Results highlight that Other Agent’s State (OAS) observation and object Knowledge Exchange (KE) in the same room both improve prediction, that their combination gives the best action-type accuracy for every model and agent, and that the gains are concentrated in the action type rather than in its parameters. Per-model bests are underlined; overall bests are bolded.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Table 3: Task performance and coordination metrics under Decoupled Partial Observability (full version of Table 1 ) with different Other Agent’s State (OAS) observation, Knowledge Exchange (KE) in the same room, and Other Agent’s Next Action (OANA) prediction settings, using four different LLM planners for both agents. Results show that OAS observation mainly improves task quality while KE mainly improves execution efficiency, that the best coordination is reached with both enabled and without OANA prediction, and that OANA prediction raises partner mentioning but lowers Success Rate in all but one configuration. SR: Success Rate, PC: Percent Complete, OA Mention: Other-Agent Mention Ratio, R/H: Robot/Human agent. Subgroup bests are underlined; overall bests are bolded for each model.

Table 4: (continued)

MS	OAS	OANA	Sim Steps $\downarrow$	SR % $\uparrow$	PC % $\uparrow$	Plan. Cycles (R / H) $\downarrow$	OA Mention (R / H) % $\uparrow$	Exploration Eff. (R/H) $\downarrow$	Offload % $\uparrow$	Extra Effo. % $\downarrow$	Extra Plan. (R / H) % $\downarrow$	Action Success (R / H) % $\uparrow$
8B	✗	✗	2383	64.7	79.5	20.6 / 19.0	11.6 / 15.4	421 / 455	51.1	60.0	22.7 / 20.9	73.0 / 68.9
	✗	✓	3113	58.5	72.9	17.2 / 17.6	67.5 / 74.9	1054 / 1306	54.6	61.0	19.1 / 28.9	71.0 / 65.4
	✓	✗	2001	62.6	78.5	18.1 / 18.3	23.5 / 32.4	498 / 452	51.5	54.8	18.9 / 16.8	70.9 / 62.4
	✓	✓	2890	60.3	75.3	17.1 / 15.7	72.1 / 77.9	1053 / 1296	56.7	57.0	13.8 / 24.3	72.4 / 68.6
70B	✗	✗	1742	76.9	87.7	12.8 / 12.5	14.3 / 15.5	387 / 428	49.1	35.6	11.5 / 15.4	84.0 / 83.3
	✗	✓	1825	75.3	86.6	13.4 / 12.0	50.8 / 52.9	456 / 539	51.8	37.7	14.5 / 13.4	81.6 / 79.7

Appendix

Table 5: Task solving performance and coordination metrics under full observability setting with different Other Agent’s State (OAS) observation and Other Agent’s Next Action (OANA) prediction settings. The 4-bit quantized Llama-3.1-70B-Instruct and Llama-3.1-8B-Instruct serve as planners for both agents. MS: model size, SR: Success Rate, PC: Percent Complete, OA Mention: Other-Agent Mention Ratio, R/H: Robot/Human agent. Subgroup bests are underlined; overall bests are bolded.

Model Size	Observability	OAS	KE	OANA	Sim Steps $\downarrow$	SR % $\uparrow$	PC % $\uparrow$	Plan. Cycles $\downarrow$		OA Mention % $\uparrow$
								R	H	R	H
8B	Decoupled Partial	✗	✗	✓	4243	55.0	73.0	21.2	20.9	85.5	88.7
	Decoupled Partial	✗	✓	✓	4121	56.2	72.9	20.7	20.4	87.7	92.1
	Decoupled Partial	✓	✗	✓	4017	56.6	74.2	19.8	19.4	74.7	81.2
	Decoupled Partial	✓	✓	✓	3890	57.2	74.5	19.2	18.7	75.6	80.8
	Full	✗	-	✓	3113	58.5	72.9	17.2	17.6	67.5	74.9

Appendix

Table 6: Comparison of task performance and coordination with enabled OANA prediction across different observability scenarios. OAS: Other Agent’s State, KE: Knowledge Exchange, SR: Success Rate, PC: Percent Complete, Plan. Cycles: Planning Cycles per agent (R: Robot / H: Human), OA Mention: Other-Agent Mention Ratio (R / H %). Subgroup bests are underlined, overall bests are bolded. Larger models and fuller observability generally lead to better task success and more efficient planning. Note: For full observability, object KE is not applicable.

Model Size	Observability	OAS	KE	OANA	Sim Steps $\downarrow$	SR % $\uparrow$	PC % $\uparrow$	Plan. Cycles $\downarrow$		OA Mention % $\uparrow$
								R	H	R	H
8B	Decoupled Partial	✗	✗	✗	4214	60.9	77.1	24.2	23.2	9.8	14.6
	Decoupled Partial	✗	✓	✗	4914	54.3	72.9	27.7	25.1	9.5	17.3
	Decoupled Partial	✓	✗	✗	4022	58.8	74.7	22.1	21.1	20.5	32.7
	Decoupled Partial	✓	✓	✗	3781	62.3	76.8	21.5	19.8	21.0	30.9
	Full	✗	-	✗	2383	64.7	79.5	20.6	19.0	11.6	15.4

Appendix

Table 7: Comparison of task performance and coordination with disabled OANA prediction across different observability scenarios. OAS: Other Agent’s State, KE: Knowledge Exchange, SR: Success Rate, PC: Percent Complete, Plan. Cycles: Planning Cycles per agent (R: Robot / H: Human), OA Mention: Other-Agent Mention Ratio (R / H %). Subgroup bests are underlined, overall bests are bolded. Larger models and fuller observability generally lead to better task success and more efficient planning. Note: For full observability, object KE is not applicable.

Model	OAS	KE	Action Type (R/H) $\uparrow$	Principal Par. (R/H) $\uparrow$	A. Type+Princ. Par. (R/H) $\uparrow$	All Params (R/H) $\uparrow$	A. Type+All Params (R/H) $\uparrow$	Principal Par. if Type (R/H) $\uparrow$	All Params if Type (R/H) $\uparrow$
Llama-3.1-8B-Instruct	✗	✗	23.1 / 20.7	18.0 / 15.4	11.5 / 10.4	11.4 / 9.6	10.1 / 8.8	50.0 / 50.1	43.6 / 42.4
	✗	✓	24.0 / 21.6	19.2 / 17.2	12.2 / 12.2	12.0 / 10.8	10.4 / 9.9	50.8 / 56.4	43.3 / 45.5
	✓	✗	25.8 / 23.6	15.2 / 15.0	11.3 / 12.3	10.9 / 10.8	10.0 / 10.2	43.8 / 51.8	38.7 / 43.1
	✓	✓	26.2 / 23.4	15.8 / 14.8	11.9 / 11.4	11.1 / 10.4	9.9 / 9.6	45.5 / 48.5	37.9 / 41.1
Llama-3.1-70B-Instruct 4-bit	✗	✗	27.5 / 26.9	26.2 / 26.2	17.3 / 18.5	19.0 / 19.9	15.4 / 16.5	63.0 / 68.7	56.1 / 61.4
	✗	✓	30.7 / 31.2	30.1 / 26.9	21.5 / 21.4	22.1 / 20.7	18.6 / 18.4	69.8 / 68.5	60.6 / 58.9

Appendix

Table 8: Other Agent’s Next Action (OANA) prediction accuracy (%) across action abstraction levels for robot (R) and human (H) agents under Decoupled Partial observability. The target action is the next action of the other agent. Results highlight that OAS observation, alone or combined with KE, improves prediction for the three stronger planners, while the Llama 8B gains almost nothing at the strictest level.

Model Size	OAS	Action Type (R/H) $\uparrow$	Principal Par. (R/H) $\uparrow$	A. Type+Princ. Par. (R/H) $\uparrow$	All Params (R/H) $\uparrow$	A. Type+All Params (R/H) $\uparrow$	Principal Par. if Type (R/H) $\uparrow$	All Params if Type (R/H) $\uparrow$
8B	✗	29.3 / 33.2	26.6 / 28.6	17.0 / 20.9	17.1 / 18.6	14.1 / 16.4	58.1 / 62.7	48.1 / 49.3
	✓	33.4 / 35.5	27.2 / 28.3	20.1 / 22.1	19.1 / 20.0	16.9 / 18.4	60.3 / 62.3	50.5 / 51.6
70B	✗	57.0 / 64.2	49.7 / 53.3	40.8 / 44.6	40.8 / 44.4	35.5 / 39.3	71.5 / 69.5	62.3 / 61.3
	✓	65.6 / 70.8	50.2 / 48.2	44.5 / 43.2	42.7 / 41.7	38.5 / 38.5	67.8 / 60.9	58.7 / 54.3

Appendix

Table 9: Full observability OANA prediction accuracy (%) across action abstraction levels for robot (R) and human (H) agents. The target action is defined as the union of the current and next actions of the other agent. Results show that allowing OAS observation mostly improves prediction accuracy in the case of the 8B. Using Llama-3.1-Instruct models as planners for the agents.

Size	OAS	Action Type (R/H) $\uparrow$	Principal Par. (R/H) $\uparrow$	A. Type+Princ. Par. (R/H) $\uparrow$	All Params (R/H) $\uparrow$	A. Type+All Params (R/H) $\uparrow$	Principal Par. if Type (R/H) $\uparrow$	All Params if Type (R/H) $\uparrow$
8B	✗	23.3 / 26.3	22.9 / 23.5	14.6 / 17.6	13.5 / 14.1	11.7 / 13.3	62.5 / 67.2	50.1 / 50.7
	✓	23.4 / 26.7	21.7 / 23.0	15.5 / 18.6	13.8 / 15.4	12.2 / 14.8	66.1 / 69.5	52.4 / 55.5
70B	✗	43.8 / 45.9	41.8 / 41.4	35.3 / 36.5	33.2 / 33.3	30.3 / 31.3	80.6 / 79.5	69.2 / 68.1
	✓	43.5 / 43.1	38.6 / 36.3	35.3 / 33.9	31.4 / 30.5	29.5 / 29.3	81.2 / 78.5	67.8 / 67.9

Appendix

Table 10: Full observability OANA prediction accuracy (%) across action abstraction levels for robot (R) and human (H) agents. The target action is defined as the next action of the other agent. Results show that allowing Other Agent’s State (OAS) observation mostly improves prediction accuracy in the case of the 8B model, except for the Principal Parameter, while for the 70B model, it hinders the performance. Using Llama-3.1-Instruct models as planners for the agents.

Model Size	Observability	OAS	KE	OANA	Action Type $\uparrow$		A. Type + Princ. Param. $\uparrow$		A. Type + All Param. $\uparrow$
					R	H	R	H	R	H
8B	Decoupled Partial	✗	✗	✓	31.8	29.5	17.2	15.8	15.7	14.1
	Decoupled Partial	✗	✓	✓	33.3	31.1	18.4	17.8	16.6	15.4
	Decoupled Partial	✓	✗	✓	36.1	33.8	17.8	17.9	16.4	15.8
	Decoupled Partial	✓	✓	✓	37.9	34.2	18.9	17.2	16.9	15.5
	Full	✗	-	✓	29.3	33.2	17.0	20.9	14.1	16.4

Appendix

Table 11: OANA prediction accuracy (%) comparison across Decoupled Partial and Full observability settings for robot (R) and human (H) agents. The target action is the union of the current and next actions of the other agent. Results show that full observability improves prediction for the Llama 70B and the 8B human agent, while the 8B robot’s Action Type accuracy falls below its best partial-observability configuration. Subgroup bests (per model scale) are underlined, and overall bests are bolded.

Model Size	Observability	OAS	KE	OANA	Action Type $\uparrow$		A. Type + Princ. Param. $\uparrow$		A. Type + All Param. $\uparrow$
					R	H	R	H	R	H
8B	Decoupled Partial	✗	✗	✓	23.1	20.7	11.5	10.4	10.1	8.8
	Decoupled Partial	✗	✓	✓	24.0	21.6	12.2	12.2	10.4	9.9
	Decoupled Partial	✓	✗	✓	25.8	23.6	11.3	12.3	10.0	10.2
	Decoupled Partial	✓	✓	✓	26.2	23.4	11.9	11.4	9.9	9.6
	Full	✗	-	✓	23.3	26.3	14.6	17.6	11.7	13.3

Appendix

Table 12: OANA prediction accuracy (%) comparison across Decoupled Partial and Full observability settings for robot (R) and human (H) agents. The target action is the next action of the other agent. Results highlight that full observability improves prediction for the Llama 70B and the 8B human agent, while the 8B robot’s Action Type accuracy falls below its best partial-observability configuration; under full observability, adding OAS does not improve the 70B’s next-action prediction. Subgroup bests (per model scale) are underlined, and overall bests are bolded.

Multi-Agent Collaboration

Multi-agent collaboration studies how autonomous agents coordinate, share information, and combine capabilities to solve tasks together.

See all 309 papers

Embodied Multi-Agent Coordination by Aligning World Models Through Dialogue
May 13, 2026Vardhan Dongre, Dilek Hakkani-TürMulti-Agent Coordination AI Agent Benchmarks
Siebel School of Computing & Data Science University of Illinois Urbana-Champaign
MACRO-LLM: LLM-Empowered Multi-Agent Collaborative Reasoning under Spatiotemporal Partial Observability
Jan 14, 2026Handi Chen, Running Zhao, Xiuzhe Wu +2Multi-Agent LLM Systems Multi-Agent Coordination
The University of Hong Kong
Partially Observable Zero-shot coordination by Predicting Intention of Partner
Oct 6, 2026Jinnyeong Yang, Yuhwan Jeong, Hoyong Kwon +3Multi-Agent Coordination Partially Observable RL
KAIST, Visual Intelligence Lab

May 13, 2026J/K move · Enter open · S save

Embodied Multi-Agent Coordination by Aligning World Models Through Dialogue

Vardhan Dongre, Dilek Hakkani-Tür

Siebel School of Computing & Data Science University of Illinois Urbana-Champaign

Multi-Agent Coordination AI Agent Benchmarks Multi-Agent Communication Embodied World Models

Open paper AskPDF arXiv

Effective collaboration between embodied agents requires more than acting in a shared environment; it demands communication grounded in each agent's evolving understanding of the world. When agents can only partially observe their surroundings, coordination without communication is provably hard, but communication can, in principle, bridge this gap by allowing agents to share observations and align their world models. In this work, we examine whether LLM-based embodied agents actually realize the ability to communicate. We extend PARTNR, a benchmark for collaborative household robotics, with a natural-language dialogue channel that enables two agents with partial observability to communicate during task execution. To evaluate whether dialogue leads to genuine world-model alignment rather than superficial coordination, we propose a framework for measuring world-model alignment defined over per-agent world graphs: observation convergence (do private world models align over time?), information novelty (do messages convey what the partner lacks?), and belief-sensitive messaging (do agents model what their partner knows?). Our experiments across three LLMs reveal that dialogue reduces action conflicts 40 to 83 percentage points but degrades task success relative to silent coordination. Using our metrics, we characterize the gap between superficial coordination and genuine world-model alignment, and identify where current models fall on this spectrum. Project Website: https://uiuc-conversational-ai-lab.github.io/partnr-dial-wmd/

Abstract

Multi-Agent Collaboration

Explore similar work

Embodied Multi-Agent Coordination by Aligning World Models Through Dialogue

MACRO-LLM: LLM-Empowered Multi-Agent Collaborative Reasoning under Spatiotemporal Partial Observability

Partially Observable Zero-shot coordination by Predicting Intention of Partner