Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use

Yanzhe Chen; Ziyi Yang; Jifeng Zhu; Qiming Huang; Ruihe An; Peiyao Xu; Hesen Yang; Runda Liu

cs.ROOct 21, 2026

Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use

Authors: Yanzhe Chen, Ziyi Yang, Jifeng Zhu, Qiming Huang, Ruihe An, Peiyao Xu, Hesen Yang, Runda Liu,

Organizations: Show Lab, National University of Singapore

Abstract

Multimodal agents (MMAs) sustain interaction through reasoning, memory, tools, and feedback, most visibly as computer-use agents, while robotic systems (RSs) couple sensing and actuation under physical dynamics. We define multimodal embodied agents (MMEAs) as goal-directed systems that couple multimodal task reasoning with physical action and revise decisions from the resulting feedback. This raises a central question: what changes when a multimodal agent moves from computer-use to robot-use? We introduce PAPAV, a unified capability-centric framework of five recurring functions: Perceive the current state, Anticipate action effects, Plan a feasible course, Act through an interface or body, and Verify the outcome. Defined by function rather than architecture, these capabilities let PAPAV identify what all three share and then compare MMEAs with MMAs and with RSs. Five physical constraints, from partial observability to unverifiable outcomes, leave fewer choices fixed in advance and less room to reverse errors. Across 64 benchmarks, explicit evaluation centers on Act; only 5 assess Anticipate and 3 assess Verify, leaving both largely hidden behind task success. These constraints also frame the open challenges we identify. PAPAV therefore provides a common basis for designing reliable physical agents and measuring progress across all five capabilities rather than task success alone.

Figures & tables

Figure 1: A Unified Capability-centric Perspective (PAPAV) from computer-use to robot-use. Perceive , Anticipate , Plan , Act , and Verify are five functional roles that recur in goal-directed interaction, whether an agent operates a computer (top) or a robot (bottom). Defined by function rather than architecture, they place both settings on a common coordinate and isolate what changes once actions acquire physical consequences.

Figure 2: Chronological overview of representative work. Robotic systems progress from visual foresight and task planning to generalist policies, world models, and multi-site evaluation; multimodal agents from tool-augmented reasoning to GUI grounding and computer-use benchmarks. The middle trunk appears in 2022 as language-model planners meet robot skills and keeps widening.

Survey	Primary Focus	Computer- Use	Robot- Use	Benchmark Analysis	Digit.–Phys. Analogy
Agent AI [ 40 ]	Multimodal interaction	$\checkmark$	$\checkmark$	$\times$	$\times$
Multimodal Agents [ 197 ]	Multimodal agent architecture	$\checkmark$	$\times$	$\checkmark$	$\times$
Embodied AI [ 120 ]	Multimodal models for embodiment	$\checkmark$	$\checkmark$	$\times$	$\times$
GUI Agents [ 181 ]	(M)LLM-based GUI Agents	$\checkmark$	$\times$	$\checkmark$	$\times$
VLA Models [ 133 ]	VLA-based embodied AI	$\times$	$\checkmark$	$\checkmark$	$\times$
Robot Agents [ 164 ]	LLM/VLM-driven robot autonomy	$\checkmark$	$\checkmark$	$\times$	$\times$

Table 1: Positioning related surveys. Checks mark dimensions central to a survey; crosses mark those outside it. Six of the ten surveys cover both computer-use and robot-use and four analyze benchmarks, yet none compares the two settings as one capability structure. P A P A V adds that digital–physical analogy as an explicit axis.

Figure 4: PAPAV from computer-use to robot-use. Each row contrasts how a shared capability is realized in multimodal agents, multimodal embodied agents, and robotic systems.

Figure 5: Perceive: MMEA vs. MMA vs. RS. Shared (center): multimodal evidence grounding. Left: digital evidence inquiry vs. embodied physical evidence acquisition. Right: sensing needs formed by the ongoing MMEA task vs. RS acquisition under supplied sensing objectives.

Figure 6: Anticipate: MMEA vs. MMA vs. RS. Shared (center): action-conditioned state-transition forecasting. Left: relatively rule-bounded digital futures vs. physical forecasts qualified by uncertainty, risk, and applicability. Right: physical counterfactuals selected for the current task vs. control-objective-scoped predictions.

Figure 7: Plan: MMEA vs. MMA vs. RS. Shared (center): hierarchical, constraint-aware course selection. Left: checkpoint-assisted digital backtracking (a representative MMA case) vs. recovery planning from the reached physical state. Right: task-level committed-course revision vs. skill-local recovery.

Figure 8: Act: MMEA vs. MMA vs. RS. Shared (center): semantic action grounding into environment-compatible forms. Left: compact agent-facing operations vs. body-grounded execution over physical time. Right: execution progress visible and consequential to the MMEA task process vs. controller-local execution state.

Figure 9: Verify: MMEA vs. MMA vs. RS. Shared (center): evidence-grounded verification against independently specified expectations. Left: structured digital checks vs. physical verdicts calibrated under partial, uncertain, potentially delayed evidence. Right: task-consumed vs. locally used monitor or skill verdicts.

Benchmark	Venue	Task	Per.	Ant.	Plan	Act	Ver.
Category: Multimodal Embodied Agent
Subcategory: Simulation
TEACh [ 152 ]	AAAI’22	Dialogue-guided tasks	◐	◐	◐	●	◐
RoCoBench [ 137 ]	ICRA’24	Collaborative coordination	○	◐	●	●	◐
PARTNR [ 20 ]	ICLR’25	Collaborative planning	●	◐	●	●	◐
EMBODIEDBENCH [ 208 ]	ICML’25	Embodied tasks	●	◐	●	●	◐

Table 2: Overview and PAPAV coverage of 64 curated benchmarks (10 MMEA, 23 MMA, and 31 RS) . Per./Ant./Ver. denote Perceive/Anticipate/Verify, respectively. ACL-F, CVPRW, and NeurIPS-DB denote Findings of ACL, CVPR Workshop, and NeurIPS Datasets and Benchmarks, respectively.

Panel of Figure 10 — Figure 10: Analysis of benchmark metrics and cross-domain PAPAV coverage. Left: Sankey diagram tracing benchmark metric types across MMA, MMEA, and RS benchmarks. Right: Three-level PAPAV capability coverage matrices for MMA, MMEA, and RS benchmarks. Green, yellow, and red denote Explicit Evaluation, Implicit Evaluation, and Not Evaluated, respectively.

Multimodal Agents

Multimodal agents perceive and reason across text, images, audio, and video to use tools, interact with interfaces, and act in the world.

See all 387 papers

A Multimodal Framework for Human-Multi-Agent Interaction
Mar 24, 2026Shaid Hasan, Breenice Lee, Sujan Sarker +1Human-Robot Interaction Human-AI Interaction
University of Virginia
RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments
Oct 7, 2026Zhiqin Yang, Chenxin Li, Xiaomeng Hu +30Robot Manipulation Benchmarks Multimodal Agents
MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments
Jun 30, 2026Qingyun Liu, Jiwen Zhang, Jingyi Hu +2Multi-Agent Collaboration AI Agent Benchmarks
1Fudan University · 3The Chinese University of Hong Kong · 2Shanghai Innovation Institute

Mar 24, 2026J/K move · Enter open · S save

A Multimodal Framework for Human-Multi-Agent Interaction

Shaid Hasan, Breenice Lee, Sujan Sarker, Tariq Iqbal

University of Virginia

Human-Robot Interaction Human-AI Interaction Multimodal Agents Multi-Agent Coordination

Open paper AskPDF arXiv

Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.

Abstract

Multimodal Agents

Explore similar work

A Multimodal Framework for Human-Multi-Agent Interaction

RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments

MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments