Computer-Use Agents Are Embodied Generalists
Organizations: Carnegie Mellon University
Abstract
In many areas of AI, general-purpose models have increasingly replaced handcrafted methods and specialized architectures. We argue that embodied AI is poised for a similar shift. Our key claim is that embodied tasks can be formulated as interaction with digital software. Specifically, we build software that renders an agent's view of a physical environment on a screen and provides controls for acting in it. A computer-use agent (CUA) can then perform embodied tasks through the same screenshots and keyboard and mouse actions that it uses to operate ordinary software. Using a single CUA design and changing only the software, our entries win first-, second-, third-, and sixth-place prizes across four ECCV workshop competitions, without any competition-specific agent tuning. These results suggest that the skills CUAs learn on software, such as visual grounding, precise clicking, and exploration, already carry over to embodied tasks, giving CUAs a far broader scope in embodied AI than previously realized. We therefore advocate that computer-use agents can be embodied generalists. Further, we advocate that, by following the training strategy that made computer-use agents general, embodied AI can converge on a single general model as well, ushering in a new era of general embodied AI. Website: https://pranjal2041.github.io/cua-embodied
Figures & tables
| Competition | Place | Metric | Ours | Best other |
|---|---|---|---|---|
| VLNVerse | 6th | Avg SR@3 | 87.30 | 90.70 |
| NDTW (coarse / fine) | 95.0 / 96.8 | 96.3 / 97.6 | ||
| DialNav | 3rd | Endpoint success rate | 65.26 | – |
| SPL | 52.87 | – | ||
| OpenSUN3D | 1st | AP50 | 25.88 | 21.45 |
| AP25 | 36.52 | 26.50 |
| Competition | Metric | API | GUI | GUI API |
|---|---|---|---|---|
| VLNVerse | Avg SR@3 | 83.9 | 85.2 | |
| DialNav | Endpoint success rate | 67.4 | 63.4 | |
| OpenSUN3D | AP50 | 26.7 | 28.1 | |
| MeViSv2-Text | Final | 95.8 | 94.6 |
Embodied AI
Embodied AI studies agents that perceive, reason, and act through bodies in physical or simulated environments.
See all 270 papers