cs.ROSep 21, 2026

RoboTalk: Learning Multi-Robot Communication and Coordination from Multimodal Demonstrations

Authors: Dorian Benhamou GoldfajnMason NakamuraSaaduddin MahmudJustin SvegliatoKyle H. WrayShlomo Zilberstein

Abstract

Multi-robot collaboration could enable more efficient and scalable solutions to complex robotic tasks, but collaboration under partial observability remains challenging. Natural-language communication offers a promising approach to coordinating robots under partial observability. However, in decentralized manipulation, jointly learning explicit inter-robot communication and skill-level action selection from multimodal demonstrations remains underexplored for small vision-language models (VLMs) intended for on-device deployment. To address this gap, we introduce RoboTalk, a synthetic data-generation pipeline and dataset of 7,950 multimodal trajectories spanning 53 mobile-manipulation kitchen tasks for training small VLMs to communicate and coordinate. The dataset includes a leader-follower planning protocol, tool calls (perception, manipulation, navigation, and communication), rationale traces, and diversified natural-language communication. Fine-tuning open-source models on our dataset can reach 77% success on novel held-out tasks, a significant improvement over the untuned open source models, which had a success rate of around ~2%.

Explore similar work

CardsList
  1. Prompting Robot Teams with Natural Language

    Sep 29, 2025Eduardo Sebastián, Nicolas Pfitzer, Ajay Shankar +1Multi-Robot SystemsExecutable Robot Behavior