Organizations: Hongyu Ding and Jing Huo are with the School of Computer Science, Nanjing University, China. · Hongyu Ding and Yifan Zhang are with the Institute of Automation, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences, Nanjing, China. · Kanyu Xu, Zixuan Chen, Jieqi Shi and Yang Gao are with the School of Intelligence Science and Technology, Nanjing University, China. · Mingjia Chen is with the School of Electrical and Automation Engineering, Nanjing Normal University, Nanjing, China.
Rapid progress in AI agents and robotics makes it a matter of the near future that humans are served by a team of robots. Such a team will receive several goals at once, each specified in whatever form is convenient for the human, such as a category name, a photo, or a spoken description. Existing benchmarks study multi-agent, multi-modal, and multi-object goal navigation in isolation, and a benchmark that places all three in one episode is missing. To fill this gap, we introduce HM3D-M3. A team of K agents starts at distributed locations in a photorealistic indoor scene and searches for an unordered set of M object goals, each given as an open-vocabulary category, an instance image, or an instance description. HM3D-M3 provides 1,000 validation episodes over 3,926 goals from 379 open-vocabulary categories, 100K observation-assignment training samples, and two team-level metrics, goal-level SR and makespan-normalized SPL. We further contribute two baselines. HiGA is a trained hierarchical goal assigner on top of a frozen open-vocabulary navigation policy, and LaViRA-M3 is a training-free team of VLM agents with shared mapping and progress analysis. Adapted multi-robot and zero-shot methods reach 16--27% SR, while our baselines reach 38--41% with complementary strengths and distinct failure modes. Furthermore, we deploy LaViRA-M3 on a heterogeneous three-robot team. Project page: https://robo-lavira.github.io/hm3d-m3/
Figures & tables
Fig. 1 : HM3D-M 3 . Top: a team of agents searches a photorealistic scene for an unordered set of goals given as a category, an instance image, or an instance description. The benchmark provides 1,000 validation episodes, 3,926 goals, 379 categories, and 100K training samples. Bottom: goal-level SR of all methods including our two complementary baselines, and the real-world deployment on three heterogeneous robots.
Benchmark
Venue
Multi-agent
Multi-goal
Unord.
Open vocab.
ON
TN
IIN
Mix
ObjectNav [ 1 ]
arXiv’20
✗
✗
–
✗
✓
✗
✗
✗
RoboTHOR [ 2 ]
CVPR’20
✗
✗
–
✗
✓
✗
✗
✗
MultiON [ 3 ]
NeurIPS’20
✗
✓
✗
✗
✓
✗
✗
✗
CollaVN [ 4 ]
NeurIPS’21
✓
✗
–
✗
✗
✗
✗
✗
InstanceImageNav [ 5 ]
arXiv’22
✗
✗
–
✗
✗
✗
✓
✗
HM3D-OVON [ 6 ]
IROS’24
✗
✗
–
✓
✓
✗
✗
✗
TABLE I : Comparison with related goal-navigation benchmarks. Unord. means that goals are given as an unordered set, and Mix means that several modalities appear within one episode.
Fig. 2 : HM3D-M 3 validation set with 1,000 episodes and 3,926 goals. (a) Episodes per (K,M) cell. (b) Goals per modality for each M . (c) Goal spread, the mean pairwise geodesic distance between goals, against the shortest team tour d∗ , correlated with r=0.71 . (d) Number of same-category instances on the same floor for each IIN and TN goal.
Fig. 3 : The two HM3D-M 3 baselines.
Method
Coordination
SR ↑
SPL ↑
ON ↑
TN ↑
IIN ↑
VLFM [ 20 ]
independent agents on a shared map
16.1
11.8
24.5
17.7
5.9
EGO-Swarm [ 21 ]
geometric dispersion
23.7
10.7
35.5
28.0
7.5
Co-NavGPT2 [ 11 ]
centralized VLM frontier allocation
24.5
9.4
37.3
28.8
7.6
DM 3 -Nav [ 13 ]
decentralized intent broadcasting
26.6
9.9
35.9
30.4
14.2
HiGA (ours)
learned goal assignment
37.9
7.1
53.0
40.2
18.8
LaViRA-M 3 (ours)
shared goal board and progress analysis
41.0
20.3
38.6
40.9
41.4
TABLE II : Main results on the HM3D-M 3 validation split with 1,000 episodes. SR and SPL are the goal-level SR and the makespan-normalized SPL of Sec. III-C , and ON, TN, and IIN are found rates per modality. Best in bold .
Variant
SR ↑
SPL ↑
ON ↑
TN ↑
IIN ↑
HiGA , assignment strategy
random
16.8
2.7
22.7
17.3
8.3
rule-based
21.6
4.3
29.5
19.7
11.3
HiGA-BC
34.1
6.0
44.7
41.7
14.3
HiGA-DAgger
37.9
7.1
53.0
40.2
18.8
LaViRA-M 3 , coordination channels
TABLE III : Ablations on the HM3D-M 3 validation split. HiGA: assignment strategy with the navigation policy fixed. LaViRA-M 3 : coordination channels. Best within each group in bold .
Fig. 4 : Scaling with team size and goal count for LaViRA-M 3 , with 200 episodes per point on validation scenes. (a) SR grows monotonically with K at M=3 while SPL stays flat. (b) SR and SPL fall with M at K=3 .
Fig. 5 : Qualitative example and failure attribution. Left, a K=4 , M=3 episode of LaViRA-M 3 with one goal per modality, with agents in different colors. Right, where missed goals go, as a fraction of all goals, for (a) LaViRA-M 3 and (b) HiGA. A missed goal is either claimed and rejected by the protocol, or never claimed.
Fig. 6 : Real-world deployment. Left, the three platforms—an Agilex Cobot Magic, a Unitree Go1, and a Unitree Go2—exchanging the shared map as voxel records. Center, a run with three goals—an image of a yellow suitcase, the category “printer”, and the description “green chair besides the coffee bar”—with first-person views at t0 to t3 ending when each robot declares its goal found. Right, the shared map growing over time, colored by robot, with regions mapped by more than one robot in pink.
Object-goal navigation has traditionally been limited to ground robots with closed-set object vocabularies. Existing multi-agent approaches depend on precomputed probabilistic graphs tied to fixed category sets, precluding generalization to novel goals at test time. We present GoalVLM, a cooperative multi-agent framework for zero-shot, open-vocabulary object navigation. GoalVLM integrates a Vision-Language Model (VLM) directly into the decision loop, SAM3 for text-prompted detection and segmentation, and SpaceOM for spatial reasoning, enabling agents to interpret free-form language goals and score frontiers via zero-shot semantic priors without retraining. Each agent builds a BEV semantic map from depth-projected voxel splatting, while a Goal Projector back-projects detections through calibrated depth into the map for reliable goal localization. A constraint-guided reasoning layer evaluates frontiers through a structured prompt chain (scene captioning, room-type classification, perception gating, multi-frontier ranking), injecting commonsense priors into exploration. We evaluate GoalVLM on GOAT-Bench val_unseen (360 multi-subtask episodes, 1032 sequential object-goal subtasks, HM3D scenes), where each episode requires navigating to a chain of 5-7 open-vocabulary targets. GoalVLM with N=2 agents achieves 55.8% subtask SR and 18.3% SPL, competitive with state-of-the-art methods while requiring no task-specific training. Ablation studies confirm the contributions of VLM-guided frontier reasoning and depth-projected goal localization.
MoniJesu James, Amir Atef Habel, Aleksey Fedoseev +1
Intelligent Space Robotics Laboratory, Center for Digital Engineering, Skolkovo Institute of Science and Technology, Moscow, Russia
We present DM3-Nav, a fully decentralized multi-agent semantic navigation system supporting multimodal open-vocabulary goal specification and multi-object missions. In our setting, decentralization implies operation without a central coordinator, global map aggregation, or shared global state at runtime. Robots operate autonomously and coordinate through ad-hoc pairwise communication, exchanging local maps, goal status, and navigation intent without synchronization. An implicit task allocation mechanism combining intent broadcasting and distance-weighted frontier selection reduces redundant exploration while preserving decentralized operation. Evaluations on HM3DSem scenes using the HM3Dv0.2 and GOAT-Bench datasets demonstrate that DM3-Nav matches or exceeds centralized and shared-map baselines while eliminating single points of failure inherent in centralized architectures. Finally, we validate our approach in a real-world office environment using two mobile robots, demonstrating successful deployment relying entirely on onboard sensing and computation. A video of our real-world experiments is available online: https://drive.google.com/file/d/1QiUSCn5rIvtuTUqtuXLPgmt6S8x9-MCZ/view?usp=drive_link
Robots collaborating with humans must convert natural language goals into actionable, physically grounded decisions. For example, executing a command such as "go two meters to the right of the fridge" requires grounding semantic references, spatial relations, and metric constraints within a 3D scene. While recent vision language models (VLMs) demonstrate strong semantic grounding capabilities, they are not explicitly designed to reason about metric constraints in physically defined spaces. In this work, we empirically demonstrate that state-of-the-art VLM-based grounding approaches struggle with complex metric-semantic language queries. To address this limitation, we propose MAPG (Multi-Agent Probabilistic Grounding), an agentic framework that decomposes language queries into structured subcomponents and queries a VLM to ground each component. MAPG then probabilistically composes these grounded outputs to produce metrically consistent, actionable decisions in 3D space. We evaluate MAPG on the HM-EQA benchmark and show consistent performance improvements over strong baselines. Furthermore, we introduce a new benchmark, MAPG-Bench, specifically designed to evaluate metric-semantic goal grounding, addressing a gap in existing language grounding evaluations. We also present a real-world robot demonstration showing that MAPG transfers beyond simulation when a structured scene representation is available.
Swagat Padhan, Lakshya Jain, Bhavya Minesh Shah +3