Organizations: Hongyu Ding and Jing Huo are with the School of Computer Science, Nanjing University, China. · Hongyu Ding and Yifan Zhang are with the Institute of Automation, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences, Nanjing, China. · Kanyu Xu, Zixuan Chen, Jieqi Shi and Yang Gao are with the School of Intelligence Science and Technology, Nanjing University, China. · Mingjia Chen is with the School of Electrical and Automation Engineering, Nanjing Normal University, Nanjing, China.
Rapid progress in AI agents and robotics makes it a matter of the near future that humans are served by a team of robots. Such a team will receive several goals at once, each specified in whatever form is convenient for the human, such as a category name, a photo, or a spoken description. Existing benchmarks study multi-agent, multi-modal, and multi-object goal navigation in isolation, and a benchmark that places all three in one episode is missing. To fill this gap, we introduce HM3D-M3. A team of K agents starts at distributed locations in a photorealistic indoor scene and searches for an unordered set of M object goals, each given as an open-vocabulary category, an instance image, or an instance description. HM3D-M3 provides 1,000 validation episodes over 3,926 goals from 379 open-vocabulary categories, 100K observation-assignment training samples, and two team-level metrics, goal-level SR and makespan-normalized SPL. We further contribute two baselines. HiGA is a trained hierarchical goal assigner on top of a frozen open-vocabulary navigation policy, and LaViRA-M3 is a training-free team of VLM agents with shared mapping and progress analysis. Adapted multi-robot and zero-shot methods reach 16--27% SR, while our baselines reach 38--41% with complementary strengths and distinct failure modes. Furthermore, we deploy LaViRA-M3 on a heterogeneous three-robot team. Project page: https://robo-lavira.github.io/hm3d-m3/
Figures & tables
Fig. 1 : HM3D-M 3 . Top: a team of agents searches a photorealistic scene for an unordered set of goals given as a category, an instance image, or an instance description. The benchmark provides 1,000 validation episodes, 3,926 goals, 379 categories, and 100K training samples. Bottom: goal-level SR of all methods including our two complementary baselines, and the real-world deployment on three heterogeneous robots.
Benchmark
Venue
Multi-agent
Multi-goal
Unord.
Open vocab.
ON
TN
IIN
Mix
ObjectNav [ 1 ]
arXiv’20
✗
✗
–
✗
✓
✗
✗
✗
RoboTHOR [ 2 ]
CVPR’20
✗
✗
–
✗
✓
✗
✗
✗
MultiON [ 3 ]
NeurIPS’20
✗
✓
✗
✗
✓
✗
✗
✗
CollaVN [ 4 ]
NeurIPS’21
✓
✗
–
✗
✗
✗
✗
✗
InstanceImageNav [ 5 ]
arXiv’22
✗
✗
–
✗
✗
✗
✓
✗
HM3D-OVON [ 6 ]
IROS’24
✗
✗
–
✓
✓
✗
✗
✗
TABLE I : Comparison with related goal-navigation benchmarks. Unord. means that goals are given as an unordered set, and Mix means that several modalities appear within one episode.
Fig. 2 : HM3D-M 3 validation set with 1,000 episodes and 3,926 goals. (a) Episodes per (K,M) cell. (b) Goals per modality for each M . (c) Goal spread, the mean pairwise geodesic distance between goals, against the shortest team tour d∗ , correlated with r=0.71 . (d) Number of same-category instances on the same floor for each IIN and TN goal.
Fig. 3 : The two HM3D-M 3 baselines.
Method
Coordination
SR ↑
SPL ↑
ON ↑
TN ↑
IIN ↑
VLFM [ 20 ]
independent agents on a shared map
16.1
11.8
24.5
17.7
5.9
EGO-Swarm [ 21 ]
geometric dispersion
23.7
10.7
35.5
28.0
7.5
Co-NavGPT2 [ 11 ]
centralized VLM frontier allocation
24.5
9.4
37.3
28.8
7.6
DM 3 -Nav [ 13 ]
decentralized intent broadcasting
26.6
9.9
35.9
30.4
14.2
HiGA (ours)
learned goal assignment
37.9
7.1
53.0
40.2
18.8
LaViRA-M 3 (ours)
shared goal board and progress analysis
41.0
20.3
38.6
40.9
41.4
TABLE II : Main results on the HM3D-M 3 validation split with 1,000 episodes. SR and SPL are the goal-level SR and the makespan-normalized SPL of Sec. III-C , and ON, TN, and IIN are found rates per modality. Best in bold .
Variant
SR ↑
SPL ↑
ON ↑
TN ↑
IIN ↑
HiGA , assignment strategy
random
16.8
2.7
22.7
17.3
8.3
rule-based
21.6
4.3
29.5
19.7
11.3
HiGA-BC
34.1
6.0
44.7
41.7
14.3
HiGA-DAgger
37.9
7.1
53.0
40.2
18.8
LaViRA-M 3 , coordination channels
TABLE III : Ablations on the HM3D-M 3 validation split. HiGA: assignment strategy with the navigation policy fixed. LaViRA-M 3 : coordination channels. Best within each group in bold .
Fig. 4 : Scaling with team size and goal count for LaViRA-M 3 , with 200 episodes per point on validation scenes. (a) SR grows monotonically with K at M=3 while SPL stays flat. (b) SR and SPL fall with M at K=3 .
Fig. 5 : Qualitative example and failure attribution. Left, a K=4 , M=3 episode of LaViRA-M 3 with one goal per modality, with agents in different colors. Right, where missed goals go, as a fraction of all goals, for (a) LaViRA-M 3 and (b) HiGA. A missed goal is either claimed and rejected by the protocol, or never claimed.
Fig. 6 : Real-world deployment. Left, the three platforms—an Agilex Cobot Magic, a Unitree Go1, and a Unitree Go2—exchanging the shared map as voxel records. Center, a run with three goals—an image of a yellow suitcase, the category “printer”, and the description “green chair besides the coffee bar”—with first-person views at t0 to t3 ending when each robot declares its goal found. Right, the shared map growing over time, colored by robot, with regions mapped by more than one robot in pink.