Racing the Clock: Teaching VLNs Budget-Conditioned Object Search
Organizations: University of California, Berkeley · Stanford University
Abstract
How do action budgets shape vision-language navigation (VLN) exploration, and what expert supervision best teaches budget-aware search? We study budget-conditioned object search: finding as many distinct instances of a requested category as possible within a budget, with discovery defined geometrically by camera field of view and proximity. Distinct discoveries provide a denser evaluation signal than single-target success. We cast expert design as a choice of scene abstraction: a single budget-indexed Bellman recursion, applied to different abstractions of the full environment, yields experts that differ in what they know. An object-level Bellman expert is given every target location and reduces search to budgeted routing. A room-level Bellman expert retains observed room connectivity, doorway evidence, and cross-scene semantic priors but withholds the contents of unexplored rooms, so it must search for targets that its student also cannot see. We distill both experts into Uni-NaVid under matched training conditions and also evaluate greedy-frontier supervision on held-out HM3D scenes. Changing only the stated budget changes learned trajectories. Despite higher object-level expert return, room-level supervision achieves the highest mean collection among evaluated VLNs at every tested budget. At 2000 steps, it finds 72.7% of reachable targets, versus 68.5% under greedy-frontier supervision, 65.5% under object-level supervision, and 44.8% for the pretrained VLN. These results show that expert-return rankings need not predict supervision quality for VLN exploration, and motivate choosing an expert's abstraction with the student's information in mind rather than by expert return alone.
Figures & tables
| Object-level expert | Room-level expert | VLN student | |
|---|---|---|---|
| Abstraction | map and all target instances | room graph with room types and contents | none; raw RGB |
| Revealed in | all of , from | observed rooms and connectivity, discovered objects, doorway evidence | RGB history and countdown |
| Withheld | — | contents of unentered rooms | everything unseen |
| Planning | deterministic routing, Eq. ( 9 ) | expected return from cross-scene transitions, Eq. ( 11 ) | none; imitates expert actions |
| Decision unit | next target instance | next frontier room | next four primitives |
| Actions depend on unseen targets | yes | no | no |
| Policy | ||||
|---|---|---|---|---|
| (a) Expert policies | ||||
| Greedy frontier expert | 24.4 6.3 | 52.1 7.1 | 85.8 4.6 | 93.8 2.9 |
| Object-level Bellman expert | 91.3 3.4 | 99.8 0.2 | 100.0 0.0 | 100.0 0.0 |
| Room-level Bellman expert (ours) | 45.7 9.8 | 91.6 6.4 | 97.1 4.2 | 98.0 3.5 |
| (b) Learned VLN policies | ||||
| Pretrained VLN | 28.3 6.1 | 35.5 6.2 | 38.8 6.6 | 44.8 7.0 |
| Budget | 50 | 150 | 400 | 2000 |
|---|---|---|---|---|
| Chairs found | ||||
| Mean SR (%) | 24.3 | 35.7 | 54.3 | 65.7 |
Vision-Language Navigation
Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.
See all 329 papers