Existing language-conditioned navigation systems typically rely on modular pipelines or trajectory generators, but the latter use each scene--instruction annotation mainly to supervise one start-conditioned rollout. To address these limitations, we present CoFL, an end-to-end policy that maps a bird's-eye view (BEV) observation and a language instruction to a continuous flow field for navigation. CoFL reformulates navigation as workspace-conditioned field learning rather than start-conditioned trajectory prediction: it learns local motion vectors at arbitrary BEV locations, turning each scene--instruction annotation into dense spatial control supervision. Trajectories are generated from any start by numerical integration of the predicted field, enabling simple real-time rollout and closed-loop recovery. To enable large-scale training and evaluation, we build a dataset of over 500k BEV image--instruction pairs, each procedurally annotated with a flow field and a trajectory derived from semantic maps built on Matterport3D and ScanNet. Evaluating on strictly unseen scenes, CoFL significantly outperforms modular Vision-Language Model (VLM)-based planners and trajectory generation policies in both navigation precision and safety, while maintaining real-time inference. Finally, we deploy CoFL zero-shot in real-world experiments with BEV observations across multiple layouts, maintaining feasible closed-loop control and a high success rate.
Figures & tables
Symbol
Domain
Description
Problem Setup
E
[0,1]2
Normalized BEV-aligned workspace.
Xfree,Xobs
⊆E
Free-space and obstacle subsets.
I
RH×W×3
RGB BEV input (top-down/high-oblique).
ℓ
text
Language instruction.
x0,τ
Xfree,[0,1]→E
Start state and trajectory.
TABLE I : Notation used in this paper.
Fig. 2 : Overview of the CoFL’s network architecture. Given an RGB BEV observation I and a language instruction ℓ , a SigLIP 2-based [ 34 , 35 ] vision–language encoder produces language-conditioned context tokens over the BEV image. The decoder then queries this context with 2D normalized spatial coordinates X and outputs the corresponding velocities V , forming a continuous flow field v(x∣I,ℓ) via bilinear interpolation of discrete velocity predictions.
Fig. 3 : Overview of the trajectory inference. The predicted flow field over the workspace guides agents from different starts toward the same goal while smoothly avoiding obstacles.
Fig. 4 : Overview of the proposed visual observation generation pipeline. Images are captured from a multi-view camera array. (a)-(b) Top-down view RGB and semantic segmentation maps. (c)-(d) Oblique view sample.
Dataset
# Scenes
# Regions
# Views
# Samples
Matterport3D [ 7 ]
90
2,056
9
245,604
ScanNet [ 8 ]
1,513
1,513
342,038
TABLE II : Statistics of the Implemented Datasets.
Fig. 5 : Overview of the procedural annotation pipeline. We combine cost-weighted geodesic distance and obstacle repulsion to derive the field and trajectory annotations.
Fig. 6 : Overview of the training/validation split. We employ a strict scene-wise split to assess generalization.
Fig. 7 : Examples of trajectories on validation set. (a) Matterport3D: “Please navigate to the sofa in the upper right” ; (b) ScanNet: “Please go to the toilet” . From left to right: ground truth, pure VLM/VLM+Planner, DP-family and CoFL. More examples are provided in Appendix D .
TABLE III : Navigation performance and field quality. We report trajectory metrics (detailed in § V-A2 ) aggregated over the validation set for all baselines and CoFL. Field metrics are reported for CoFL only.
Fig. 8 : Latency–performance trade-offs of real-time models (DP-family and CoFL). DP-family: {1, 2, 3, 5 , 7, 10, 15, 20} denoising steps (horizon 100); CoFL: {50, 75, 100 , 125, 150, 175, 200} grid size (100-step Euler rollout). Underlines mark the operating points reported in Table III .
L
Encoder param (M)
FGE ↓
CR ↓
AE ↓
ME ↓
Lat. (ms)
2
395.28
0.14
0.22
18.73 ∘
0.06
15.08
4
414.18
0.13
0.22
18.16 ∘
0.06
15.81
6
433.08
0.13
0.24
18.76 ∘
0.06
16.21
TABLE IV : Encoder Fusion Depth
L~
Head param (M)
FGE ↓
CR ↓
AE ↓
ME ↓
Lat. (ms)
1
8.27
0.14
0.23
18.55 ∘
0.06
12.41
2
15.36
0.13
0.22
18.16 ∘
0.06
15.81
3
22.45
0.14
0.22
18.01 ∘
0.06
20.69
TABLE V : Decoder Query Depth
d
Overall param (M)
FGE ↓
CR ↓
AE ↓
ME ↓
Lat. (ms)
512
399.63
0.13
0.26
18.67 ∘
0.07
12.29
768
429.54
0.13
0.22
18.16 ∘
0.06
15.81
1024
471.25
0.13
0.22
17.90 ∘
0.06
22.55
TABLE VI : Hidden Dimension
Variant
Head param (M)
FGE ↓
CR ↓
AE ↓
ME ↓
Lat. (ms)
W/O
15.36
0.13
0.22
17.89 ∘
0.07
15.93
With
20.09
0.14
0.23
18.30 ∘
0.07
49.34
TABLE VII : Self-Attention in CoFL Decoder
Variant
FGE ↓
CR ↓
Curv
PLR
Stabilized rescaling ( (1−tk)+βtkαv )
0.13
0.22
0.11
0.87
No stabilizer ( 1−tkv )
0.75
1.00
0.18
4.57
Unit-speed rollout ( v/∥v∥ )
0.12
0.57
1.68
3.70
TABLE VIII : Ablation of rollout parameterization.
Fig. 9 : Example of query distributions (in yellow) for the supervision coverage ablation. Workspace supervises the full BEV domain, Tube supervises a corridor around the reference path, and On-trajectory supervises only the demonstrated trajectory.
Variant
FGE ↓
CR ↓
Curv
PLR
AE ↓
ME ↓
Full workspace
0.13
0.22
0.11
0.87
18.73 ∘
0.06
Tube
0.12
0.61
1.66
3.70
24.14 ∘
0.08
On-trajectory
0.34
0.91
0.52
2.48
71.71 ∘
0.46
DP-T-DDPM
0.16
0.78
0.08
0.91
–
–
TABLE IX : Ablation of supervision coverage. All variants use the same model and query budget, differing only in where training queries are sampled.
Fig. 10 : Example trajectories from real-world experiments. (a)–(d) E1 : from left to right, instructions are “ move to the right side of the chair ”, “ move to the left side of the table ”, “ move to the back side of the box ”, and “ move to the front side of the right chair ”. (e)–(h) E2 : three static layouts and one reactive adversary, all with the same instruction “ move to the chair ”. Trajectories are replanned at each control step from the latest observation and visualized in the figure.
Setting
Case
On-Target ↑
CR ↓
PN (m)
CLR (m)
TTS (s)
PL (m)
E1: W/O Obstacles
C1: Target = chair
1.00
–
–
–
14.13
2.83
C2: Target = table
0.80
–
–
–
13.37
2.79
C3: Target = box
0.60
–
–
–
33.88
4.79
C4: Target = chair pair (language disambiguation)
1.00
–
–
–
18.33
4.04
Avg
0.85
–
–
–
18.67
3.52
E2: With Obstacles
C5: Target = chair , Static layout A
1.00
0.20
0.12
0.04
22.69
3.78
TABLE X : Real-world target navigation under two settings. Each setting contains four task cases (C1–C8; 5 trials per case). PN denotes penetration depth in collided trials, while CLR denotes the minimum obstacle clearance in non-collision trials.
Fig. 11 : Four collision trials drawn from E2 as examples. Trajectories are colored by elapsed time from start to end. The static obstacle is shown with diagonal hatching, and the moving adversary is shown with a dashed orange outline. The deepest-penetration frame is highlighted by the red robot outline, with the penetration depth annotated.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Encoder
Trajectory denoiser
Sampler
DP-C-DDPM
Vision–Language Encoder same as CoFL
Temporal U-Net (1D Conv; FiLM)
DDPM (stochastic)
DP-T-DDPM
Vision–Language Encoder same as CoFL
Transformer (token-conditioning)
DDPM (stochastic)
DP-C-Flow/ODE
Vision–Language Encoder same as CoFL
Temporal U-Net (1D Conv; FiLM)
ODE (deterministic)
Appendix
TABLE XI : DP-family baselines used in this work . All variants reuse the same SigLIP-based vision–language encoder and differ in the denoiser architecture and sampler.
Fig. 12 : More examples of trajectories on validation set. From left to right: ground truth, pure VLM/VLM+Planner, DP-family and CoFL. (a-b) ScanNet: “please walk to the door” and “can you approach the bathtub” ; (c-d) ScanNet: “go to ahead of the table” and “can you approach right side of the table” ; (e-f) Matterport3D: “can you proceed to back of the chair in the lower right” and “please travel to the chair in the lower left” .
Fig. 13 : Effect of pseudo receding-horizon segmentation n on DP (DP-T-DDPM, 5 denoising steps, fixed seed 42). The red horizontal line indicates CoFL with a fixed 100×100 query grid.
Vision-Language Navigation has increasingly emphasized high-level instruction reasoning, memory, global map construction, and instruction decomposition, while the low-level action representation remains comparatively underexplored. We propose CoFL-S, a low-level vision-language-action framework that predicts a language-conditioned flow field over the robot's local visible sector and generates continuous trajectories by rolling out the predicted field. To train this low-level representation, we convert each VLN-CE episode, originally a whole-episode instruction paired with an action sequence, into frame-level local supervision with aligned sub-instructions and matched action, trajectory, and dense flow-field targets. For evaluation, we introduce a continuous-time Habitat benchmark that isolates low-level action interfaces from instruction decomposition and executes all methods through a shared velocity-command controller, enabling decomposition-independent closed-loop comparison across different planner frequencies rather than fixed discrete forward-and-turn transitions in VLN-CE. Under matched encoders and training settings, CoFL-S consistently outperforms action-token and action-chunk baselines across planner frequencies in the continuous-time Habitat benchmark, and zero-shot real-world closed-loop deployment further shows its advantage over both baselines beyond simulation.
Haokun Liu, Zhaoqi Ma, Yicheng Chen +5
Dragon Lab, Department of Mechanical Engineering, The University of Tokyo
Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires agents to follow natural-language instructions in unseen scenes. While Large Models (LMs) have advanced VLN-CE, their performance remains severely degraded by real-world visual corruptions, a critical yet underexplored domain constraint. We introduce Temporal Conditional Flow Decorruptor (FlowDec), a novel image restoration framework tailored for LM-based VLN-CE. FlowDec integrates a hybrid temporal conditioning strategy to align the generative flow path with historical context and employs action-centroid guided filtering to dynamically assess and integrate outputs. Extensive experiments demonstrate that FlowDec outperforms state-of-the-art decorruption methods in both navigation accuracy and generation latency. Our approach establishes a robust, efficient paradigm for resilient embodied navigation in unpredictable real-world conditions.
Yufei Zhang, Changhao Chen
The Hong Kong University of Science and Technology (Guangzhou), China
Recent navigation systems achieve strong benchmark results, yet real-world deployment often remains visibly stop-and-go. This bottleneck arises because the sense-inference-execution loop is still blocking: after each new observation, the controller must wait for sensing, transmission, and inference before motion can continue. Reducing action-generation cost alone therefore does not remove redundant waiting. To address this issue, we present LiveVLN, a training-free framework for more continuous embodied navigation by augmenting pretrained VLM navigators with multi-step action continuation. Instead of pausing for each full sense-and-inference round, LiveVLN overlaps execution with the processing of newly arrived observations, allowing refreshed future actions to be handed off before the current executable prefix is exhausted. This design keeps actions continuously available during motion, reducing idle waiting and enabling smoother online execution. The framework operates at runtime and can be integrated with compatible pretrained VLM navigators. Across R2R and RxR, LiveVLN preserves benchmark performance while reducing waiting time and improving action availability. In real-world deployments, it cuts average episode waiting time by up to 77.7% and shortens wall-clock episode time by 12.6% on StreamVLN and 19.6% on NaVIDA, yielding more coherent execution during deployment. Code is available at https://github.com/NIneeeeeem/LiveVLN.
Xiangchen Wang, Weiye Zhu, Teng Wang +5
Southern University of Science and Technology · Harbin Institute of Technology, Shenzhen · Spatialtemporal AI