EgoMoD: Predicting Global Maps of Dynamics from Local Egocentric Observations
Authors: Iacopo Catalano, David Morilla-Cabello, Jorge Pena-Queralta, Eduardo Montijano
Organizations: I. Catalano is with the University of Turku, Finland. · E. Montijano and D. Morilla-Cabello are with the Instituto de Investigación en Ingeniería de Aragón, Universidad de Zaragoza, Spain. · J. Peña-Queralta is with the Centre for Artificial Intelligence, Zürich University of Applied Sciences, Winterthur, Switzerland.
Efficient navigation in dynamic environments requires anticipating how motion patterns evolve beyond the robot's immediate perceptual range, enabling preemptive rather than purely reactive planning in crowded scenes. Maps of Dynamics (MoDs) offer a structured representation of motion tendencies in space useful for long-term global planning, but constructing them traditionally requires global environment observations over extended periods of time. We introduce EgoMoD, the first approach that learns to predict future MoDs directly from short egocentric video clips collected during robot operation. Our method learns to infer environment-wide motion tendencies from local dynamic cues using a video- and pose-conditioned architecture trained with MoDs computed from external observations as privileged supervision, allowing local observations to serve as predictive signals of global motion structure. Thanks to this, we offer the capacity to forecast future motion dynamics over the whole environment rather than merely extend past patterns in the robot's field of view. As a site-specific dynamic prior, EgoMoD replaces the external global sensing infrastructure required by prior MoD methods at inference time with standard onboard sensors. Experiments in large simulated environments show that EgoMoD predicts future MoDs under limited observability, while evaluation with real images showcases its zero-shot transferability to real systems.
Figures & tables
Fig. 1 : EgoMoD learns to predict global maps of dynamics (bottom) from local video observations (top) from the egocentric vision of the robot, leveraging priors from a privileged expert.
Fig. 2 : EgoMoD architecture overview. A short egocentric video sequence is processed by a frozen foundation video encoder to extract spatio-temporal features. These features are combined with a learned pose embedding via a Transformer encoder, reshaped to a spatial grid and decoded through convolutional and upsampling layers to produce allocentric Maps of Dynamics.
Fig. 3 : Example of construction of our Maps of Dynamics
Parameter
Value
Parameter
Value
Map of Dynamics
Regularization
Direction bins B
8
Input (patch) dropout
0.2
Temporal horizon T
10 / 20 s
Grid cell size
0.30 m
Loss Weights
Optimization
Valid region weight wvalid
5.0
Learning rate
1×10−4
TABLE I : Model, training, and loss hyperparameters.
Fig. 4 : Overview of the hospital simulation environment shown from a top-down view (left) and a perspective view (right).
Horizon
Observability
Method
Flow
Entropy
Direction
ECR
↓ MSE
↓ MAE
↑ SSIM
↓ MSE
↓ MAE
↑ SSIM
↑ Acc.
↑ IoU
↑ ECR
10 s
Local
Mean-MoD
59.26
5.38
0.52
19.26
4.27
0.23
0.70
0.23
0.00
Local-StefMap [ 5 ]
204.86
8.83
0.31
30.66
5.33
0.19
0.29
0.04
−
EgoMoD (ours)
48.43
4.55
0.66
13.00
2.86
0.30
0.44
0.11
0.23
Global
Mean-MoD
63.94
5.65
0.84
19.26
4.27
0.73
0.69
0.26
0.00
Local-StefMap [ 5 ]
185.24
8.35
0.81
33.3
5.74
0.73
0.24
0.02
−
TABLE II : Quantitative comparison on the hospital simulation environment. EgoMoD outperforms both baselines on flow metrics and achieves consistent improvements across evaluation scopes and horizons. Local-StefMap stores only within-FOV predictions, making global ECR unavailable ( − ).
Ablation
Flow
Entropy
Direction
ECR
↓ MAE
↓ MAE
↑ Acc.
↑ ECR
baseline
4.78
2.92
0.42
0.21
vision: no_motion
4.87
3.15
0.41
0.19
vision: motion
4.94
2.67
0.36
0.21
vision: static
5.32
2.28
0.36
0.22
vision: OOD
5.75
3.91
0.24
0.06
TABLE III : Ablation study on the model input.
Fig. 5 : Qualitative results showing predicted flow, entropy, and direction maps from our model on test scenes, compared to ground truth.
Fig. 7 : Qualitative results for the MoD predicted by our network on a real video clip.
Fig. 8 : ECR-based A* planning on EgoMoD’s predicted direction map. Direction- aware costs route A → B with the dominant flow (left) and redirect B → A through an alternative path (right) to avoid moving against it.
This paper proposes a transferable Map of Dynamics (MoD) framework that generalizes to unknown environments using only egocentric 3D LiDAR point clouds to overcome the long-standing limitation of traditional MoD methods. While MoDs are essential for encoding human motion characteristics to enable accurate pedestrian trajectory prediction or safe robot navigation, traditional approaches suffer from site-specificity, requiring exhaustive trajectory accumulation at every new location. Extending recent advances in neural implicit modeling, our framework trains a continuous, LiDAR-based MoD estimator across diverse environments. To mitigate the inherent sparsity and temporal bias of real-world trajectory data, we introduce a position-balanced sampling strategy and a multi-task learning architecture that jointly predicts motion distributions and a spatial frequency score map. The latter is further augmented by visibility-aware losses to compensate for incomplete observation data. Comprehensive experiments demonstrate that our method effectively reconstructs underlying motion maps even in unknown locations from a single instantaneous LiDAR scan, despite highly sparse training data. Finally, we show that our improvements enhance the reliability of downstream trajectory prediction.
Azusa Sawada, Allan Wang, Hideo Saito +1
NEC Corporation · Miraikan - The National Museum of Emerging Science and Innovation · Keio University +1
Safe and efficient robot operation in complex human environments can benefit from good models of site-specific motion patterns. Maps of Dynamics (MoDs) provide such models by encoding statistical motion patterns in a map, but existing representations use discrete spatial sampling and typically require costly offline construction. We propose a continuous spatio-temporal MoD representation based on implicit neural functions that directly map coordinates to the parameters of a Semi-Wrapped Gaussian Mixture Model. This removes the need for discretization and imputation for unevenly sampled regions, enabling smooth generalization across both space and time. Evaluated on two public datasets with real-world people tracking data, our method achieves better accuracy of motion representation and smoother velocity distributions in sparse regions while still being computationally efficient, compared to available baselines. The proposed approach demonstrates a powerful and efficient way of modeling complex human motion patterns and high performance in the trajectory prediction downstream task. The code is publicly available at https://github.com/test-bai-cpu/nemo-map.
Yufei Zhu, Shih-Min Yang, Andrey Rudenko +3
Orebro University, Sweden · Technical University of Munich, Germany · Aalto University, Finland
Modeling the cognitive and experiential factors of human navigation is central to deepening our understanding of human-environment interaction and to enabling safe social navigation and effective assistive wayfinding. Most existing methods focus on forecasting motions in fully observed scenes and often neglect human factors that capture how people feel and respond to space. To address this gap, we propose EgoCogNav, a multimodal egocentric navigation framework that jointly forecasts perceived path uncertainty, trajectories and head motion from egocentric video, gaze, and motion history. To facilitate research in the field, we introduce the Cognition-aware Egocentric Navigation (CEN) dataset consisting of 6 hours real-world egocentric recordings capturing diverse navigation behaviors in real-world scenarios. Experiments show that EgoCogNav learns the perceived uncertainty that strongly correlates with human-like behaviors such as scanning, hesitation, and backtracking while improving trajectory and head-motion forecasting on held-out navigation recordings.
Zhiwen Qiu, Ziang Liu, Wenqian Niu +2
Cornell University, Ithaca, NY, USA · Georgia Institute of Technology, Atlanta, GA, USA