Comparing Utility of Inertial, Occupancy, Semantic, and Intent Information in Human Motion Prediction During Daily Tasks

Max Burns; Maisha Khanum; Monroe Kennedy; Steven H. Collins

cs.ROSep 29, 2026

Comparing Utility of Inertial, Occupancy, Semantic, and Intent Information in Human Motion Prediction During Daily Tasks

Authors: Max Burns, Maisha Khanum, Monroe Kennedy, Steven H. Collins

Organizations: Department of Mechanical Engineering, Stanford University, Stanford CA, USA.

Abstract

Accurate human motion prediction is crucial for robotic systems operating around people, particularly in complex indoor spaces. In this study, we assess the relative importance of different sources of information in indoor motion prediction with a human motion diffusion model. We collected a dataset of nine naive human subjects conducting simulated indoor daily activities while wearing a pair of Meta Aria glasses. This dataset includes ten buildings from a university campus, and encompasses 238 minutes of navigation between daily tasks. Overall, we demonstrate a 42% improvement beyond a constant velocity baseline. Including body motion, scene representation, and eye gaze fixation data significantly reduced prediction error. Semantic information was found to be useful for indoor motion prediction, but to a lesser degree than in outdoor navigation. Providing explicit intent information reduced error beyond any other addition, suggesting that incorporating explicit intent estimation or user input are fundamental for finer prediction of indoor motion. One of the few indicators of intent, eye gaze fixation, was found to be especially useful in predicting deceleration, and provided basic spatial information to the model in the absence of an occupancy map. These results are a first step towards predicting human motion in highly ambiguous indoor scenarios. Code will be made public upon acceptance. Project page: https://human-motion-diffusion.github.io/

Figures & tables

Fig. 1: Example task set and reconstruction from protocol. Paths taken by the subject are shown in red, tasks are labeled in green with explanations. Example frames of egocentric data used for the 3D reconstruction are included.

Fig. 2: An overview of the data processing and motion prediction approach. (a) Describes the flow of data through the model for each of the multi-modal token sources. (b) The structure of the diffusion noise estimator, which consists of four sequential blocks with self- and cross-attention components. (c) A demonstration of the trajectory diffusion process. Further denoising steps result in smoother, more accurate trajectories.

Fig. 3: Example RGB frames from task-focused navigation protocol. Ten campus buildings were navigated by nine study participants.

ID	Train	Test	Total (min)	Routes	Locations
1	15.2	7.4	22.5	53	A, B, D
2	18.5	5.1	23.6	45	A, J
3	21.8	7.2	28.9	67	A, F, H
4	18.7	7.0	25.7	58	A, B, C
5	24.3	6.9	31.2	64	A, D, E
6	15.4	7.3	22.7	59	A, H, I

TABLE I: Summary of Collected Data

Fig. 4: Layers of the 3D voxel meshes reconstructed from egocentric vision data. Color, semantic labels, and gaze fixation layers are shown. The participant was instructed to “Open the microwave in the kitchenette”.

Fig. 5: Example BEV frames of the kitchenette scene depicted in Fig. 4 , with example diffusion predictions from model AVOSG.

Fig. 6: Relative performance of models with different information levels, across categories of motion, without gaze. Velocity and occupancy information provided the largest relative error reduction, while semantic information provided a smaller benefit. Abbreviations are C - Constant Velocity, A - Acceleration, V - Velocity, O - Occupancy, S - Semantic.

Fig. 8: A comparison of median ADE by trial progress for AVOS and AVOSG. Error reduction due to gaze inclusion is largest closer to the end of the trajectory. The x-axis is normalized by the percentage temporal progress through the trial. Error bars are standard deviation of ADE.

Model	All $\downarrow$	Steady $\downarrow$	Turn $\downarrow$	Accel. $\downarrow$	Decel. $\downarrow$
C	0.77	0.34	1.17	1.47	0.92
AVOSG	0.45	0.31	0.60	0.87	0.51
+ Goal	0.32	0.26	0.42	0.83	0.17
+ A*	0.26	0.23	0.30	0.63	0.18

TABLE II: Ablation Results, Known Goal. Median ADE reported (m)

Fig. 9: The impact of including explicit goal information in the best-performing ablated model (AVOSG). Goal information provides large benefits in turning and deceleration, but has limited impact on acceleration prediction accuracy.

Model	All $\downarrow$	Steady $\downarrow$	Turn $\downarrow$	Accel. $\downarrow$	Decel. $\downarrow$
No Holdout	0.61	0.29	0.84	0.95	0.67
w/ Holdout	0.64	0.32	0.85	0.98	0.76

TABLE III: Ablation Results, Subject Holdout. Median ADE reported (m)

Explore similar work

Ego-Human Motion Prediction with 3D-Aware LLM
Jul 8, 2026Yujin Bae, Jaewoo Jeong, Hyeonseong Kim +1Human Motion Prediction Egocentric Vision
Visual Intelligence Lab., KAIST, Korea
Long-Term Prediction of Local and Global Human Motion with Occlusion Recovery
Jun 26, 2026Qiaoyue Yang, Sven Heutger, Christopher Niemann +3Human Motion Prediction Human Motion
Neuro-Information Technology, Otto-von-Guericke-University Magdeburg, Magdeburg, 39106, Germany
Recall to Predict: Grounding Motion Forecasting in Interpretable Motion Bank
May 2, 2026Abhishek Vivekanandan, Ahmed Abouelazm, J. Marius ZöllnerHuman Motion Prediction Motion
FZI Forschungszentrum Informatik · Karlsruhe Institute of Technology (KIT) Karlsruhe, Germany

Jul 8, 2026, cs.CVJ/K move · Enter open · S save

Ego-Human Motion Prediction with 3D-Aware LLM

Yujin Bae, Jaewoo Jeong, Hyeonseong Kim, Kuk-Jin Yoon

Visual Intelligence Lab., KAIST, Korea

Human Motion Prediction Egocentric Vision Motion Cross-Modal

Open paper AskPDF arXiv

Anticipating human motion from an egocentric perspective is fundamental for proactive assistance in AR/VR, human-robot collaboration, and embodied AI. While recent works incorporate language as a semantic prior to reduce the ill-posed nature of egocentric forecasting, they largely neglect the 3D spatial and semantic context that governs how motion unfolds, and treat pose and language prediction as separate inference streams. We introduce Ego3DLM, built on two core principles: accurate motion forecasting requires explicit spatial and semantic understanding of the 3D environment, and pose and language must be predicted holistically in a single pass, since motion is inherently tied to the semantic interpretation of actions being performed. Given three-point tracking, 3D scene features, and egocentric video, Ego3DLM simultaneously decodes past pose, future pose, past narration, and future narration in a single autoregressive pass, grounding predicted poses and descriptions in one another to enforce cross-modal and temporal consistency. We adopt a three-stage training scheme: (1) spatial-semantic scene awareness pretraining; (2) holistic instruction tuning over all four outputs in a single pass; and (3) GRPO-based reinforcement finetuning with intra- and inter-modal rewards that directly optimize pose-language fidelity. Experiments on the Nymeria benchmark demonstrate that Ego3DLM achieves state-of-the-art performance across future motion prediction, past motion tracking, and motion description, showing that 3D scene grounding and holistic cross-modal prediction yield physically plausible and semantically coherent motion forecasts. The project page is available at https://jaewoo97.github.io/Ego3DLM/.

Abstract

Explore similar work

Ego-Human Motion Prediction with 3D-Aware LLM

Long-Term Prediction of Local and Global Human Motion with Occlusion Recovery

Recall to Predict: Grounding Motion Forecasting in Interpretable Motion Bank