Predicting the short-term future trajectory of a person from RGB-D cameras is a key capability for mobile robots navigating in dynamic and human populated environment. The key challenge to deploying learned predictors on real robots is the train-deploy pipeline gap. Training data are often prepared with an idealized, smoothed ground truth, whereas at inference, the model is fed noisy estimates from pose detectors and depth sensors. The model is therefore evaluated on an input distribution it never encountered during training. Most existing models rely on recurrent neural networks (RNNs) or transformers, which either struggle to capture short-term temporal dependencies or suffer from computational cost. We present TrajMamba, a lightweight, bidirectional state space model for short-term human trajectory prediction using an RGB-D camera. The model uses a single shared processing path for both training and deployment, aligning human pose extraction, depth back projection, and smoothing processes. It encodes a short window of persons's hip center velocities with stacked bidirectional Mamba blocks and decodes future velocities with a GRU head. On the ETH/UCY benchmark, TrajMamba reaches an average ADE of 0.29 m FDE of 0.51 m. With 262K parameters, it runs in real time on modest hardware and is deployed on a TIAGo robot with an onboard NVIDIA Jetson Orin.
Figures & tables
Fig. 1 : Problem setting. A TIAGo PAL robot [ 1 ] mounted RGB-D camera observes a person. MediaPipe extracts the hip center from each frame, which is then back-projected to 3D using depth information. The goal is to estimate the person’s future trajectory from the observed hip positions.
Fig. 2 : Framework structure. Normalized observed velocities pass through a linear layer and several bidirectional Mamba blocks (forward + backward, merged by a learned gate). The last time step (LTS) of the encoder initializes a GRU decoder, whose outputs go through a small MLP to predict future normalized velocities.
Fig. 3 : Preprocessing: From raw RGB-D frames to normalized velocities. Frames are paired by timestamp, MediaPipe locates the hip, and depth gives 3D position. The signal is smoothed (Kalman + EMA), downsampled, and split into windows. Velocities are computed via finite differences and normalized.
Fig. 4 : Inside a Mamba block. The input splits into two branches: one goes through a depthwise convolution and the selective SSM , the other acts as a gate. The SSM uses the input dependent Δ to discretize a fixed HiPPO matrix A , then runs a recurrent scan. The gated product is then projected and added to the input residual.
Fig. 5 : Prediction Visualization on the ETH/UCY benchmark: Each column corresponds to one scene (ETH, UNIV, Zara1, Zara2, Hotel) and each row shows a different sample. Blue: observed history; green star: start; black: ground-truth future; red: best prediction; thin colored curves: the spread of sampled predictions.
Fig. 6 : Real Time prediction across three motion patterns. Top: Person motion right → left of the Robot. Middle: Person motion left → right of the Robot. Bottom: Person motion toward the camera.
Dataset
BR-GAN [ 24 ]
Social-GAN [ 25 ]
CNN [ 11 ]
Trajectron [ 26 ]
Social-STGCNN [ 12 ]
STGAT [ 4 ]
SR-LSTM [ 13 ]
DTGAN [ 27 ]
(Ours)
ETH
0.73 / 1.37
0.81 / 1.52
1.04 / 2.07
0.59 / 1.14
0.64 / 1.11
0.57 / 0.89
0.63 / 1.25
0.68 / 1.43
0.55 / 0.85
UNIV
0.53 / 1.07
0.60 / 1.26
0.57 / 1.21
0.54 / 1.13
0.44 / 0.79
0.49 / 0.96
0.51 / 1.10
0.51 / 1.07
0.31 / 0.58
ZARA1
0.35 / 0.71
0.34 / 0.69
0.43 / 0.90
0.43 / 0.83
0.34 / 0.53
0.29 / 0.59
0.41 / 0.90
0.31 / 0.67
0.22 / 0.38
ZARA2
0.35 / 0.72
0.42 / 0.84
0.34 / 0.75
0.43 / 0.85
0.30 / 0.48
0.24 / 0.54
0.32 / 0.70
0.28 / 0.59
0.17 / 0.31
HOTEL
0.55 / 1.13
0.72 / 1.61
0.59 / 1.27
0.35 / 0.66
0.49 / 0.85
0.41 / 0.86
0.37 / 0.74
0.30 / 0.52
0.20 / 0.43
AVG
0.50 / 1.00
0.58 / 1.18
0.59 / 1.24
0.47 / 0.92
0.44 / 0.75
0.40 / 0.77
0.45 / 0.94
0.42 / 0.85
0.29 / 0.51
TABLE I : Comparison with state of the art methods on ETH/UCY benchmark (ADE / FDE in meters).
Future trajectory prediction of a tracked pedestrian from an egocentric perspective is a key task in areas such as autonomous driving and robot navigation. The challenge of this task lies in the complex dynamic relative motion between the ego-camera and the tracked pedestrian. To address this challenge, we propose an ego-motion-guided trajectory prediction network based on the Mamba model. Firstly, two Mamba models are used as encoders to extract pedestrian motion and ego-motion features from pedestrian movement and ego-vehicle movement, respectively. Then, an ego-motion guided Mamba decoder that explicitly models the relative motion between the pedestrian and the vehicle by integrating pedestrian motion features as historical context with ego-motion features as guiding cues to capture decoded features. Finally, the future trajectory is generated from the decoded features corresponding to the future timestamps. Extensive experiments demonstrate the effectiveness of the proposed model, which achieves state-of-the-art performance on the PIE and JAAD datasets.
Yusheng Peng, Gaofeng Zhang, Liping Zheng
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China · School of Software, Hefei University of Technology, Hefei 230601, China
Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajectories, each paired with a fine-grained natural-language instruction rather than a coarse verb-noun label. We further propose DreamTraj, which predicts a 6-DoF object trajectory from a single RGB image and a task instruction, requiring no video, depth, or CAD model at inference: rather than generating a video, it reads motion from the internal representations of a frozen image-to-video diffusion model at an early denoising step. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states into relative 6-DoF poses. To our knowledge, this is the first approach to directly decode object 6-DoF trajectories from intermediate video diffusion representations rather than generated pixels. DreamTraj sets a new state of the art on both translation and rotation against forecasters that consume multi-frame or privileged inputs, and runs 4.6x faster than generate-then-extract pipelines.
Tongsheng Ding, Zhen Luo, Yixuan Yang +4
1Southern University of Science and Technology · 2Shanghai Innovation Institute · 3Harbin Institute of Technology, Shenzhen +1
Human motion prediction combines the tasks of trajectory forecasting and human pose prediction. For each of the two tasks, specialized models have been developed. Combining these models for holistic human motion prediction is non-trivial, and recent methods have struggled to compete on established benchmarks for individual tasks. To address this, we propose a simple yet effective transformer-based model for human motion prediction. The model employs a stack of self-attention modules to effectively capture both spatial dependencies within a pose and temporal relationships across a motion sequence. This simple, streamlined, end-to-end model is sufficiently versatile to handle pose-only, trajectory-only, and combined prediction tasks without task-specific modifications. We demonstrate that this approach achieves state-of-the-art results across all tasks through extensive experiments on a wide range of benchmark datasets, including Human3.6M, AMASS, ETH-UCY, and 3DPW.