Forecasting future human motion requires jointly reasoning about an agent's global trajectory and 3D body pose, which are tightly coupled, particularly in team sports where multiple interacting players introduce complex social dynamics. We exploit a socio-temporal encoder that fuses trajectory and pose at the input level and models them jointly through shared factorized attention. The model is further enhanced by a per-joint loss reweighting scheme that emphasizes dynamic joints and prevents degenerate solutions, and a frequency-domain trajectory smoothing term that improves trajectory prediction while indirectly benefiting pose forecasting. To ensure fair comparison with prior joint forecasting methods, we identify and correct evaluation biases arising from overlapping sequence construction, adopting instead a non-overlapping protocol and a deterministic, oracle-free strategy for multi-future prediction. Our approach improves over prior joint forecasting methods on both WorldPose and 3DPW, with the largest gains on pose accuracy, demonstrating that the proposed joint formulation generalizes beyond the densely interactive scenarios that motivate it.
Figures & tables
Figure 1: Overview. Prior trajectories and poses are first projected independently into a shared latent space. The fused representation is processed by a socio-temporal encoder whose output is fed into two task-specific heads implemented as independent MLPs.
Input Modalities
APE ↓
PPE ↓
Pose only
123.0
90.7
Pose + Trajectory
121.8
88.9
Table 1: Effect of trajectory and pose as an auxiliary input on pose/trajectory-only forecasting , evaluated on WorldPose [ 32 ] . The model is trained to predict poses (left) / trajectories (right) exclusively; only the input modalities are varied.
Fusion
Params
ADE ↓
FDE ↓
JPE ↓
APE ↓
PPE ↓
Sequential
2.58M
1074.6
2186.7
1131.9
116.3
85.3
Cross Att
1.45M
951.4
1962.8
1012.7
115.1
84.1
Late Halved
0.74M
850.9
1875.2
907.3
115.2
84.4
Late
2.58M
810.7
1804.9
863.6
114.8
84.2
Early Sum
1.31M
681.2
1663.4
736.8
113.0
83.9
Early Concat
1.32M
670.8
1642.0
723.5
112.6
83.7
Table 2: Fusion topology on WorldPose. All lines share backbone, losses, protocol and training recipe, only the fusion point changes. Rebalancing/smoothing are disabled to isolate fusion.
Joints
Smoothing
ADE ↓
FDE ↓
JPE ↓
APE ↓
PPE ↓
0
✗
670.8
1642.0
723.5
112.6
83.7
2.5
✗
676.3
1651.1
720.2
105.9
81.4
5
✗
683.3
1667.2
719.8
101.3
79.1
7.5
✗
714.6
1694.0
745.6
99.8
78.7
10
✗
848.5
1832.5
874.7
98.7
78.5
0
✓
611.7
1528.1
659.1
106.7
82.6
Table 3: Ablation studies evaluated on WorldPose [ 32 ] . Joint rebalancing factor in the top rows show that larger factors steadily improve pose accuracy at the expense of trajectory accuracy. Trajectory smoothing enabled for the last two rows improves trajectory accuracy directly, and pose accuracy indirectly, by removing unrealistic high-frequency jitter in the predictions
Figure 2: Effect of trajectory smoothing on predicted player paths. Left and central part: In each crop, the solid opaque line shows the observed prior trajectory, the solid semi-transparent line shows the ground-truth future trajectory, and the dashed opaque line shows the predicted path. Left and central representations are without and with the smoothing term enabled, respectively. Right part: Qualitative pose forecasting on WorldPose [ 32 ] test set. Ground-truth poses are shown in blue and predicted poses in red. Each row shows a single player action at five temporally equidistant frames within the 2-second forecasting horizon. Best viewed with zooming.
Method
Futures
WorldPose [ 32 ]
3DPW [ 52 ]
ADE ↓
FDE ↓
JPE ↓
APE ↓
PPE ↓
ADE ↓
FDE ↓
JPE ↓
APE ↓
PPE ↓
StaticBaseline
–
–
–
–
169.8
124.3
–
–
–
108.5
79.5
SimpliHuMoN [ 2 ]
1
757.0
1920.3
803.4
157.3
109.2
613.0
1155.5
693.4
191.6
139.4
SimpliHuMoN [ 2 ]
6
763.0
1988.3
804.4
149.8
106.7
615.6
1161.5
702.2
202.2
174.8
T2P [ 30 ]
1
584.6
1668.0
631.6
142.2
94.4
636.8
1156.9
667.5
109.7
80.5
T2P [ 30 ]
6
564.1
1625.4
613.0
142.1
94.1
650.1
1165.5
682.4
109.1
80.2
Table 4: Quantitative comparison against T2P [ 30 ] , SimpliHuMoN [ 2 ] , and a static baseline on WorldPose [ 32 ] and 3DPW [ 52 ] , with 1 second of observed context and a 2 second forecasting horizon. Futures denotes the number of sampled continuations used at inference for generative baselines, averaged when greater than one. Our model achieves competitive results on WorldPose [ 32 ] and, despite the limited social context available, the best results on 3DPW [ 52 ] .
Figure A1 : Per-joint error under uniform loss weighting (left) and with joint rebalancing (right). Each joint is colored by its mean absolute error on the test set, on a shared scale from 0 (light, low error) to 10 (dark, high error) across both skeletons, so the two panels are directly comparable. The targeted distal joints (elbows, wrists, ankles, and feet), outlined in cyan, are those with consistently above-average error under uniform weighting, and rebalancing visibly reduces their error.
Accurately forecasting the movement of people in complex scenes requires reasoning over the past and present state of the entire environment. In this context, effectively incorporating object information and social interactions into a unified framework remains particularly challenging. To address this, we propose Object-Conditioned Social Diffusion (OCSD), a conditional diffusion model that integrates motion history, multi-person interactions, and object cues into a single framework. OCSD uses an object-conditioning mechanism that modulates denoising at every timestep, enabling fine-grained human-object reasoning, and a social encoder that models the interactions between all humans in the scene. As a result, our model naturally handles varying group sizes, complex social interactions, and supports sampling multiple plausible futures. Extensive experiments show that OCSD achieves state-of-the-art results on the Humans in Kitchens (HiK) and HOI-M3 benchmarks. It reduces the two-second path error by 121.5 mm (31.3%) on HiK and 130.5 mm (33.2%) on HOI-M3 compared to prior work, and produces more realistic long-term forecasts.
Serdar Ozsoy, Lars Doorenbos, Juergen Gall
University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
Human motion prediction combines the tasks of trajectory forecasting and human pose prediction. For each of the two tasks, specialized models have been developed. Combining these models for holistic human motion prediction is non-trivial, and recent methods have struggled to compete on established benchmarks for individual tasks. To address this, we propose a simple yet effective transformer-based model for human motion prediction. The model employs a stack of self-attention modules to effectively capture both spatial dependencies within a pose and temporal relationships across a motion sequence. This simple, streamlined, end-to-end model is sufficiently versatile to handle pose-only, trajectory-only, and combined prediction tasks without task-specific modifications. We demonstrate that this approach achieves state-of-the-art results across all tasks through extensive experiments on a wide range of benchmark datasets, including Human3.6M, AMASS, ETH-UCY, and 3DPW.
Human motion describes the three-dimensional full-body movement of a person. Anticipating such motion holds significant relevance across a wide range of application domains such as human-robot interaction, autonomous driving, animation, and healthcare. In recent research, spatial and temporal dependencies are modeled by bidirectional attention mechanisms. These typically anticipate human motion in an autoregressive manner which could cause an accumulation of errors over time. As a consequence, they solely focus on local pose forecasting. To address these limitations, we propose a non-autoregressive transformer based on spatio-temporal attention, and train it not only for local pose anticipation, but also for global motion prediction in space. Furthermore, to enhance its applicability in real-world scenarios, our model is also trained to recover missing joints due to occlusions, and is capable of processing varying lengths of history observations. Our code is publicly available at https://github.com/Q-Y-Yang/Prediction-of-Local-and-Global-Human-Motion.
Qiaoyue Yang, Sven Heutger, Christopher Niemann +3