Accurately forecasting the movement of people in complex scenes requires reasoning over the past and present state of the entire environment. In this context, effectively incorporating object information and social interactions into a unified framework remains particularly challenging. To address this, we propose Object-Conditioned Social Diffusion (OCSD), a conditional diffusion model that integrates motion history, multi-person interactions, and object cues into a single framework. OCSD uses an object-conditioning mechanism that modulates denoising at every timestep, enabling fine-grained human-object reasoning, and a social encoder that models the interactions between all humans in the scene. As a result, our model naturally handles varying group sizes, complex social interactions, and supports sampling multiple plausible futures. Extensive experiments show that OCSD achieves state-of-the-art results on the Humans in Kitchens (HiK) and HOI-M3 benchmarks. It reduces the two-second path error by 121.5 mm (31.3%) on HiK and 130.5 mm (33.2%) on HOI-M3 compared to prior work, and produces more realistic long-term forecasts.
Figures & tables
Figure 1: Multi-person human motion forecasts in complex scenes with OCSD. We show the last observed frame on the left. The next two panels display forecast poses after 1 second (middle) and 2 seconds (right), with grey skeletons denoting the ground truth and colored skeletons representing the predictions. The gray skeletons are well aligned with the forecast poses, which shows that our model can generate plausible poses in complex scenes by integrating context.
Figure 2: Overview of OCSD. Top: Our method takes a sequence of poses for a dynamic number of people in the scene, along with object information. These inputs are split into the self-stream and two conditioning streams for the objects and social context. From this, OCSD predicts the poses of all people for the next Tout frames. Bottom: Details of our main model, following a U-Net architecture. We integrate the social context for each reference person into the bottleneck, while object information is added throughout the network via FiLM modules.
Dataset
Method
Path Error (mm)
Pose Error (mm)
0.5s
1.0s
1.5s
2.0s
mean
0.5s
1.0s
1.5s
2.0s
mean
HIK [ 42 ]
ContAware [ 26 ]
138.3
251.3
352.4
430.8
239.3
87.8
117.1
136.1
147.8
106.8
GIMO [ 59 ]
143.0
259.7
384.2
487.3
258.6
85.5
121.6
142.0
153.0
109.3
STAG [ 25 ]
124.7
245.4
352.4
479.2
239.7
81.7
110.9
132.5
140.9
100.6
MutualDistance [ 51 ]
128.7
253.2
372.5
479.2
246.0
82.9
117.2
138.5
148.2
105.9
T2P [ 20 ]
88.6
199.6
318.8
447.1
208.7
74.2
108.6
127.5
142.6
96.9
Table 1: Short-term pose forecasting results. Baseline results are taken from [ 39 ] . Our approach consistently reaches the best results.
NDMS ↑
UMWR ↑
2s
4s
6s
8s
10s
MRT [ 47 ]
0.16
0.09
0.08
0.07
0.07
-
HisRep [ 28 ]
0.23
0.12
0.07
0.06
0.06
0.06
SiMLPe [ 17 ]
0.27
0.15
0.07
0.07
0.06
0.06
TriPod [ 1 ]
0.13
0.18
0.14
0.12
0.11
0.11
SAST [ 31 ]
0.17
0.41
0.21
0.15
0.14
0.15
Table 2: Long-term pose forecasting results on HiK. Baseline results are taken from [ 31 ] . Unlike previous approaches, our method performs well even for a 10-second horizon.
Method
Path Error (mm)
Pose Error (mm)
0.5s
1.0s
1.5s
2.0s
mean
0.5s
1.0s
1.5s
2.0s
mean
OCSD
78.9
132.7
197.8
266.9
137.6
69.8
99.4
112.3
121.9
88.4
only Cross-Att.
83.0
138.5
204.5
268.8
141.8
70.3
100.2
113.3
121.6
88.9
only FiLM
87.5
146.3
213.9
284.7
149.6
74.5
102.3
114.3
123.9
91.6
Table 3: Ablating object integration via cross-attention and FiLM. We report the path and pose error on HiK. Our combined mechanism is important to get the best results.
Method
Path Error (mm)
Pose Error (mm)
0.5s
1.0s
1.5s
2.0s
mean
0.5s
1.0s
1.5s
2.0s
mean
OCSD
78.9
132.7
197.8
266.9
137.6
69.8
99.4
112.3
121.9
88.4
self + social
81.6
141.9
213.8
281.5
146.5
69.7
100.3
112.9
124.3
89.1
self + object
83.1
140.1
208.1
277.4
144.7
72.5
102.7
114.9
123.5
90.4
only self
84.5
146.2
217.7
289.8
149.9
70.8
99.2
112.2
122.9
88.5
Table 4: The importance of context for short-term forecasting. We show the path and pose error on HiK. Integrating all forms of context leads to the best performance.
Method
NDMS ↑
UMWR ↑
2s
4s
6s
8s
10s
OCSD
0.34
0.47
0.45
0.45
0.44
0.44
self + social
0.33
0.47
0.45
0.44
0.44
0.44
self + object
0.33
0.47
0.45
0.45
0.44
0.44
only self
0.32
0.48
0.46
0.45
0.45
0.45
Table 5: The impact of context for long-term forecasting. On the HiK dataset, realism score (NDMS) increases with the context while diversity score (UMWR) is lower because of the introduced constraints from environment and people.
Method
Path Error (mm)
Pose Error (mm)
0.5s
1.0s
1.5s
2.0s
mean
0.5s
1.0s
1.5s
2.0s
mean
OCSD
78.9
132.7
197.8
266.9
137.6
69.8
99.4
112.3
121.9
88.4
w/o weighting
82.1
139.8
208.2
277.4
144.8
69.6
101.6
117.3
124.8
90.0
Table 6: Ablating loss weighting. We report the average path and pose error over four samples on HiK. Using the loss weighting improves the results.
Figure 3: Qualitative examples of 10-second predictions for the HiK dataset. We visualize two generated samples for the same input. We show the last observed frame on the left, followed by the final predicted frame from two distinct generated samples. These examples demonstrate the model’s ability to generate diverse predictions, as shown by the purple/yellow person walking to different locations.
Figure 4: 2-second prediction for the HOI-M 3 dataset. We show the last observed frame on the left. The next two panels display forecast poses after 1 second (middle) and 2 seconds (right). Purple skeletons represent our predictions, while gray skeletons represent ground-truth poses. The model correctly continues the stand up motion from the sofa and the walk motion of the other person.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : Qualitative comparison on the HiK dataset (2-second prediction). We compare our predictions to the ground truth (GT). Red/Blue: Past GT poses. Green/Yellow: Predicted future poses. Gray: Future GT for comparison. Purple/Yellow: Predicted future and past GT poses of a person with large motion. The long gray and purple lines show the GT and predicted trajectory, respectively. OCSD’s predictions overlap significantly with the GT (Gray), indicating high accuracy.
Figure S2 : Object interactions on the HiK dataset (2-second prediction). Each row shows the last observed frame (Frame 24), 1-second prediction (Frame 49), and 2-second prediction (Frame 74). Visualization scheme follows Fig. S1 . Row (a) shows the writing activity with interaction with the whiteboard, while row (b) shows the sitting down activity with interaction with the armchair.
Figure S3 : Qualitative comparison with SAST (HiK dataset, 2-second prediction). Each row shows the last observed frame (Frame 24), 1 -second prediction (Frame 49), and 2 -second prediction (Frame 74). Visualization scheme follows Fig. S1 . SAST (row b) does not forecast that the purple person walks from the left group to the right group, while OCSD (row a) successfully predicts the walking motion.
Figure S4 : 2-second prediction on the HOI-M 3 dataset. Purple skeletons represent our predictions, while gray skeletons represent GT. The model accurately captures the pose dynamics over the 2-second horizon.
Figure S5 : Qualitative comparison on the HiK dataset (10-second prediction). We visualize two stochastic samples for the same scene. (a) Last observed GT at t=2 s (frame 49). (b - c) Final predicted frame at t=12 s (frame 299) from two distinct generated samples. Visualization scheme follows Fig. S1 . The purple/yellow person walks to different locations in (b) and (c). This demonstrates the model’s ability to generate diverse predictions.
Figure S6 : Qualitative comparison with SAST (HiK dataset, 10-second prediction). Each row shows the last observed GT at Frame 49 ( t=2 sec), prediction at Frame 174 ( t=7 sec), and prediction at Frame 299 ( t=12 sec). Visualization scheme follows Fig. S1 . For SAST, the purple/yellow person hallucinates a chair and sits down at a location where there is no chair (row b). Our model produces plausible motion for the persons (row a) over the full 10 -second prediction horizon.
Figure S7 : Limitation regarding moving objects. Purple skeletons represent our predictions, while gray skeletons represent GT. Since the future position of the objects are unknown for the model during the prediction interval, predictions deviate from the actual human-object interaction. This is a 2-second prediction on the HOI-M 3 dataset.
Table S1 : Social Encoder ablation on HiK. Replacing mean pooling with cross-attention or integrating social context at multiple U-Net layers instead of the bottleneck increases mean path error (mm) and pose error (mm).
Method
Path Error (mm) ↓
Pose Error (mm) ↓
walking
sink
coffee
walking
sink
coffee
OCSD
306.0
57.3
73.6
100.5
57.3
58.3
self + social
328.3
78.3
94.0
98.6
65.0
62.2
self + object
319.8
68.4
84.9
99.5
65.8
62.2
only self
334.1
81.4
96.0
98.3
63.8
62.4
Appendix
Table S2 : Context effect per activity on HiK. Context improves mean path error for all three representative activities. For pose error, benefits provided by context are only evident in the object-centered social activities sink and coffee.
Backbone
Path Error (mm) ↓
Pose Error (mm) ↓
0.5s
1.0s
1.5s
2.0s
mean
0.5s
1.0s
1.5s
2.0s
mean
U-Net
78.9
132.7
197.8
266.9
137.6
69.8
99.4
112.3
121.9
88.4
DiT
88.5
148.6
223.6
294.3
154.0
74.7
103.2
121.3
133.7
94.5
Appendix
Table S3 : U-Net vs. DiT backbone on HiK. Diffusion transformer performs worse than U-Net.
Method
Foot sliding (mm/s)
Interp.depth (mm)
Interp.rate
MPJVE (mm/s)
GT
219.3
38.5
0.0338
N/A
OCSD
208.2
42.3
0.0343
330.2
Appendix
Table S4 : Additional motion quality metrics on HOI-M 3 . OCSD closely matches ground truth foot sliding and penetration statistics.
Samples
UMWR ↑
2s
4s
6s
8s
10s
4
0.47
0.45
0.45
0.44
0.44
16
0.56
0.48
0.46
0.46
0.45
Appendix
Table S5 : Effect of sample count on diversity (UMWR) on HiK. Diversity scores increase with more samples.
K
Path Error (mm) ↓
Pose Error (mm) ↓
0.5s
1.0s
1.5s
2.0s
mean
0.5s
1.0s
1.5s
2.0s
mean
1
79.0
131.0
200.8
270.9
139.0
69.5
101.8
115.0
121.8
89.0
5
57.5
90.7
134.8
181.4
95.5
54.2
75.0
87.5
93.1
68.3
10
51.8
79.2
116.7
156.1
83.5
50.1
68.3
79.8
85.1
62.5
20
46.5
69.3
103.0
137.4
74.0
47.1
63.3
74.0
79.4
58.3
Appendix
Table S6 : Best-of-K analysis on HiK. Accuracy improves steadily with K. OCSD produces diverse, yet consistently high-quality predictions.
Forecasting future human motion requires jointly reasoning about an agent's global trajectory and 3D body pose, which are tightly coupled, particularly in team sports where multiple interacting players introduce complex social dynamics. We exploit a socio-temporal encoder that fuses trajectory and pose at the input level and models them jointly through shared factorized attention. The model is further enhanced by a per-joint loss reweighting scheme that emphasizes dynamic joints and prevents degenerate solutions, and a frequency-domain trajectory smoothing term that improves trajectory prediction while indirectly benefiting pose forecasting. To ensure fair comparison with prior joint forecasting methods, we identify and correct evaluation biases arising from overlapping sequence construction, adopting instead a non-overlapping protocol and a deterministic, oracle-free strategy for multi-future prediction. Our approach improves over prior joint forecasting methods on both WorldPose and 3DPW, with the largest gains on pose accuracy, demonstrating that the proposed joint formulation generalizes beyond the densely interactive scenarios that motivate it.
Marc Peral, Guillem Capellera, Luis Ferraz +4
Institut de Rob`otica i Inform`atica Industrial, CSIC-UPC · Kognia Sports Intelligence · IT University of Copenhagen
Human motion prediction combines the tasks of trajectory forecasting and human pose prediction. For each of the two tasks, specialized models have been developed. Combining these models for holistic human motion prediction is non-trivial, and recent methods have struggled to compete on established benchmarks for individual tasks. To address this, we propose a simple yet effective transformer-based model for human motion prediction. The model employs a stack of self-attention modules to effectively capture both spatial dependencies within a pose and temporal relationships across a motion sequence. This simple, streamlined, end-to-end model is sufficiently versatile to handle pose-only, trajectory-only, and combined prediction tasks without task-specific modifications. We demonstrate that this approach achieves state-of-the-art results across all tasks through extensive experiments on a wide range of benchmark datasets, including Human3.6M, AMASS, ETH-UCY, and 3DPW.
3D multi-person motion prediction requires modeling both individual kinematics and inter-person interactions. While Flow Matching is effective for multi-hypothesis generation to improve prediction accuracy, directly predicting skeletal sequences from pure noise often compromises structural consistency and introduces unreliable cross-agent interactions during early noise-dominated integration steps. To address this, we propose a Prior-Guided Residual Flow Matching framework. First, a Deterministic Coarse Prior (DCP) establishes a kinematic anchor, formulating the generative process as a conditional flow over motion residuals to simplify the generative objective and preserve structural stability. Second, a Dynamic Cross-Interaction (DCI) mechanism temporally synchronizes inter-agent message-passing with the integration progress, ensuring the extraction of reliable social contexts and improving multi-person motion fidelity. Finally, a decoupled joint-motion architecture with bidirectional fusion effectively preserves fine-grained kinematic coherence. Extensive experiments demonstrate that our approach achieves state-of-the-art prediction accuracy across multiple datasets. Code is available at https://github.com/Wei-Wei-a/Residual-Flow-Matching-with-Dynamic-Cross-Interaction-for-3D-Multi-Person-Motion-Prediction.