Multi-Person Human Motion Forecasting in Complex Scenes

Serdar Ozsoy; Lars Doorenbos; Juergen Gall

cs.CVAug 27, 2026

Multi-Person Human Motion Forecasting in Complex Scenes

Authors: Serdar Ozsoy, Lars Doorenbos, Juergen Gall

Organizations: University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany

Abstract

Accurately forecasting the movement of people in complex scenes requires reasoning over the past and present state of the entire environment. In this context, effectively incorporating object information and social interactions into a unified framework remains particularly challenging. To address this, we propose Object-Conditioned Social Diffusion (OCSD), a conditional diffusion model that integrates motion history, multi-person interactions, and object cues into a single framework. OCSD uses an object-conditioning mechanism that modulates denoising at every timestep, enabling fine-grained human-object reasoning, and a social encoder that models the interactions between all humans in the scene. As a result, our model naturally handles varying group sizes, complex social interactions, and supports sampling multiple plausible futures. Extensive experiments show that OCSD achieves state-of-the-art results on the Humans in Kitchens (HiK) and HOI-M3 benchmarks. It reduces the two-second path error by 121.5 mm (31.3%) on HiK and 130.5 mm (33.2%) on HOI-M3 compared to prior work, and produces more realistic long-term forecasts.

Figures & tables

Figure 1: Multi-person human motion forecasts in complex scenes with OCSD. We show the last observed frame on the left. The next two panels display forecast poses after 1 second (middle) and 2 seconds (right), with grey skeletons denoting the ground truth and colored skeletons representing the predictions. The gray skeletons are well aligned with the forecast poses, which shows that our model can generate plausible poses in complex scenes by integrating context.

Figure 2: Overview of OCSD. Top: Our method takes a sequence of poses for a dynamic number of people in the scene, along with object information. These inputs are split into the self-stream and two conditioning streams for the objects and social context. From this, OCSD predicts the poses of all people for the next

T_{out}

frames. Bottom: Details of our main model, following a U-Net architecture. We integrate the social context for each reference person into the bottleneck, while object information is added throughout the network via FiLM modules.

Dataset	Method	Path Error (mm)					Pose Error (mm)
		0.5s	1.0s	1.5s	2.0s	mean	0.5s	1.0s	1.5s	2.0s	mean
HIK [ 42 ]	ContAware [ 26 ]	138.3	251.3	352.4	430.8	239.3	87.8	117.1	136.1	147.8	106.8
	GIMO [ 59 ]	143.0	259.7	384.2	487.3	258.6	85.5	121.6	142.0	153.0	109.3
	STAG [ 25 ]	124.7	245.4	352.4	479.2	239.7	81.7	110.9	132.5	140.9	100.6
	MutualDistance [ 51 ]	128.7	253.2	372.5	479.2	246.0	82.9	117.2	138.5	148.2	105.9
	T2P [ 20 ]	88.6	199.6	318.8	447.1	208.7	74.2	108.6	127.5	142.6	96.9

Table 1: Short-term pose forecasting results. Baseline results are taken from [ 39 ] . Our approach consistently reaches the best results.

	NDMS $\uparrow$	UMWR $\uparrow$
		2s	4s	6s	8s	10s
MRT [ 47 ]	0.16	0.09	0.08	0.07	0.07	-
HisRep [ 28 ]	0.23	0.12	0.07	0.06	0.06	0.06
SiMLPe [ 17 ]	0.27	0.15	0.07	0.07	0.06	0.06
TriPod [ 1 ]	0.13	0.18	0.14	0.12	0.11	0.11
SAST [ 31 ]	0.17	0.41	0.21	0.15	0.14	0.15

Table 2: Long-term pose forecasting results on HiK. Baseline results are taken from [ 31 ] . Unlike previous approaches, our method performs well even for a 10-second horizon.

Method	Path Error (mm)					Pose Error (mm)
	0.5s	1.0s	1.5s	2.0s	mean	0.5s	1.0s	1.5s	2.0s	mean
OCSD	78.9	132.7	197.8	266.9	137.6	69.8	99.4	112.3	121.9	88.4
only Cross-Att.	83.0	138.5	204.5	268.8	141.8	70.3	100.2	113.3	121.6	88.9
only FiLM	87.5	146.3	213.9	284.7	149.6	74.5	102.3	114.3	123.9	91.6

Table 3: Ablating object integration via cross-attention and FiLM. We report the path and pose error on HiK. Our combined mechanism is important to get the best results.

Method	Path Error (mm)					Pose Error (mm)
	0.5s	1.0s	1.5s	2.0s	mean	0.5s	1.0s	1.5s	2.0s	mean
OCSD	78.9	132.7	197.8	266.9	137.6	69.8	99.4	112.3	121.9	88.4
self + social	81.6	141.9	213.8	281.5	146.5	69.7	100.3	112.9	124.3	89.1
self + object	83.1	140.1	208.1	277.4	144.7	72.5	102.7	114.9	123.5	90.4
only self	84.5	146.2	217.7	289.8	149.9	70.8	99.2	112.2	122.9	88.5

Table 4: The importance of context for short-term forecasting. We show the path and pose error on HiK. Integrating all forms of context leads to the best performance.

Method	NDMS $\uparrow$	UMWR $\uparrow$
		2s	4s	6s	8s	10s
OCSD	0.34	0.47	0.45	0.45	0.44	0.44
self + social	0.33	0.47	0.45	0.44	0.44	0.44
self + object	0.33	0.47	0.45	0.45	0.44	0.44
only self	0.32	0.48	0.46	0.45	0.45	0.45

Table 5: The impact of context for long-term forecasting. On the HiK dataset, realism score (NDMS) increases with the context while diversity score (UMWR) is lower because of the introduced constraints from environment and people.

Method	Path Error (mm)					Pose Error (mm)
	0.5s	1.0s	1.5s	2.0s	mean	0.5s	1.0s	1.5s	2.0s	mean
OCSD	78.9	132.7	197.8	266.9	137.6	69.8	99.4	112.3	121.9	88.4
w/o weighting	82.1	139.8	208.2	277.4	144.8	69.6	101.6	117.3	124.8	90.0

Table 6: Ablating loss weighting. We report the average path and pose error over four samples on HiK. Using the loss weighting improves the results.

Panel of Figure 3 — Figure 3: Qualitative examples of 10-second predictions for the HiK dataset. We visualize two generated samples for the same input. We show the last observed frame on the left, followed by the final predicted frame from two distinct generated samples. These examples demonstrate the model’s ability to generate diverse predictions, as shown by the purple/yellow person walking to different locations.

Panel of Figure 4 — Figure 4: 2-second prediction for the HOI-M 3 dataset. We show the last observed frame on the left. The next two panels display forecast poses after 1 second (middle) and 2 seconds (right). Purple skeletons represent our predictions, while gray skeletons represent ground-truth poses. The model correctly continues the stand up motion from the sofa and the walk motion of the other person.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Qualitative comparison on the HiK dataset (2-second prediction). We compare our predictions to the ground truth (GT). Red/Blue: Past GT poses. Green/Yellow: Predicted future poses. Gray: Future GT for comparison. Purple/Yellow: Predicted future and past GT poses of a person with large motion. The long gray and purple lines show the GT and predicted trajectory, respectively. OCSD’s predictions overlap significantly with the GT (Gray), indicating high accuracy.

Panel of Figure 12 — Figure S2 : Object interactions on the HiK dataset (2-second prediction). Each row shows the last observed frame (Frame 24), 1-second prediction (Frame 49), and 2-second prediction (Frame 74). Visualization scheme follows Fig. S1 . Row (a) shows the writing activity with interaction with the whiteboard, while row (b) shows the sitting down activity with interaction with the armchair.

Panel of Figure 13 — Figure S3 : Qualitative comparison with SAST (HiK dataset, 2-second prediction). Each row shows the last observed frame (Frame 24), $1$ -second prediction (Frame 49), and $2$ -second prediction (Frame 74). Visualization scheme follows Fig. S1 . SAST (row b) does not forecast that the purple person walks from the left group to the right group, while OCSD (row a) successfully predicts the walking motion.

Panel of Figure 14 — Figure S4 : 2-second prediction on the HOI-M 3 dataset. Purple skeletons represent our predictions, while gray skeletons represent GT. The model accurately captures the pose dynamics over the 2-second horizon.

Panel of Figure 15 — Figure S5 : Qualitative comparison on the HiK dataset (10-second prediction). We visualize two stochastic samples for the same scene. (a) Last observed GT at $t=2$ s (frame 49). (b - c) Final predicted frame at $t=12$ s (frame 299) from two distinct generated samples. Visualization scheme follows Fig. S1 . The purple/yellow person walks to different locations in (b) and (c). This demonstrates the model’s ability to generate diverse predictions.

Panel of Figure 16 — Figure S6 : Qualitative comparison with SAST (HiK dataset, 10-second prediction). Each row shows the last observed GT at Frame 49 ( $t=2$ sec), prediction at Frame 174 ( $t=7$ sec), and prediction at Frame 299 ( $t=12$ sec). Visualization scheme follows Fig. S1 . For SAST, the purple/yellow person hallucinates a chair and sits down at a location where there is no chair (row b). Our model produces plausible motion for the persons (row a) over the full $10$ -second prediction horizon.

Panel of Figure 17 — Figure S7 : Limitation regarding moving objects. Purple skeletons represent our predictions, while gray skeletons represent GT. Since the future position of the objects are unknown for the model during the prediction interval, predictions deviate from the actual human-object interaction. This is a 2-second prediction on the HOI-M 3 dataset.

Table S1 : Social Encoder ablation on HiK. Replacing mean pooling with cross-attention or integrating social context at multiple U-Net layers instead of the bottleneck increases mean path error (mm) and pose error (mm).

Method	Path Error (mm) $\downarrow$			Pose Error (mm) $\downarrow$
	walking	sink	coffee	walking	sink	coffee
OCSD	306.0	57.3	73.6	100.5	57.3	58.3
self + social	328.3	78.3	94.0	98.6	65.0	62.2
self + object	319.8	68.4	84.9	99.5	65.8	62.2
only self	334.1	81.4	96.0	98.3	63.8	62.4

Appendix

Table S2 : Context effect per activity on HiK. Context improves mean path error for all three representative activities. For pose error, benefits provided by context are only evident in the object-centered social activities sink and coffee.

Backbone	Path Error (mm) $\downarrow$					Pose Error (mm) $\downarrow$
	0.5s	1.0s	1.5s	2.0s	mean	0.5s	1.0s	1.5s	2.0s	mean
U-Net	78.9	132.7	197.8	266.9	137.6	69.8	99.4	112.3	121.9	88.4
DiT	88.5	148.6	223.6	294.3	154.0	74.7	103.2	121.3	133.7	94.5

Appendix

Table S3 : U-Net vs. DiT backbone on HiK. Diffusion transformer performs worse than U-Net.

Method	Foot sliding (mm/s)	Interp.depth (mm)	Interp.rate	MPJVE (mm/s)
GT	219.3	38.5	0.0338	N/A
OCSD	208.2	42.3	0.0343	330.2

Appendix

Table S4 : Additional motion quality metrics on HOI-M 3 . OCSD closely matches ground truth foot sliding and penetration statistics.

Samples	UMWR $\uparrow$
	2s	4s	6s	8s	10s
4	0.47	0.45	0.45	0.44	0.44
16	0.56	0.48	0.46	0.46	0.45

Appendix

Table S5 : Effect of sample count on diversity (UMWR) on HiK. Diversity scores increase with more samples.

K	Path Error (mm) $\downarrow$					Pose Error (mm) $\downarrow$
	0.5s	1.0s	1.5s	2.0s	mean	0.5s	1.0s	1.5s	2.0s	mean
1	79.0	131.0	200.8	270.9	139.0	69.5	101.8	115.0	121.8	89.0
5	57.5	90.7	134.8	181.4	95.5	54.2	75.0	87.5	93.1	68.3
10	51.8	79.2	116.7	156.1	83.5	50.1	68.3	79.8	85.1	62.5
20	46.5	69.3	103.0	137.4	74.0	47.1	63.3	74.0	79.4	58.3

Appendix

Table S6 : Best-of-K analysis on HiK. Accuracy improves steadily with K. OCSD produces diverse, yet consistently high-quality predictions.

Human Motion Prediction

Human motion prediction forecasts how people will move, supporting applications from animation and activity understanding to safe robot interaction.

See all 53 papers

Unified Trajectory and 3D Pose Forecasting for Multi-Agent Sports Analysis
Oct 21, 2026Marc Peral, Guillem Capellera, Luis Ferraz +4Human Motion Prediction Multi-Agent Trajectory Prediction
Institut de Rob`otica i Inform`atica Industrial, CSIC-UPC · Kognia Sports Intelligence · IT University of Copenhagen
SimpliHuMoN: Simplifying Human Motion Prediction
Mar 4, 2026Aadya Agrawal, Alexander SchwingHuman Motion Prediction Human Trajectory Prediction
University of Illinois Urbana-Champaign, USA
Residual Flow Matching with Dynamic Cross-Interaction for 3D Multi-Person Motion Prediction
Aug 4, 2026Wei Wei, Yinyuan Zhao, Ruixuan YuFlow Matching Human Motion Prediction
Shandong University

Oct 21, 2026J/K move · Enter open · S save

Unified Trajectory and 3D Pose Forecasting for Multi-Agent Sports Analysis

Marc Peral, Guillem Capellera, Luis Ferraz, Antonio Rubio+3

Institut de Rob`otica i Inform`atica Industrial, CSIC-UPC · Kognia Sports Intelligence · IT University of Copenhagen

Human Motion Prediction Multi-Agent Trajectory Prediction Sports Analytics

Open paper AskPDF arXiv

Forecasting future human motion requires jointly reasoning about an agent's global trajectory and 3D body pose, which are tightly coupled, particularly in team sports where multiple interacting players introduce complex social dynamics. We exploit a socio-temporal encoder that fuses trajectory and pose at the input level and models them jointly through shared factorized attention. The model is further enhanced by a per-joint loss reweighting scheme that emphasizes dynamic joints and prevents degenerate solutions, and a frequency-domain trajectory smoothing term that improves trajectory prediction while indirectly benefiting pose forecasting. To ensure fair comparison with prior joint forecasting methods, we identify and correct evaluation biases arising from overlapping sequence construction, adopting instead a non-overlapping protocol and a deterministic, oracle-free strategy for multi-future prediction. Our approach improves over prior joint forecasting methods on both WorldPose and 3DPW, with the largest gains on pose accuracy, demonstrating that the proposed joint formulation generalizes beyond the densely interactive scenarios that motivate it.

Abstract

Human Motion Prediction

Explore similar work

Unified Trajectory and 3D Pose Forecasting for Multi-Agent Sports Analysis

SimpliHuMoN: Simplifying Human Motion Prediction

Residual Flow Matching with Dynamic Cross-Interaction for 3D Multi-Person Motion Prediction