Modeling realistic pedestrian trajectories requires accounting for both social interactions and environmental context, yet most existing approaches largely emphasize social dynamics. We propose EnvSocial-Diff: a diffusion-based crowd simulation model informed by social physics and augmented with environmental conditioning and individual--group interaction. Our structured environmental conditioning module explicitly encodes obstacles, objects of interest, and lighting levels, providing interpretable signals that capture scene constraints and attractors. In parallel, the individual--group interaction module goes beyond individual-level modeling by capturing both fine-grained interpersonal relations and group-level conformity through a graph-based design. Experiments on multiple benchmark datasets demonstrate that EnvSocial-Diff outperforms the latest state-of-the-art methods, underscoring the importance of explicit environmental conditioning and multi-level social interaction for realistic crowd simulation. Code is here: https://github.com/zqyq/EnvSocial-Diff.
Figures & tables
Figure 1: Environmental factors are important in crowd simulation. The target pedestrian is influenced by nearby neighbors, obstacles, objects of interest (OOI), and lighting conditions. The scene image is divided into grids to calculate lighting information.
Symbol
Meaning
Symbol
Meaning
i,j
Pedestrian indices
t
Time step index
Sit
State of i at t , [pit,vit,ait]
H
Prediction horizon (number of future steps)
pit
Position of i at time t
vit
Velocity of i at time t
ait
Acceleration of i at time t
fθ
Denoising network in acceleration space
Fidest
Destination-driving force of i
cit
Conditioning signals [Fienv⊕Fisocial⊕Fihist]
Fienv
Environment-induced force on i
Fisocial
Social interaction force on i
Table 1: Summary of main notations used in EnvSocial-Diff.
Figure 2: EnvSocial-Diff pipeline. Pedestrian motion is modeled as in the Social Force Model (SFM), where the destination force Fidest is applied outside the diffusion process to preserve long-term intent. The conditioning signals cit=[Fienv⊕Fisocial⊕Fihist] aggregate three interactive components: (1) Environmental Conditioning — obstacle and OOI features are encoded via cross-attention with pedestrians, while lighting features are extracted from grid-based scene brightness; (2) Individual–Group Interactions — GNNs encode individual-level (sim ij1 , sim ij2 ), group-level (sim i3 ), and relative motion ( rij ) to produce the social force Fisocial ; and (3) Historical Trajectories — short-term motion trends are encoded from recent states using an LSTM. Given cit and Gaussian noise ϵ∼N(0,1) , the denoiser fθ performs reverse diffusion to recover clean accelerations y^i,0t , which are then combined with the destination force to yield the final prediction a^it .
Group
Models
GC
UCY
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
Physics-based
CA
2.7080
5.4990
-
0.0620
-
1492
8.3360
79.4200
-
2.0220
-
4504
SFM
1.2590
2.1140
-
0.0150
-
622
2.5390
6.5710
-
0.1290
-
434
Data-driven
STGCNN
8.1608
15.8372
-
0.5296
5.1438
2076
7.5121
18.7721
-
0.5149
5.1695
1348
PECNet
2.0669
4.3054
-
0.0397
0.7431
1142
3.9674
16.1412
-
0.1504
2.0986
1348
MID
8.4257
35.1797
-
0.3737
4.2773
1620
8.2915
47.8711
-
0.4384
4.7109
1076
Table 2: Quantitative comparison on GC and UCY datasets. Results for baselines are directly reported from SPDiff ( Chen et al., 2024 ) , except for E- V2 -SC and Ours , which are reproduced under the same experimental settings. Here, Ours corresponds to our proposed EnvSocial-Diff model.
Figure 3: Comparison with baselines on UCY. (A) Predicted trajectories: our method (cyan) follows the ground truth (blue) more closely than SFM (magenta) and SPDiff (orange). (B) Error curves over time: our method consistently achieves lower MAE and OT, especially at longer horizons.
Model
Environment
GC
UCY
Obs
OOI
Light
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
SPDiff
✓
✗
✗
0.9718
1.5450
0.9538
0.0100
0.3418
942
1.8853
4.2221
1.9000
0.0699
0.7496
634
✓
✗
✓
0.9359
1.4345
0.9404
0.0099
0.3395
958
1.8395
3.8602
1.9463
0.0669
0.7357
622
Ours
✗
✗
✗
0.9127
1.3909
0.9246
0.0087
0.3261
946
1.8597
3.8945
1.9948
0.0626
0.7353
738
✓
✗
✗
0.8990
1.3727
0.9162
0.0087
0.3253
1000
1.8337
3.8550
1.9987
0.0604
0.7259
730
✓
✓
✗
0.8873
1.3455
0.9037
0.0087
0.3279
910
1.8271
3.8541
1.9671
0.0586
0.7212
648
Table 3: Ablation study on structured environmental factors. The first two rows report results of the baseline SPDiff ( Chen et al., 2024 ) and its variant extended with an explicit lighting module (SPDiff+Lighting). The lower block corresponds to our proposed EnvSocial-Diff (Ours), where obstacles, objects of interest (OOI), and lighting are progressively added. Results are reported on GC and UCY datasets across six metrics, with checkmarks indicating the included factors.
Model Variant
GC
UCY
rij
simij1
simij2
simi3
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
✓
✗
✗
✗
0.9066
1.3897
0.9192
0.0093
0.3272
990
1.9055
4.0101
2.0525
0.0628
0.7730
580
✓
✓
✗
✗
0.8946
1.3570
0.9194
0.0086
0.3303
1000
1.8846
3.8502
2.0139
0.0588
0.7720
752
✓
✓
✓
✗
0.9208
1.4137
0.9242
0.0087
0.3404
1024
1.8725
3.7937
2.0262
0.0575
0.7761
614
✓
✓
✓
✓
0.8861
1.3339
0.8997
0.0087
0.3269
906
1.8182
3.7292
1.8656
0.0598
0.7249
522
Table 4: Ablation on the IGI module. Starting from relative motion rij , we incrementally add individual-level similarities ( simij1 : approach tendency; simij2 : motion alignment) and the group-level similarity ( simi3 : group conformity). Results on GC and UCY show consistent gains on most metrics, with the full configuration yielding the strongest overall performance.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Comparison between the original GC subregion and the full GC scene. The left image highlights the cropped subarea (blue box) used in prior work, which limits spatial and interaction diversity. The right image shows the complete GC scene, covering a broader area with higher pedestrian density and environmental complexity, used in our extended evaluation.
Dataset: Full GC
Method
Social
Env
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
COL ↓
SPDiff
2.4624
4.6824
2.6707
0.0044
0.8873
1910
Ours
✓
✗
2.4478
4.4662
2.6683
0.0044
0.8836
1750
✓
✓
2.3527
4.1334
2.5891
0.0039
0.8473
1710
Appendix
Table 5: Comparison on the full GC dataset. SPDiff uses its own social interaction module and a basic environment treatment limited to obstacle repulsion. In contrast, our method replaces the social interaction module with the proposed IGI design (✓) and additionally integrates richer environmental cues (✓), including obstacles, objects of interest, and lighting. ✗ indicates the corresponding module is not used. In the table, Social and Env are marked with ‘*‘ for SPDiff’s built-in designs (social interaction and obstacle-only environment), ✓ when replaced by our modules, and ✗ when omitted.
Figure 5: Qualitative comparison between the GT, Ours, and SPDiff. (A) Near obstacles, SPDiff trajectories maybe to pass much closer to obstacles, whereas both GT and our method keep a more reasonable distance. (B) In crowded regions, SPDiff produces several near-collision interactions, while our predictions remain smoother and more socially consistent. (C)
Figure 6: Additional qualitative results for multiple target pedestrians. Each subplot visualizes one scene with a selected target pedestrian (green triangle). Ground-truth future trajectory (GT) is shown in blue, while predictions from SFM (magenta), SPDiff (orange), and our method (cyan) are overlaid. Our approach consistently aligns more closely with the GT across diverse scenarios.
Env
GC
UCY
Obs
OOI
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
✗
✗
0.9320
1.4425
0.9391
0.0094
0.3362
926
1.9320
3.9726
2.0367
0.0609
0.7706
550
✗
✓
0.9038
1.3645
0.9164
0.0090
0.3333
898
1.9119
3.7120
2.2160
0.0606
0.7563
688
✓
✗
0.8960
1.3562
0.9160
0.0086
0.3277
962
1.8346
4.0420
1.9951
0.0634
0.7042
710
✓
✓
0.8861
1.3339
0.8997
0.0087
0.3269
906
1.8182
3.7292
1.8626
0.0598
0.7249
522
Appendix
Table 6: Ablation on enhanced environmental features. We compare four settings: no enhancement (✗, ✗), OOI-only enhancement (✗, ✓), obstacle-only enhancement (✓, ✗), and joint enhancement with both (✓, ✓). Results on GC and UCY datasets show that the joint enhancement strategy achieves the most consistent improvements across metrics.
Dataset: UCY Students03
obstacle
lighting
OOI
MAE ↓
MMD ↓
OT ↓
COL ↓
✗
✗
✗
2.8943
7.9564
0.0954
308
✓
✗
✗
2.7822
7.3759
0.0899
266
✓
✓
✗
2.7324
7.2187
0.0803
216
✓
✓
✓
2.6742
6.7032
0.0708
216
Appendix
Table 7: Ablation study of adding environmental factors to the SFM method on UCY Students03 . ✓ / ✗ indicate whether the corresponding environmental module is enabled. Progressive addition of obstacle, lighting, and object-of-interest (OOI) cues leads to consistent improvements across all evaluation metrics.
Dataset: UCY Zara1
Obstacle
Lighting
OOI
MAE ↓
MMD ↓
OT ↓
✗
✗
✗
2.5954
9.2648
1.6676
✓
✗
✗
1.9585
5.2413
1.1437
✓
✓
✗
1.8981
5.1045
1.3224
✓
✓
✓
1.8282
4.7196
1.0968
Appendix
Table 8: Ablation study of adding environmental factors to the SFM method on UCY Zara1 . ✓ / ✗ indicate whether the corresponding environmental module is enabled. The results show that incorporating obstacle, lighting, and object-of-interest (OOI) cues progressively improve trajectory prediction performance in terms of MAE, MMD, and OT.
Figure 7: Qualitative comparisons on the GC dataset. Each subplot shows the predicted trajectory of a target pedestrian within the marked evaluation area (blue rectangle). Ground-truth (GT) future trajectories are depicted in black, while predictions from SPDiff and our method are shown in orange and cyan, respectively. Our approach produces more accurate and socially plausible predictions, particularly in scenarios involving sharp turns, long-range navigation, or subtle environmental conditioning.
GC Dataset
UCY Dataset
Top_k
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
2
0.8918
1.3409
0.9088
0.0086
0.3273
902
1.8149
3.6431
1.9209
0.0596
0.7251
588
4
0.8896
1.3321
0.8998
0.0087
0.3284
914
1.8142
3.7501
1.9542
0.0601
0.7007
552
6 (Ours)
0.8861
1.3339
0.8997
0.0087
0.3269
906
1.8182
3.7292
1.8656
0.0598
0.7249
522
8
0.8921
1.3599
0.9128
0.0089
0.3384
953
1.8154
3.7276
1.8999
0.0595
0.7159
476
Appendix
Table 9: Sensitivity analysis of the hyperparameter Top k on GC and UCY datasets. Top k controls how many nearest neighbors are used when computing individual-level similarities within the IGI module. All metrics follow the lower-is-better convention ( ↓ ). We adopt k=6 as it provides the most balanced overall performance.
GC Dataset
UCY Dataset
Step
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
50
0.8870
1.3361
0.9019
0.0087
0.3269
984
1.8173
3.7486
1.8596
0.0616
0.7148
486
60
0.8860
1.3355
0.9006
0.0087
0.3273
912
1.8142
3.7358
1.8688
0.0603
0.7124
432
70(Ours)
0.8861
1.3339
0.8997
0.0087
0.3269
906
1.8182
3.7292
1.8656
0.0598
0.7249
522
80
0.8862
1.3449
0.8999
0.0088
0.3285
908
1.8214
3.8168
1.8705
0.0612
0.7118
582
Appendix
Table 10: Sensitivity analysis of the diffusion step count on GC and UCY datasets. The diffusion step controls the number of denoising iterations during sampling. All metrics follow the lower-is-better convention ( ↓ ). We adopt 70 steps as the default, as it provides a stable and well-balanced performance across all evaluation metrics.
GC Dataset
UCY Dataset
Grid_Size
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
Grid_Size
MAE ↓
OT ↓↓
FDE ↓
MMD ↓
DTW ↓
Col ↓
220(Ours)
0.8861
1.3339
0.8997
0.0087
0.3269
906
90
1.8277
3.8858
1.8794
0.0608
0.7188
546
250
0.8842
1.3308
0.8999
0.0087
0.3275
914
100
1.8470
3.9977
1.9471
0.0629
0.7254
596
300
0.8848
1.3393
0.8986
0.0086
0.3276
916
110(Ours)
1.8182
3.7292
1.8656
0.0598
0.7249
522
400
0.8840
1.3396
0.9000
0.0086
0.3276
918
120
1.8314
3.8239
1.9613
0.0600
0.7196
561
Appendix
Table 11: Sensitivity analysis of the grid size used to construct the global scene representation on GC and UCY datasets. The grid size controls the spatial resolution of the environmental encoding, with lower-is-better metrics ( ↓ ). We adopt a grid size of 220 for GC and 110 for UCY, as these settings yield the most balanced performance across the evaluation metrics for each dataset.
Dataset
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
COL ↓
GC (repulsive)
0.8970
1.3827
0.8974
0.0087
0.3370
894.0
GC (ours)
0.8861
1.3339
0.8997
0.0087
0.3269
906
UCY (repulsive)
1.8609
3.8766
1.9302
0.0646
0.7066
510.0
UCY (ours)
1.8182
3.7292
1.8656
0.0598
0.7249
522
Appendix
Table 12: Comparison between our method and the repulsive-force variant.
Dataset
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
COL ↓
GC (variant)
0.8862
1.3449
0.8999
0.0088
0.3284
908.0
GC (ours)
0.8861
1.3339
0.8997
0.0087
0.3269
906
UCY (variant)
2.0764
4.4494
2.1363
0.0730
0.7854
642.0
UCY (ours)
1.8182
3.7292
1.8656
0.0598
0.7249
522
Appendix
Table 13: Comparison between our full model and a simplified variant in which the environmental conditioning is removed from the diffusion inputs and applied only as forces outside the denoiser.
Dataset
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
COL ↓
GC (auto)
0.8919
1.3440
0.9059
0.0088
0.3365
1034.0
GC (ours)
0.8861
1.3339
0.8997
0.0087
0.3269
906
UCY (auto)
1.9083
3.9220
2.0423
0.0614
0.7466
644.0
UCY (ours)
1.8182
3.7292
1.8656
0.0598
0.7249
522
Appendix
Table 14: Performance comparison when replacing manually curated annotations with raw GroundSAM detections, which introduce missing and inaccurate boxes.
Method
MAE ↓
OT ↓
FDE ↓
MMD ↓
DTW ↓
COL ↓
SPDiff
0.4692
0.3153
1.7100
0.0886
0.2302
0.0
Ours
0.4083
0.2454
0.4639
0.0660
0.2162
0.0
Appendix
Table 15: Evaluation on the ETH dataset to assess cross-dataset generalization.
Dataset
MAE
OT
FDE
MMD
DTW
COL
GC (density)
0.9132
1.4298
0.9447
0.0092
0.3407
1140
GC (ours)
0.8861
1.3339
0.8997
0.0087
0.3269
906
UCY (density)
1.8542
3.9204
1.9490
0.0606
0.7223
690
UCY (ours)
1.8182
3.7292
1.8656
0.0598
0.7249
522
Appendix
Table 16: Replacing lighting conditioning with a global crowd-density representation constructed from a K×K density grid.
Pedestrian behavior modeling predicts and simulates how people move, interact, choose routes, and respond to vehicles, robots, hazards, and surroundings.
Forecasting pedestrian motion has always been fundamental for autonomous navigation in crowded environments. While attention-based methods achieve strong performance, they suffer from quadratic computational complexity in modeling social interactions, limiting scalability. Additionally, the existing methods often achieve high accuracy on prediction benchmarks at the individual level, but fail to fully capture the natural movement behaviors of crowds in real-world scenarios, particularly group structures. In this study, we propose Social Graph Mamba (SGM), a novel architecture that replaces attention-based social reasoning with Selective State Space Models (SSMs) operating on dynamically constructed interaction graphs. SGM introduces a dynamic interaction graph with social triplet factorization to decompose crowd interactions sequentially, and a community-aware module to effectively discover group structures via differentiable MinCut optimization and conditions both the embedding space and multi-modal decoder on group membership. Our experiments on standard benchmarks (ETH/UCY, SDD) demonstrate competitive performance with linear sequence complexity compared to quadratic attention-based methods. We further validate SGM in physical robot experiments by integrating predicted trajectories into a Social Force Model (SFM) for real-world implementation.
Hong-Son Nguyen, Yen-Chen Liu
Department of Mechanical Engineering, National Cheng Kung University, Tainan 70101, Taiwan
Crowd simulation plays a central role in robot navigation, autonomous driving, and urban planning. For these applications, realistic simulation requires crowds to adapt their behavior to environmental changes and user objectives. However, existing methods that rely on predefined control settings have limited flexibility in accommodating new user-specified objectives. To address this limitation, we propose Ctrl-CWM, a multi-agent Controllable Crowd World Model that integrates crowd generation and run-time control. Our key idea is to adapt the world-model principle of planning using imagined futures to crowd simulation. To this end, Ctrl-CWM consists of an encoder that learns a representation of human motion dynamics, an actor that proposes pedestrian displacements, a critic that evaluates imagined crowd trajectories, and a planner that selects actions. We first learn human motion dynamics through trajectory prediction on real-world pedestrian videos and then freeze the encoder to preserve them. Using this representation, the actor generates imagined crowd trajectories through repeated state updates, and the planner combines the critic's scores with user costs to select actions. Repeated planning advances the simulated crowd, while additional user costs introduce new control objectives without retraining. We extensively evaluate crowd generation under varied agent arrival conditions and run-time control across avoidance and attraction scenarios. Ctrl-CWM outperforms the state-of-the-art method on most crowd realism and collision metrics, and adapts crowd behaviors to user-specified objectives introduced during simulation. The project page is available at https://jungyu0413.github.io/Ctrl-CWM
Pedestrian simulation is a critical component for training and deploying social robot navigation approaches, yet it remains a largely rigid system that repeatedly requires manual data generation to define even simple scenarios. We propose GROVE, a text-to-scenario pedestrian simulation framework that combines state-of-the-art approaches to produce realistic, socially challenging scenarios for social robot navigation. Our framework allows users to customize one of several common presets (emergency, queuing, normal) or even enter a fully independent prompt to generate a highly customizable pedestrian simulation. Multiple modules separately ensure the realism and soundness of long-horizon human behavior, medium-horizon pedestrian navigation, and short-horizon robot/social interactions. Each module is tuned by the prompt in a way that reflects the user intent across all aspects of pedestrian simulation. By dynamically selecting one of several state-of-the-art (SotA) approaches in our modules based on the scenario, we capture many situational nuances of pedestrian behavior in order to narrow the simulation-to-real (sim2real) gap. The human simulation is directly integrated into Isaac Sim, Gazebo, and RViz simulators for robot deployment in highly social environments. We validate our approach through qualitative comparison against existing pedestrian simulation baselines across scenarios of varying complexity in residential, hospital, and office environments. The result is a high-fidelity pedestrian simulation that challenges social robot navigation with complex, diverse, realistic human behaviors.
Duc Tai Nguyen, Volodymyr Shcherbyna, Anh Do Duc +3
1Singapore Management University · 2Technical University Berlin · 3National University of Singapore