Crowd simulation aims to generate agent behaviours that are rich, natural, and appropriate to their surrounding environment. Recent advances in Machine Learning have enabled such behavioural patterns to be learned directly from trajectory data, allowing simulation models to capture increasingly complex crowd dynamics. However, the capabilities of these systems remain strongly dependent on the type and diversity of the data used during training. When evaluating how well crowd simulators generalise to real-world datasets, most existing metrics focus primarily on trajectory-level differences rather than behaviour-level similarity. In this paper, we present Trajeform, a framework that trains a reconstruction VAE (Variational Autoencoder) on real trajectory data to learn a meaningful continuous embedding that preserves behavioural characteristics of real crowds. The trained encoder can then be applied to synthetic trajectories generated by simulators and used to compare the resulting embedding distributions. Additionally, the learned embedding can be used to analyse supplementary data, such as trajectories corresponding to a simulator's principal behaviours or captured data from controlled environments (e.g., cycling), providing insight into their ability to span the real behavioural embedding space. The framework compares data from three simulators (CCP, modCCP, and RVO) against ground-truth datasets and is evaluated quantitatively and qualitatively using latent-space analysis, distributional metrics, and trajectory generation through sampling from the aggregated posterior distribution. The results indicate that the learned latent space captures meaningful trajectory dependencies found in the real data, making the framework useful both as an additional measure of synthetic data believability and as a tool for informing the training of future trajectory models.
Figures & tables
Fig. 1: Model architecture.
Architecture
Test Latent Loss ↓
conv2gru
.381
conv2conv
.304
gru2gru
.440
gru2conv
.414
conv2gru_mog
.443
TABLE I: Cross architecture experimentation on the complete dataset assembly (Flock, Students, Zara, and ETH)
Fig. 2: Real Data Features.
Fig. 3: Model Feature Maps. Instances of convolution features for two different trajectory samples (left and middle), and the heatmap of the learned latent layer (right).
Fig. 4: Trained and test latent values.
Mean Latent Loss ↓
w/ full model
w/hold-out
w/o Flock
1.123
1.012
w/o Students
.775
.952
w/o Zara
.724
.842
w/o ETH
1.286
2.160
TABLE II: Experiments of the effect of holding out a dataset for training, and then testing using the excluded data.
Fig. 5: Latent visualisations of targeted samples.
Fig. 6: Latent clustering and cluster centres’ principal behaviours.
Fig. 7: Framework usage on simulator-specific data, mimicking the real data from three simulation types, with their cluster centres.
Fig. 8: Framework usage on additional simulator-specific data, i.e., CEDRL, and the Zara-specific GREIL and PAG-Crowd data.
Dataset
Recon. Loss ↓
Active dims ↑
Latent Entropy ↑
All
.230
7
.916
Students
.323
7
.956
Zara
.317
8
.845
ETH
.078
4
.919
TABLE III: Testing for VAE model collapse by training dedicated per-dataset models.
Fig. 9: Latents of a controlled scenario: single file movements.
Fig. 10: Latents of cyclists’ movements in a controlled oval scenario.
Fig. 11: Editing the latent distribution of the observed data. Example of bringing one of its means to coincide with that of the real data (left), and showing the points sampled according to the edited cluster centre (right - yellow points).
Fig. 12: Projections of specific behaviour scenarios onto the embedding space from the principal profiles in modCCP (right), compared to mixture of behaviours predicted for the real data (left).
Fig. 13: Original projections from MPACT [ 4 ] .
Fig. 14: Generating simulator-specific parameters guided from the informed distribution (a), either via nearest neighbours (b), or a trained parameter model (c). The new set of parameters span the simulator (MPACT) space more faithfully to the real projections compared to sampling the entire space (d).
Fig. 15: Embedding Distribution Self-Consistency Check. Sampled points from the inferred distribution (MoG), generate the corresponding trajectories which are then passed through the encoder to produce new latents, projected in a new posterior distribution which should highly overlap the inferred MoG.
Fig. 16: Boxplots of trajectory feature distributions, (x,z) positions and speed, for real data, and decoded data from our posterior samples, MPACT, RVO, and CCP baseline samples.
Fig. 17: Difference correlation matrices of real against generated data, sampled from different empirical distributions, i.e.,our latent posterior, prior, and empirical RVO and MPACT.
Metric
Prior
MoG
RVO
CCP
MPACT
Latent Metric ↓
3.833
.705
2.732
4.433
1.867
MMD ↓
.344
.008
.030
.101
.029
SWD ↓
2.148
.328
1.035
1.215
.505
Energy ↓
27.736
.541
6.498
5.762
1.585
JSD x↓
.442
.007
.014
.086
.024
JSD z↓
.261
.011
.116
.021
.032
TABLE IV: Behaviour Similarity Metrics of Different Posterior-Generated Trajectories. These trajectories are generated from our decoder, subject to latents sampled from a source-specific inferred MoG. We compare these sets of data against the real data, testing for similarity between the distributions of several metrics.
Metric
RVO
CCP
MPACT
Latent Metric ↓
2.732
4.433
1.867
Latent Energy ↓
.891
.987
.479
MoG WD ↓
1.354
2.259
1.300
Fréchet Distance ↓
1.757
3.845
.855
TABLE V: Additional metrics measured on the latent space for the RVO, CCP, and MPACT-generated data projected and compared against the real latents.
Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.
Crowd simulation plays a central role in robot navigation, autonomous driving, and urban planning. For these applications, realistic simulation requires crowds to adapt their behavior to environmental changes and user objectives. However, existing methods that rely on predefined control settings have limited flexibility in accommodating new user-specified objectives. To address this limitation, we propose Ctrl-CWM, a multi-agent Controllable Crowd World Model that integrates crowd generation and run-time control. Our key idea is to adapt the world-model principle of planning using imagined futures to crowd simulation. To this end, Ctrl-CWM consists of an encoder that learns a representation of human motion dynamics, an actor that proposes pedestrian displacements, a critic that evaluates imagined crowd trajectories, and a planner that selects actions. We first learn human motion dynamics through trajectory prediction on real-world pedestrian videos and then freeze the encoder to preserve them. Using this representation, the actor generates imagined crowd trajectories through repeated state updates, and the planner combines the critic's scores with user costs to select actions. Repeated planning advances the simulated crowd, while additional user costs introduce new control objectives without retraining. We extensively evaluate crowd generation under varied agent arrival conditions and run-time control across avoidance and attraction scenarios. Ctrl-CWM outperforms the state-of-the-art method on most crowd realism and collision metrics, and adapts crowd behaviors to user-specified objectives introduced during simulation. The project page is available at https://jungyu0413.github.io/Ctrl-CWM
Virtual clients offer a cost-effective approach to support applications such as A/B testing, recommender system development, and interface evaluation. However, building them requires access to large-scale, semantically faithful, fine-grained online user trajectories. These data are difficult to obtain because proprietary logs are subject to privacy restrictions and small businesses often lack sufficient traffic. Consequently, existing public datasets either abstract away fine-grained user interaction details or preserve rich context but remain platform-specific and small-scale. To address this gap, we propose SimTrace, a framework that generates faithful, fine-grained synthetic multimodal clickstreams through a computer-use client agent that is grounded in real user trajectories and the given web environment. SimTrace anonymizes real interactions and constructs a simulated twin of the given web environment, then uses both to generate synthetic interaction trajectories. Each action is paired with its corresponding web observations and user context, yielding a shareable alternative to confidential logs for developing computer-use agent-style virtual clients. We apply SimTrace to an e-commerce setting and evaluate both its fidelity and downstream utility. SimTrace outperforms competing baselines on 7 out of 8 fidelity metrics. Models trained on synthetic data achieve performance comparable to those trained on real data on downstream tasks such as purchase prediction and recommendation. For next action prediction task, augmenting real data with synthetic data further improves accuracy by 11.0% relative to training on real data alone. We release SimTrace as an open-source package to facilitate research on online user behavior modeling.
Pedestrian simulators need a behaviour rule for every agent, but privacy usually limits the data for setting one to aggregate statistics, namely zone-level device counts and origin-to-destination (OD) flows, with no individual trajectories. Such aggregates under-determine individual behaviour, because many different sets of decisions reproduce the same counts. We fine-tune a language model crowd agent so that the simulated population matches the observed destination composition, the fraction of the departing crowd heading to each point of interest. We read this target from the OD flow and reweight the model's own destination distribution onto it by iterative proportional fitting. Because fine-tuning inflates the dominant destination class, we fit the low-rank adapter to trajectories resampled to a corrected training composition that reaches the target after this inflation. On mobile network counts from two baseball games the fine-tuned agent runs without inference-time correction, cutting the destination-share error by 25%, while the grid correlation remains similar across policies.
Tatsuya Amano, Hirozumi Yamaguchi
The University of Osaka, Suita, Japan · RIKEN Center for Computational Science, Kobe, Japan