Natural language offers an intuitive way to direct a virtual crowd while a simulation is running. Handwritten rules execute efficiently but cannot respond flexibly to varied instructions, whereas asking a large language model (LLM) to control the crowd at every update introduces repeated latency, cost, and the risk of unsupported actions. We present CrowdDirector, a framework that uses an LLM only when the user adds or changes an instruction. The LLM translates the request into a structured instruction, and a scene validator checks whether the entities and actions it references exist in the current Unity scene. Accepted instructions are stored and remain continuously active as the simulation is running. At each simulation tick, a graph neural network uses the current scene state and active instruction to choose an available action and target for each agent, while Unity handles movement, interactions, and animation. The network is trained offline on simulated scenarios, with LLM guidance on how agents should respond to instructions. Our evaluations show the scene validator's feasibility decisions matched human judgements more often than the LLM's own assessment, and on a behavioural benchmark CrowdDirector follows user instructions more closely than handwritten rules while avoiding repeated LLM calls. It therefore provides a practical middle ground between fixed rules and continuous LLM control.
Figures & tables
Figure 1. CrowdDirector directs a running crowd with natural language. Four instructions are active at once: visitors gather for a party in the Event Hall, agents waiting at the broken coffee machine ( ⊗ ) are redirected to another drink source, visitors collect in the Music Room before a concert, and interested visitors move to the Gallery. Circles mark the agents each instruction affects. Each instruction is interpreted once, then applied locally without further language inference. A top-down view of a 2D venue in which four natural-language instructions are active at once, with coloured circles marking the agents each one affects. Four inset panels above show close-ups of the event hall, the broken coffee machine, the music room, and the gallery.
Figure 2. Overview of CrowdDirector. (a) Offline policy training. In generated worlds, simulation rollouts and a constrained LLM ranking score the same set of available actions, giving qroll and qLLM . Averaging them yields the hybrid target qtrain , which trains the graph policy. Because both distributions cover the same candidates, the blend cannot introduce an unavailable action. (b) Runtime orchestration. An LLM interprets a new instruction once (1), and the scene validator checks it against the live scene, storing an accepted result as a directive and rejecting an unsupported target (2). The current scene and active directives are then rebuilt as a crowd graph with a candidate list for each agent (3), the trained policy selects one candidate per agent (4), and Unity executes the result (5). A two-part block diagram. The upper part shows offline training, where simulation rollouts and a constrained LLM ranking of the same candidate actions are averaged into the target that trains the graph policy; the lower part shows the five runtime stages, from interpreting an instruction through scene validation, graph construction, and policy selection to execution in Unity.
Action
Target
Example
walk
zone
walk to the lounge
leave
exit
leave by the fire exit
drink
object
drink at the water cooler
rest
object
rest on the sofa
work
object
work at a desk
observe
object
look at the whiteboard
Table 1. Action types the candidate generator can list, with examples from the running example. The actions available to an agent depend on what the current scene provides.
Figure 3. Architecture of the graph policy. Type-specific encoders produce 64-dimensional node representations, which are updated by three relational GNN layers. A shared multilayer perceptron scores each candidate from its agent representation, target representation, their interaction, and the 34-dimensional candidate features. A left-to-right architecture diagram in which six typed node inputs pass through type-specific encoders and three relational graph neural network layers. A shared multilayer perceptron then scores each candidate action from the agent representation, the target representation, their element-wise product, and 34 candidate features.
Field
Example
Instruction
“The coffee machine is broken; redirect anyone waiting for a drink.”
Relevant state
coffee_machine_1: broken ; agent_07: waiting, thirst high
Candidate 0
use water_cooler_1
Candidate 1
continue current_activity
Candidate 2
move to lounge
LLM ranking
[0,2,1]
Table 2. Abridged example of an instruction-related LLM label. Probabilities follow the displayed candidate order.
Split
Use
Worlds
Decisions
LLM-blended
full/train
train
1,280
40,283
35,115
full/validation
selection
258
8,340
7,369
full/composition
held out
268
8,406
8,306
ext/train
train
584
9,483
4,624
ext/composition
held out
102
1,685
616
new-event/train
train
284
5,409
5,409
Table 3. Generated training corpus. Validation is used for checkpoint selection; composition splits are held out.
Approach
Runtime language use
Q2 Jaccard ↑
Q3 ↑
Acceptable ↑
Prohibited ↓
CrowdDirector
per new instruction
0.595
0.611
0.479
0.211
Handwritten rules
none
0.300
0.233
0.312
0.368
Per-tick LLM control
every tick
0.683
0.667
0.566
0.217
Table 4. Policy evaluation. Panel A compares alternative approaches. Q2 Jaccard measures agreement on the affected scope; Q3 measures agreement on the preferred candidate action. The per-tick LLM is a comparison method rather than a local decision maker. Panel B reports controlled ablations with identical data, candidates, masks, labels, and optimization. Learned results are means over three seeds. Best value in bold, second best underlined.
Feasibility assessor
Agreement ↑
95% CI
Scene validator
0.83
[0.61,1.00]
LLM self-assessment
0.47
[0.21,0.72]
Table 5. Feasibility agreement with participant Q1 judgements. Best value in bold.
(a) Authoring and recurrent control
Approach
Calls/instr.
Calls/update
Calls/session
Latency/update ↓
Cost/update
Cost/session ↓
CrowdDirector
1
0
1+N=6
20.5 ms
$0
$0.03
Whole-crowd LLM
1
1
1+N+T=606
3,880 ms
$0.0038
$2.30
Table 6. Runtime and scaling. (a) separates the sparse cost of authoring an instruction from the recurrent cost of control, instantiated for a session of N=5 instructions and T=600 updates. (b) reports CPU scaling at approximately constant agent density. (c) reports the fraction of whole-crowd LLM calls that assigned every agent before reaching the output-token limit, over 15 calls per crowd size. Measured on an Intel i9-13900H (14 physical cores), CPU only. Best value in bold.
Crowd simulation plays a central role in robot navigation, autonomous driving, and urban planning. For these applications, realistic simulation requires crowds to adapt their behavior to environmental changes and user objectives. However, existing methods that rely on predefined control settings have limited flexibility in accommodating new user-specified objectives. To address this limitation, we propose Ctrl-CWM, a multi-agent Controllable Crowd World Model that integrates crowd generation and run-time control. Our key idea is to adapt the world-model principle of planning using imagined futures to crowd simulation. To this end, Ctrl-CWM consists of an encoder that learns a representation of human motion dynamics, an actor that proposes pedestrian displacements, a critic that evaluates imagined crowd trajectories, and a planner that selects actions. We first learn human motion dynamics through trajectory prediction on real-world pedestrian videos and then freeze the encoder to preserve them. Using this representation, the actor generates imagined crowd trajectories through repeated state updates, and the planner combines the critic's scores with user costs to select actions. Repeated planning advances the simulated crowd, while additional user costs introduce new control objectives without retraining. We extensively evaluate crowd generation under varied agent arrival conditions and run-time control across avoidance and attraction scenarios. Ctrl-CWM outperforms the state-of-the-art method on most crowd realism and collision metrics, and adapts crowd behaviors to user-specified objectives introduced during simulation. The project page is available at https://jungyu0413.github.io/Ctrl-CWM
Crowd simulation aims to generate agent behaviours that are rich, natural, and appropriate to their surrounding environment. Recent advances in Machine Learning have enabled such behavioural patterns to be learned directly from trajectory data, allowing simulation models to capture increasingly complex crowd dynamics. However, the capabilities of these systems remain strongly dependent on the type and diversity of the data used during training. When evaluating how well crowd simulators generalise to real-world datasets, most existing metrics focus primarily on trajectory-level differences rather than behaviour-level similarity. In this paper, we present Trajeform, a framework that trains a reconstruction VAE (Variational Autoencoder) on real trajectory data to learn a meaningful continuous embedding that preserves behavioural characteristics of real crowds. The trained encoder can then be applied to synthetic trajectories generated by simulators and used to compare the resulting embedding distributions. Additionally, the learned embedding can be used to analyse supplementary data, such as trajectories corresponding to a simulator's principal behaviours or captured data from controlled environments (e.g., cycling), providing insight into their ability to span the real behavioural embedding space. The framework compares data from three simulators (CCP, modCCP, and RVO) against ground-truth datasets and is evaluated quantitatively and qualitatively using latent-space analysis, distributional metrics, and trajectory generation through sampling from the aggregated posterior distribution. The results indicate that the learned latent space captures meaningful trajectory dependencies found in the real data, making the framework useful both as an additional measure of synthetic data believability and as a tool for informing the training of future trajectory models.
LLM-agent simulations make natural-language social scenarios easy to instantiate, but their outputs can be overread as predictions and are often difficult to compare with explicit social dynamics. We present AgoraSim, a hybrid agent-based modeling framework for scenario-oriented social reaction analysis. AgoraSim resolves textual or multimodal artifacts into editable ABM configurations, runs ratio-controlled populations that mix LLM, vision-language, custom-endpoint, random, and classical agents, and compares the same scenario against matched classical reference dynamics. All agents emit a shared structured decision object, enabling common action spaces, interaction protocols, metrics, and audit records. Exposed through a local UI, Python SDK/CLI, and REST API, AgoraSim helps users inspect scenario trajectories, compare modeling assumptions, and identify cases that warrant empirical validation.
Chung-Chi Chen
Human-Agent Ally Lab (HAA Lab) National Institute of Informatics, Japan