Natural language offers an intuitive way to direct a virtual crowd while a simulation is running. Handwritten rules execute efficiently but cannot respond flexibly to varied instructions, whereas asking a large language model (LLM) to control the crowd at every update introduces repeated latency, cost, and the risk of unsupported actions. We present CrowdDirector, a framework that uses an LLM only when the user adds or changes an instruction. The LLM translates the request into a structured instruction, and a scene validator checks whether the entities and actions it references exist in the current Unity scene. Accepted instructions are stored and remain continuously active as the simulation is running. At each simulation tick, a graph neural network uses the current scene state and active instruction to choose an available action and target for each agent, while Unity handles movement, interactions, and animation. The network is trained offline on simulated scenarios, with LLM guidance on how agents should respond to instructions. Our evaluations show the scene validator's feasibility decisions matched human judgements more often than the LLM's own assessment, and on a behavioural benchmark CrowdDirector follows user instructions more closely than handwritten rules while avoiding repeated LLM calls. It therefore provides a practical middle ground between fixed rules and continuous LLM control.
Figures & tables
Figure 1. CrowdDirector directs a running crowd with natural language. Four instructions are active at once: visitors gather for a party in the Event Hall, agents waiting at the broken coffee machine ( ⊗ ) are redirected to another drink source, visitors collect in the Music Room before a concert, and interested visitors move to the Gallery. Circles mark the agents each instruction affects. Each instruction is interpreted once, then applied locally without further language inference. A top-down view of a 2D venue in which four natural-language instructions are active at once, with coloured circles marking the agents each one affects. Four inset panels above show close-ups of the event hall, the broken coffee machine, the music room, and the gallery.
Figure 2. Overview of CrowdDirector. (a) Offline policy training. In generated worlds, simulation rollouts and a constrained LLM ranking score the same set of available actions, giving qroll and qLLM . Averaging them yields the hybrid target qtrain , which trains the graph policy. Because both distributions cover the same candidates, the blend cannot introduce an unavailable action. (b) Runtime orchestration. An LLM interprets a new instruction once (1), and the scene validator checks it against the live scene, storing an accepted result as a directive and rejecting an unsupported target (2). The current scene and active directives are then rebuilt as a crowd graph with a candidate list for each agent (3), the trained policy selects one candidate per agent (4), and Unity executes the result (5). A two-part block diagram. The upper part shows offline training, where simulation rollouts and a constrained LLM ranking of the same candidate actions are averaged into the target that trains the graph policy; the lower part shows the five runtime stages, from interpreting an instruction through scene validation, graph construction, and policy selection to execution in Unity.
Action
Target
Example
walk
zone
walk to the lounge
leave
exit
leave by the fire exit
drink
object
drink at the water cooler
rest
object
rest on the sofa
work
object
work at a desk
observe
object
look at the whiteboard
Table 1. Action types the candidate generator can list, with examples from the running example. The actions available to an agent depend on what the current scene provides.
Figure 3. Architecture of the graph policy. Type-specific encoders produce 64-dimensional node representations, which are updated by three relational GNN layers. A shared multilayer perceptron scores each candidate from its agent representation, target representation, their interaction, and the 34-dimensional candidate features. A left-to-right architecture diagram in which six typed node inputs pass through type-specific encoders and three relational graph neural network layers. A shared multilayer perceptron then scores each candidate action from the agent representation, the target representation, their element-wise product, and 34 candidate features.
Field
Example
Instruction
“The coffee machine is broken; redirect anyone waiting for a drink.”
Relevant state
coffee_machine_1: broken ; agent_07: waiting, thirst high
Candidate 0
use water_cooler_1
Candidate 1
continue current_activity
Candidate 2
move to lounge
LLM ranking
[0,2,1]
Table 2. Abridged example of an instruction-related LLM label. Probabilities follow the displayed candidate order.
Split
Use
Worlds
Decisions
LLM-blended
full/train
train
1,280
40,283
35,115
full/validation
selection
258
8,340
7,369
full/composition
held out
268
8,406
8,306
ext/train
train
584
9,483
4,624
ext/composition
held out
102
1,685
616
new-event/train
train
284
5,409
5,409
Table 3. Generated training corpus. Validation is used for checkpoint selection; composition splits are held out.
Approach
Runtime language use
Q2 Jaccard ↑
Q3 ↑
Acceptable ↑
Prohibited ↓
CrowdDirector
per new instruction
0.595
0.611
0.479
0.211
Handwritten rules
none
0.300
0.233
0.312
0.368
Per-tick LLM control
every tick
0.683
0.667
0.566
0.217
Table 4. Policy evaluation. Panel A compares alternative approaches. Q2 Jaccard measures agreement on the affected scope; Q3 measures agreement on the preferred candidate action. The per-tick LLM is a comparison method rather than a local decision maker. Panel B reports controlled ablations with identical data, candidates, masks, labels, and optimization. Learned results are means over three seeds. Best value in bold, second best underlined.
Feasibility assessor
Agreement ↑
95% CI
Scene validator
0.83
[0.61,1.00]
LLM self-assessment
0.47
[0.21,0.72]
Table 5. Feasibility agreement with participant Q1 judgements. Best value in bold.
(a) Authoring and recurrent control
Approach
Calls/instr.
Calls/update
Calls/session
Latency/update ↓
Cost/update
Cost/session ↓
CrowdDirector
1
0
1+N=6
20.5 ms
$0
$0.03
Whole-crowd LLM
1
1
1+N+T=606
3,880 ms
$0.0038
$2.30
Table 6. Runtime and scaling. (a) separates the sparse cost of authoring an instruction from the recurrent cost of control, instantiated for a session of N=5 instructions and T=600 updates. (b) reports CPU scaling at approximately constant agent density. (c) reports the fraction of whole-crowd LLM calls that assigned every agent before reaching the output-token limit, over 15 calls per crowd size. Measured on an Intel i9-13900H (14 physical cores), CPU only. Best value in bold.