LLM-Based Hierarchical Coordinated Control with Continuation-Aware Policy Learning

Changhong He; Jinda Gao; Xinkuan Liu; Le Zhang; Xizi Luo; Yu Mei

cs.AIAug 15, 2026

LLM-Based Hierarchical Coordinated Control with Continuation-Aware Policy Learning

Authors: Changhong He, Jinda Gao, Xinkuan Liu, Le Zhang, Xizi Luo, Yu Mei

Organizations: Beihang University, Beijing, China · Baidu, Inc., Beijing, China

Abstract

Coordinating multiple interacting units in complex engineering systems is challenging when system interactions are difficult to model, operational information is heterogeneous, and low-level actions must satisfy strict constraints. We propose an LLM-based hierarchical framework in which the LLM coordinates interacting units based on heterogeneous operational context, while task-specific controllers or optimizers generate executable and constraint-aware actions. We further introduce Continuation-Aware GRPO to capture the consequences of coordination decisions over subsequent control intervals. Rather than judging a decision only by its immediate outcome, the method also evaluates how the system evolves afterward under the current policy. We validate the framework on multi-ramp traffic control and virtual power plant (VPP) energy management, using simplified system models for training and more realistic simulators for evaluation. Across both tasks, the proposed method consistently outperforms direct task-specific control and optimization, end-to-end reinforcement learning, rule-based and RL-based hierarchical coordination, and prompting-only LLM coordinators, demonstrating the value of heterogeneous-context reasoning, hierarchical execution, and continuation-aware policy learning.

Figures & tables

Figure 1: Overview of the proposed LLM-based hierarchical coordination framework and Continuation-Aware GRPO.

Method	Seen	Unseen	High-U	Overall	Gain
Direct Control
Feedback Control	16283.4 $\pm$ 41.8	15576.8 $\pm$ 50.6	14917.3 $\pm$ 68.3	15592.5 $\pm$ 49.1	–
End-to-End RL	16537.6 $\pm$ 163.7	15684.9 $\pm$ 191.5	14813.2 $\pm$ 228.4	15678.6 $\pm$ 181.2	+0.55%
Hierarchical Coordination
Fixed-Mode Hierarchy	16502.8 $\pm$ 36.2	15792.4 $\pm$ 44.9	15158.7 $\pm$ 60.5	15818.0 $\pm$ 43.7	+1.45%
Hierarchical RL	16843.1 $\pm$ 98.6	16117.6 $\pm$ 121.4	15434.8 $\pm$ 151.8	16131.8 $\pm$ 110.2	+3.46%

Table 1: Traffic throughput in SUMO over five matched runs (mean

\pm

SD). Gain is relative to Feedback Control; higher is better.

Method	Seen	Unseen	High-U	Overall	Gain
Direct Control
MPC-MILP	32783.5 $\pm$ 47.2	33887.9 $\pm$ 65.8	35843.3 $\pm$ 111.4	34171.6 $\pm$ 64.2	–
End-to-End RL	35076.9 $\pm$ 512.4	37423.6 $\pm$ 647.3	40947.2 $\pm$ 923.8	37815.9 $\pm$ 608.5	$-10.66\%$
Hierarchical Coordination
Fixed-Mode Hierarchy	32984.7 $\pm$ 52.6	34052.2 $\pm$ 71.9	35796.4 $\pm$ 119.8	34277.8 $\pm$ 70.6	$-0.31\%$
Hierarchical RL	32183.7 $\pm$ 267.5	33237.2 $\pm$ 351.6	35124.6 $\pm$ 505.2	33515.2 $\pm$ 317.4	+1.92%

Table 2: VPP operating cost in OpenDSS over five matched runs (mean

\pm

SD). Gain is relative to MPC-MILP; lower is better.

	Multi-Ramp Traffic Control			Virtual Power Plant
Method	LTM $\uparrow$	SUMO $\uparrow$	Gap $\downarrow$	Fast Simulator $\downarrow$	OpenDSS $\downarrow$	Gap $\downarrow$
Qwen3-8B	16312.6 $\pm$ 33.4	15270.7 $\pm$ 55.1	6.39 $\pm$ 0.21%	34025.2 $\pm$ 87.9	36876.2 $\pm$ 144.7	8.38 $\pm$ 0.26%
End-to-End RL	17438.7 $\pm$ 147.8	15678.6 $\pm$ 181.2	10.09 $\pm$ 1.07%	33785.3 $\pm$ 438.6	37815.9 $\pm$ 608.5	11.94 $\pm$ 2.03%
Hierarchical RL	17043.9 $\pm$ 91.5	16131.8 $\pm$ 110.2	5.35 $\pm$ 0.54%	30684.7 $\pm$ 229.4	33515.2 $\pm$ 317.4	9.23 $\pm$ 0.95%
Ours	17286.4 $\pm$ 66.7	16625.4 $\pm$ 84.5	3.82 $\pm$ 0.33%	30941.8 $\pm$ 142.7	32679.2 $\pm$ 223.6	5.62 $\pm$ 0.54%

Table 3: Cross-simulator performance over five matched runs (mean

\pm

SD). Gap measures degradation from the fast to the more realistic simulator.

Variant	Seen	Unseen	High-U	Overall	Gain
GRPO ( $H=\Delta$ )	16861.4 $\pm$ 112.8	16327.6 $\pm$ 151.3	15767.1 $\pm$ 211.6	16318.7 $\pm$ 136.4	–
Ours ( $H=4\Delta$ )	17043.6 $\pm$ 75.9	16617.9 $\pm$ 92.7	16214.7 $\pm$ 118.6	16625.4 $\pm$ 84.5	+1.88%

Table 4: Ablation of continuation-aware return construction over five runs with matched evaluation seeds (mean

\pm

SD). Gain is relative to GRPO with

H=\Delta

Parameter	Value
$v^{\text{f}}$	$22.22$ m/s
$v^{\text{w}}$	$5.56$ m/s
per-lane jam density	$0.1333$ veh/m
per-lane capacity	$0.5$ veh/s
lane efficiency	$1.0$
LTM internal step $h$	$5$ s

Table 5: Corridor fundamental-diagram and control-loop parameters. Capacity is imposed as the independent cap of Eq. ( 20 ) rather than derived from the wave speeds and jam density, so the four diagram values need not satisfy the triangular identity exactly; the binding value is whichever term of the minimum is smaller.

Mode	$\delta$
Sensitive	$0.04$
Standard	$0.10$
Sluggish	$0.20$
Always-Open	—

Table 6: Ramp mode-to-parameter mapping. Always-Open holds the gate open instead of setting a bandwidth.

Parameter	Value	Parameter	Value
$C_{\text{bat}}$	$1800$ kWh	$C_{\text{th}}$	$150$ kWh/ ∘ C
$\eta_{\text{ch}},\eta_{\text{dis}}$	$0.95$	$uA$	$45$ kW/ ∘ C
$\bar{P}^{\text{ch}},\bar{P}^{\text{dis}}$	$350$ kW	$\mathrm{COP}$	$3.0$
SoC range	$[0.20,0.90]$	$T$ band	$[22.0,26.5]\,^{\circ}$ C
$\bar{P}^{\text{ev,max}}$	$600$ kW	$T^{\text{set}}$	$24.0\,^{\circ}$ C
$\bar{P}^{\text{pv,max}}$	$1200$ kW	$\bar{P}^{\text{hv,max}}$	$300$ kW

Table 7: VPP physical and control-loop parameters. The thermal pair gives a building time constant

C_{\text{th}}/uA\approx 3.3

h; holding the setpoint against a

15\,^{\circ}

C indoor–outdoor difference draws

uA\,\Delta T/\mathrm{COP}=225

kW, three quarters of the HVAC rating, so comfort is a binding constraint rather than a free one.

\bar{P}^{\text{grid}}

is the contracted import cap, enforced through the penalised hinge

\sigma^{\text{g}}

rather than as a variable bound, so import may exceed it at a cost; the physical connection limit is not binding. The value shown is nominal and varies by scenario.

Resource	$\theta$	Cons.	Bal.	Pro.
PV	$\delta^{\text{pv}}$	$0.35$	$0.25$	$0.10$
Battery	$\mathrm{SoC}^{\text{res}}$	$0.30$	$0.25$	$0.22$
Battery	$\alpha^{\text{dis}}$	$0.70$	$0.85$	$1.00$
EV	$w^{\text{ev}}$	$1.45$	$1.00$	$0.70$
HVAC	$w^{\text{hv}}$	$0.75$	$1.00$	$1.60$

Table 8: Mode-to-parameter mapping

\theta(z)

. Each row is governed solely by the mode selected for the resource in the first column.

Category	Multi-ramp traffic control	Virtual power plant
Numerical measurements	per-ramp flow $q$ , speed $v$ , occupancy $o$ ; gate state $g$	$\mathrm{SoC}$ , PV power, base load, indoor and outdoor temperature, setpoint, import cap, time-of-use price
Recent trends	$3$ -step series of $q$ , $v$ , $o$ over the last $3$ min	$8$ -step series of PV, base load, outdoor temperature over the last $40$ min
Forecasts	mainline inflow and per-ramp arrivals over the next interval, with stated reliability	PV, base load, and outdoor temperature over the MPC window, with stated reliability
Uncertainty estimates	corridor demand-uncertainty level	PV uncertainty level
Resource urgency	minutes since last closure, per ramp	residual EV energy, urgent share $\omega$ , required charging pace
Operational priorities	per-ramp service priority tier	HVAC comfort priority

Table 9: Operational-context categories serialised into the prompt, following the decomposition of

c_{t}

in the main paper. Derived signals are computable from the measurements and observed history alone. The forecast row is a forecast in the strict sense — the same one the low-level layer plans with, carrying error and a stated reliability — and no realised future value enters the prompt.

Setting	Value
group size $G$	$8$
horizon $H$	$4\Delta$ ( $12$ min / $120$ min)
clip $\epsilon_{\text{c}}$	$0.2$
KL coefficient $\beta$	$0.05$
$\varepsilon_{\text{std}}$	$10^{-8}$
importance ratio	sequence level

Table 10: Continuation-Aware GRPO hyperparameters, shared by both tasks.

Setting	Value
trunk	MLP $2\times 256$ , $\tanh$
clip	$0.2$
discount $\gamma$	$0.99$
GAE $\lambda$	$0.95$
learning rate	$3\!\times\!10^{-4}$
entropy coefficient	$0.01$

Table 11: PPO settings shared by the End-to-End RL and Hierarchical RL baselines.

Ramp	1	2	3	4	5	6	7	8
SUMO	$.28$	$.22$	$.25$	$.23$	$.22$	$.23$	$.25$	$.25$
LTM	$.948$	$.926$	$.782$	$.824$	$.220$	$.230$	$.449$	$.188$

Table 12: Hysteresis centres

c^{\text{ctr}}_{r}

per simulator. Ramps 5 and 6 sit in the tail of the SUMO occupancy distribution, so quantile alignment leaves them at the SUMO value and the two rows agree there.

Aspect	Training model	Evaluation model
Multi-ramp traffic control
Representation	link-level cumulative counts, triangular fundamental diagram	individual vehicles with car-following and lane-change behaviour
Omitted in training	lane changing, weaving in the merge influence area, driver heterogeneity, stochastic gap acceptance	—
Observables	link flow, speed, density from the state update	detector counts and occupancies over $60$ s windows
Virtual power plant
Representation	single-bus power balance	three-phase feeder power flow on EPRI Ckt5

Table 13: What the fast training model omits relative to the evaluation environment. In both tasks the omission is a spatial mechanism that the high-level decision space does not address directly, so transfer tests whether a coordination policy learned on aggregate dynamics survives disaggregation.

Series	Source	Coverage	Transformation
Multi-ramp traffic control
Corridor demand	expressway detector records	northbound Western Expressway, Changchun	per-origin episode totals shaped by the sampled temporal profile and scaled by the level multipliers of Table 17
Virtual power plant
PV generation	NASA POWER, all-sky surface shortwave irradiance	Suzhou, 2020, hourly	scaled to the $1200$ kW array, interpolated to $5$ min
Outdoor temperature	NASA POWER, 2 m air temperature	Suzhou, 2020, hourly	used directly as $T^{\text{out}}$ , interpolated to $5$ min
Park base load	Building Data Genome 2, cleaned electricity meters	office buildings, 2016–2017, hourly	season-conditional hour-of-day profile, normalised then mapped to $[1200,1800]$ kW

Table 14: Exogenous data sources and the transformation applied to each. Measured series supply temporal shape; amplitude comes from the modelled installation.

Tier	Hours	Price (CNY/kWh)
Valley	$0$ – $8$	$0.35$
Shoulder	$11$ – $13$ , $22$ – $24$	$0.65$
Peak	$8$ – $11$ , $13$ – $17$	$1.00$
Critical peak	$17$ – $22$	$1.25$

Table 15: Time-of-use tariff; hours are local clock time. Tier prices are perturbed per scenario by an independent factor in

[0.95,1.05]

Family	Binding pressure
Multi-ramp traffic control
Upstream-dominant	strong mainline, weak ramps
Ramp-dominant	moderate mainline, several strong ramps
Balanced moderate	no single binding location
Balanced heavy	mainline and ramps both near capacity
Virtual power plant

Table 16: Operating-pattern families. Splits are defined over families; quantitative realisations, and for the corridor the temporal profile, vary within a family.

	Weak	Medium	Strong
Mainline multiplier	$0.75$	$1.00$	$1.20$
Ramp multiplier	$0.65$	$1.00$	$1.35$
Ramp rate cap (veh/h)	$900$	$1200$	$2000$

Table 17: Corridor demand levels. Warm-up runs at

0.50

of the level amplitude.

Component	MPC-MILP	Ours
Energy purchased at the tariff	$15019.4$	$14897.0$
Import above the contracted cap	$17715.6$	$17050.4$
EV energy undelivered at deadline	$1415.3$	$719.3$
Comfort-band excursion	$21.3$	$12.5$
Total	$34171.6$	$32679.2$

Table 18: Composition of the reported VPP operating cost, in CNY per episode, for the direct-control baseline and our policy. Every row is the mean over scenarios of the per-scenario charge, not a mean rate applied to a mean quantity: the EV row is

\frac{1}{n}\sum_{i}\bar{\lambda}_{\text{e}}(\omega_{i})u_{i}

with

u_{i}

the shortfall in scenario

i

, so the average rate it implies differs between methods —

16.4

CNY/kWh for MPC-MILP against

14.8

for ours — because the shortfall is not independent of the urgent share. Our policy’s residual shortfalls fall preferentially in low-

\omega

scenarios, which is the intended behaviour of the proactive EV mode and is worth

0.7

CNY/kWh of the difference on its own. The comfort row is the excursion of Table 20 at

\lambda_{\text{d}}=15.0

, and the over-cap row is a

\lambda_{\text{g}}=8.0

CNY/kWh charge, roughly eight times the average tariff, which is why it is large even though the over-cap volume is a small share of total import.

Method	Bottleneck speed (km/h) $\uparrow$	Congestion duration (min) $\downarrow$	Mean ramp queue (veh) $\downarrow$	Spillover events $\downarrow$
Feedback Control	$58.4\pm 0.7$	$62.3\pm 1.8$	$12.4\pm 0.3$	$3.1\pm 0.4$
End-to-End RL	$60.1\pm 2.4$	$58.7\pm 5.9$	$18.9\pm 1.7$	$7.6\pm 1.5$
Fixed-Mode Hierarchy	$59.6\pm 0.6$	$59.4\pm 1.6$	$12.9\pm 0.3$	$3.4\pm 0.4$
Hierarchical RL	$63.2\pm 1.5$	$51.8\pm 3.4$	$13.6\pm 0.7$	$3.8\pm 0.6$
Qwen3-8B	$54.7\pm 0.8$	$71.6\pm 2.1$	$15.2\pm 0.5$	$5.2\pm 0.7$
Gemini 3.1 Pro	$58.9\pm 0.7$	$61.2\pm 1.9$	$12.7\pm 0.4$	$3.2\pm 0.4$

Table 19: Corridor auxiliary metrics in SUMO, overall scores over five matched runs. Bottleneck speed is the time mean over the

180

-minute evaluation window at the critical mainline section; free-flow speed is

v^{\text{f}}=80

km/h.

Method	Violation steps $\downarrow$	Curtailed (kWh) $\downarrow$	Unmet EV (kWh) $\downarrow$	Excursion ( ∘ C $\cdot$ h) $\downarrow$
MPC-MILP	$9.4\pm 0.6$	$412\pm 28$	$86.3\pm 5.1$	$1.42\pm 0.11$
End-to-End RL	$26.8\pm 3.9$	$1348\pm 214$	$392.6\pm 58.3$	$4.87\pm 0.72$
Fixed-Mode Hierarchy	$9.8\pm 0.7$	$431\pm 31$	$91.7\pm 5.6$	$1.51\pm 0.12$
Hierarchical RL	$11.2\pm 1.3$	$528\pm 61$	$71.4\pm 8.9$	$1.18\pm 0.15$
Qwen3-8B	$18.3\pm 1.4$	$892\pm 86$	$264.1\pm 26.4$	$3.42\pm 0.31$
Gemini 3.1 Pro	$12.1\pm 0.9$	$574\pm 48$	$148.9\pm 14.2$	$2.06\pm 0.19$

Table 20: Virtual power plant auxiliary metrics in OpenDSS, overall scores over five matched runs. Violation steps count

5

-minute intervals with a voltage or thermal violation; Excursion is the integrated indoor-temperature deviation outside the comfort band. The episode is the

12

-hour window

10

00

–

22

00

\Delta_{\text{low}}=5

min, so an episode contains

144

low-level steps.

Method	Corridor	VPP
Qwen3-8B	$4.7\%$	$3.1\%$
Gemini 3.1 Pro	$0.4\%$	$0.2\%$
Claude Sonnet 4.5	$0.5\%$	$0.3\%$
Ours	$0.03\%$	$0.00\%$

Table 21: Schema-violation rate over all decision steps of the test set.

	Multi-ramp traffic control			Virtual power plant
Method	LTM $\uparrow$	SUMO $\uparrow$	Gap $\downarrow$	Fast model $\downarrow$	OpenDSS $\downarrow$	Gap $\downarrow$
Direct control	$16295.2\pm 38.4$	$15592.5\pm 49.1$	$4.31\pm 0.19\%$	$31954.0\pm 41.6$	$34171.6\pm 64.2$	$6.94\pm 0.22\%$
End-to-End RL	$17438.7\pm 147.8$	$15678.6\pm 181.2$	$10.09\pm 1.07\%$	$33785.3\pm 438.6$	$37815.9\pm 608.5$	$11.94\pm 2.03\%$
Fixed-Mode Hierarchy	$16584.2\pm 34.9$	$15818.0\pm 43.7$	$4.62\pm 0.21\%$	$31981.3\pm 46.3$	$34277.8\pm 70.6$	$7.18\pm 0.24\%$
Hierarchical RL	$17043.9\pm 91.5$	$16131.8\pm 110.2$	$5.35\pm 0.54\%$	$30684.7\pm 229.4$	$33515.2\pm 317.4$	$9.23\pm 0.95\%$
Qwen3-8B	$16312.6\pm 33.4$	$15270.7\pm 55.1$	$6.39\pm 0.21\%$	$34025.2\pm 87.9$	$36876.2\pm 144.7$	$8.38\pm 0.26\%$

Table 22: Cross-simulator performance for all methods over five matched runs. Direct control is Feedback Control for the corridor and MPC-MILP for the VPP. The four rows also present in the main paper are reproduced unchanged.

	Multi-ramp traffic control: throughput $\uparrow$			Virtual power plant: cost $\downarrow$
Horizon	High-U	Overall	Gain	High-U	Overall Gain	Train time
$H=\Delta$	$15767.1\pm 211.6$	$16318.7\pm 136.4$	—	$34843.1\pm 589.2$	$33581.5\pm 327.8$ —	$1.0\times$
$H=2\Delta$	$16018.4\pm 156.2$	$16512.3\pm 104.7$	$+1.19\%$	$34104.6\pm 468.3$	$33042.8\pm 271.4$ $+1.60\%$	$1.6\times$
$H=4\Delta$	$16214.7\pm 118.6$	$16625.4\pm 84.5$	$+1.88\%$	$33516.3\pm 371.5$	$32679.2\pm 223.6$ $+2.69\%$	$3.1\times$
$H=8\Delta$	$16257.3\pm 121.7$	$16663.9\pm 88.3$	$+2.12\%$	$33438.9\pm 379.4$	$32601.7\pm 231.8$ $+2.92\%$	$6.2\times$
$H=\Delta$ , $4\times$ data	$15911.2\pm 183.4$	$16402.8\pm 118.9$	$+0.52\%$	$34512.7\pm 521.6$	$33298.4\pm 296.3$ $+0.84\%$	$2.9\times$

Table 23: Continuation-horizon sweep over five runs with matched evaluation seeds. Gain is relative to

H=\Delta

on the overall score. Train time is wall-clock relative to

H=\Delta

at fixed optimiser steps. For the corridor

\Delta=3

min; for the VPP

\Delta=30

min. The last row is the compute-matched control:

H=\Delta

with four times as many snapshots per optimiser step, which issues the same number of generation calls and simulates the same number of intervals as

H=4\Delta

Withheld category	Throughput $\uparrow$	VPP cost $\downarrow$
None (ours)	$16625.4\pm 84.5$	$32679.2\pm 223.6$
Recent trends	$16308.7\pm 107.4$	$33127.5\pm 248.9$
Forecasts	$16394.2\pm 96.8$	$33218.4\pm 241.7$
Uncertainty estimates	$16471.6\pm 91.3$	$33006.8\pm 234.2$
Priorities and events	$16512.9\pm 88.6$	$32894.1\pm 229.4$

Table 24: Leave-one-out operational-context ablation, overall scores over five runs. Rows are ordered by the corridor loss.

Context supplied	Throughput $\uparrow$	VPP cost $\downarrow$
Full	$16625.4\pm 84.5$	$32679.2\pm 223.6$
$-$ forecasts	$16394.2\pm 96.8$	$33218.4\pm 241.7$
$-$ uncertainty	$16281.5\pm 112.6$	$33452.7\pm 258.6$
Measurements only	$15984.3\pm 131.8$	$34016.9\pm 287.3$

Table 25: Cumulative context stripping, overall scores over five runs. Each row removes the named category in addition to those above it; the last row also removes recent trends, priorities, and events, leaving only the current numerical measurements.

Comparison	Difference	$95\%$ interval
Throughput, vehicles per episode
vs. Feedback Control	$+1032.9$	$[+947,+1113]$
vs. Fixed-Mode Hierarchy	$+807.4$	$[+729,+881]$
vs. End-to-End RL	$+946.8$	$[+812,+1069]$
vs. Hierarchical RL	$+493.6$	$[+408,+574]$
vs. Gemini 3.1 Pro	$+965.7$	$[+884,+1041]$

Table 26: Paired scenario-level bootstrap,

10{,}000

resamples. Positive values favour our method in both tasks: more vehicles discharged, fewer CNY spent.

Corridor roles, share of all ramp-steps
	Sensitive	Standard	Sluggish	Always-Open
Ours	$31.4$	$42.7$	$21.8$	$4.1$
Qwen3-8B	$18.2$	$68.4$	$9.6$	$3.8$
VPP modes (ours), share of all resource-steps
Resource	Conservative	Balanced	Proactive
PV	$24.6$	$38.2$	$37.2$

Table 27: Mode-selection shares in percent over the

300

test scenarios. Rows sum to

100

Corridor: Sensitive / Sluggish share by occupancy trend
	Rising	Flat	Falling
Sensitive	$44.2$	$19.7$	$38.6$
Sluggish	$14.1$	$29.0$	$17.4$
VPP: proactive-battery share by tariff tier
Shoulder Peak Critical
$21.4$ $44.6$ $68.7$

Table 28: Mode shares in percent conditioned on the context variable the mode is intended to respond to. Trend bins are defined on the signed change of merge-area occupancy over the observation window and cover

27

46

and

27

percent of ramp-steps; the urgency bins cover

20

35

and

45

percent of VPP steps; the tariff bins cover

2

5

and

5

hours of the episode. Weighting each row by these proportions recovers the corresponding marginal share in Table 27 .

Phase	Bottleneck occ.	Sens.	Slug.	Open
Warm-up ( $0$ – $15$ min)	$0.11$	$0$	$0$	$0$
Rise ( $15$ – $45$ min)	$0.19$	$3$	$1$	$0$
Plateau ( $45$ – $120$ min)	$0.24$	$2$	$5$	$0$
Decline ( $120$ – $160$ min)	$0.17$	$5$	$1$	$0$
Recovery ( $160$ – $195$ min)	$0.09$	$1$	$2$	$3$

Table 29: Modal role assignment across the phases of one balanced-heavy corridor scenario; counts are out of eight controlled ramps, the remainder being Standard. Occupancy is the mean at the critical merge.

Pattern family	Baseline	Gain
Corridor, throughput vs. Feedback Control
Upstream-dominant	$16043.2$	$+2.3\%$
Ramp-dominant	$15108.6$	$+8.5\%$
Balanced moderate	$15797.4$	$+6.1\%$
Balanced heavy	$15420.8$	$+9.8\%$
VPP, cost vs. MPC-MILP

Table 30: Gain by operating-pattern family. Each family contributes

25

scenarios to each of the three subsets, so

75

of the

300

test scenarios, and Baseline is that family’s direct-control score. The overall gains of

6.62\%

and

4.37\%

in the main paper are the baseline-weighted aggregate

\sum_{f}B_{f}g_{f}/\sum_{f}B_{f}

, not the unweighted mean of the four percentages; the two differ because the families sit at different absolute levels. Equal representation does make the baseline column average to the overall direct-control scores of

15592.5

and

34171.6

Language Model-Based Control

Language model-based control uses language models to interpret goals, choose actions, and steer AI systems or physical processes.

See all 180 papers

Jun 18, 2026

Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution

Reinforcement learning (RL) has achieved strong performance in sequential decision-making, yet scaling to complex multi-agent environments remains challenging due to sparse rewards, large state-action spaces, and the difficulty of learning coordinated strategies. We propose a hierarchical architecture where a pretrained large language model (LLM) acts as a centralized strategic controller that selects among specialized RL skill policies for a team of agents, while RL policies handle reactive low-level execution. We evaluate this hybrid system in a competitive 2v2 King of the Hill environment against behavior tree (BT) and ``Flat'' RL (end-to-end training without skill decomposition) baselines. The LLM+RL system achieves task performance statistically equivalent to hand-crafted BT (46.4% vs 51.5% win rate,

p=0.103

) while both significantly outperform Flat RL trained without skill decomposition. A user study (

n=15

) reveals that 60% of participants perceive LLM+RL agents as the most human-like (

p=0.027

), citing behavioral adaptability and tactical variability. These results demonstrate that pretrained LLM reasoning can effectively orchestrate pretrained RL skills, achieving competitive multi-agent coordination and superior perceived believability without manual rule engineering.

Jannik Hösch, Alessandro Sestini, Florian Fuchs +6

1Electronic Arts (EA) · 2KTH Royal Institute of Technology, Stockholm, Sweden

Sequential MARL Multi-Agent Coordination Game-Playing Agents Hierarchical RL LLM-Guided RL Multi-Agent Planning

Ask PDF

May 9, 2026

Iterative Critique-and-Routing Controller for Multi-Agent Systems with Heterogeneous LLMs

Multi-agent large language model (LLM) systems often rely on a controller to coordinate a pool of heterogeneous models, yet existing controllers are typically limited to one-shot routing: they select a model once and return its output directly. Such routing-only designs provide no mechanism to critique intermediate drafts or support iterative refinement. To address this limitation, we propose a critique-and-routing controller that casts multi-agent coordination as a sequential decision problem. At each turn, the controller evaluates the current draft, decides whether to stop or continue, and, if needed, selects the next agent for further refinement. We formulate this process as a finite-horizon Markov Decision Process (MDP) with explicit agent-utilization constraints, design a composite reward for controller decisions across turns, and optimize the controller via policy gradients under a Lagrangian-relaxed objective. Extensive experiments across multiple heterogeneous multi-agent systems and seven reasoning benchmarks show that our method consistently outperforms state-of-the-art baselines and substantially narrows the gap to the strongest agent, while using it for fewer than 25% of total calls.

Wenzhi Fang, Liangqi Yuan, Guangchen Lan +2

Purdue University · Yonsei University

Multi-Agent LLM Systems Multi-Agent Coordination LLM Routing Multi-Agent Reasoning RL for LLM Agents

Ask PDF

Apr 10, 2026

C $^2$ T: Captioning-Structure and LLM-Aligned Common-Sense Reward Learning for Traffic--Vehicle Coordination

State-of-the-art (SOTA) urban traffic control increasingly employs Multi-Agent Reinforcement Learning (MARL) to coordinate Traffic Light Controllers (TLCs) and Connected Autonomous Vehicles (CAVs). However, the performance of these systems is fundamentally capped by their hand-crafted, myopic rewards (e.g., intersection pressure), which fail to capture high-level, human-centric goals like safety, flow stability, and comfort. To overcome this limitation, we introduce C2T, a novel framework that learns a common-sense coordination model from traffic-vehicle dynamics. C2T distills "common-sense" knowledge from a Large Language Model (LLM) into a learned intrinsic reward function. This new reward is then used to guide the coordination policy of a cooperative multi-intersection TLC MARL system on CityFlow-based multi-intersection benchmarks. Our framework significantly outperforms strong MARL baselines in traffic efficiency, safety, and an energy-related proxy. We further highlight C2T's flexibility in principle, allowing distinct "efficiency-focused" versus "safety-focused" policies by modifying the LLM prompt.

Yuyang Chen, Kaiyan Zhao, Yiming Wang +3

The State Key Laboratory of Internet of Things for Smart City, University of Macau · Wuhan University · Computer and Information Science, University of Macau +1

Traffic Signal Control Multi-Agent Coordination Cooperative MARL LLM-Guided RL

Ask PDF

Abstract

Language Model-Based Control

Explore similar work

Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution

Iterative Critique-and-Routing Controller for Multi-Agent Systems with Heterogeneous LLMs

C2^22T: Captioning-Structure and LLM-Aligned Common-Sense Reward Learning for Traffic--Vehicle Coordination

C $^2$ T: Captioning-Structure and LLM-Aligned Common-Sense Reward Learning for Traffic--Vehicle Coordination