A Schema Bounded Language Model for Refining Robot Policies Without Destabilizing Local Learning
Authors: Chongwen Dong, Mithun Paul Saint-Germain, Pinjari Asif, Carlo R. daCunha
Organizations: Department of Mechanical Engineering Northern Arizona University 15600 S. McConnell Dr., Flagstaff, AZ 86011 · School of Informatics, Computing, and Cyber Systems Northern Arizona University 1295 Knoles Dr., Flagstaff, AZ 86011 · Helen and John C. Hartmann Department of Electrical Engineering and School of Applied Engineering and Technology New Jersey Institute of Technology 323 Martin Luther King Jr. Blvd., Newark, NJ 07102
This paper addresses navigation by composite heterogeneous robots in a decentralized system when policy reasoning and local control operate at different update levels. In a NetLogo--Python implementation, three robots share motion dynamics but use different LLM backends. Each robot independently combines a large language model (LLM) policy agent, an Upper Confidence Bound (UCB) bandit, and a Double Deep Q-Network (Double DQN) controller; no central LLM generates team actions. LLM inference is confined to round-level policy generation and refinement rather than tick-level action selection. The robots perform cross-LLM communication through a shared round summary containing policies, outcomes, and learning feedback. UCB performs refinement-mode selection, and the policy-conditioned Double DQN performs tick-level action selection from navigation variables, active policy parameters, and the LLM action prior. Each of the four configurations was evaluated over 30 rounds. In the fixed simulation, the complete configuration reached the goal in all 90 correlated robot--round records and achieved the lowest median completion time (42 ticks) and P90 (73.2 ticks); its median was 25.0--39.1% lower than those of the other configurations. These observations provide descriptive, configuration-level evidence from the evaluated configurations.
Figures & tables
Fig. 1: Information flow within composite robot ri . Numbered arrows keep the round-level policy path visually separate from the tick-level Double DQN loop; all four components and their persistent states are owned by the same robot.
Fig. 2: Representative communication topology. The board is common to all robots, whereas policy generation, UCB1, Double DQN, and turtle execution remain robot-local. Solid arrows denote round-level information exchange; the dashed arrow denotes the robot-local tick-level environment loop.
Fig. 3: Implemented control and update cycle. Once all turtles are terminal, tick-level Double DQN learning is followed by completed-round history and UCB updates. Shared-board construction and parallel LLM regeneration occur after every non-final round. The final round terminates before board construction or another LLM call.
Component
Activation condition
Value
Base
Every executed transition
−0.01
Away
dt+1>dt+εd
−0.02
Near-away
dt≤dnear and away
−0.05
Corrective turn
at=TURN_ONLY , ∣et∣>α ,
+0.005
∣et+1∣<∣et∣−εe
Other turn-only
TURN_ONLY and corrective condition false
−0.01
TABLE I: Implemented Tick-Level Reward Components
Configuration
Shared board
UCB
Tick-level control
Input dim.
Prior bias
C1: Local LLM–DQN
No
No
Vanilla DQN
5
No
C2: Cross-LLM–UCB + DQN
Yes
Yes
Vanilla DQN
5
No
C3: Cross-LLM–UCB, no DQN
Yes
Yes
Regional rule
–
–
C4: Complete configuration
Yes
Yes
Double DQN
15
0.05
TABLE II: Controller Configurations
Fig. 4: Representative NetLogo rollout with three composite heterogeneous robots and the common goal patch. Text labels show the active runtime action and local best-tick record for that displayed state.Each robot is connected to an LLM, specifically, LLama3.1, Phi4, Llama3.3 respectively.
Fig. 5: Completion ticks over 30 rounds for the four bundled configurations. Each curve corresponds to one fixed composite heterogeneous robot, and the dashed line is the pooled median. All panels use the same piecewise-linear vertical scale, which expands 0–100 ticks and compresses 100–300 ticks. The break marks the change of scale; values above 300 ticks are placed at the upper boundary and labeled with their actual values, while the C2 timeout is marked as TO .
Fig. 6: Successful robot–round completion ticks by configuration. Boxes span the IQR, horizontal bars show the median, whiskers span the 5th–95th percentiles, and lightly jittered points show individual successful observations. The C2 timeout is excluded here and retained in Table III and the worst-decile cost.
Metric
C1
C2
C3
C4
Success (SR %)
90/90 (100.0)
89/90 (98.9)
90/90 (100.0)
90/90 (100.0)
Median [IQR]
69.0 [49.3,87.8]
56.0 [48.0,66.0]
57.0 [51.3,64.0]
42.0 [40.0,59.8]
Mean
91.5
72.7
64.0
56.3
P90
105.8
89.2
89.0
73.2
Early
98
87
46
57
Late
60
48
78
41
TABLE III: Overall performance over 30 rounds. Completion-time statistics exclude timeouts; P90 is the 90th percentile of successful completion ticks. Early and late medians pool successful observations from rounds 1–5 and 26–30.
Config.
LBSE ↑
WDC0.10 (ticks) ↓
C1
0.310
330.0
C2
0.339
313.3
C3
0.339
127.3
C4
0.431
151.8
TABLE IV: Complementary efficiency and slow-tail summaries.
Config.
Llama-3.3
Phi-4
Llama-3.1
C1
57.5
80.0
46.5
C2
55.0
57.0
48.0 †
C3
56.5
57.0
56.0
C4
46.5
35.0
54.5
TABLE V: Successful completion-tick medians by composite robot.