RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs
Organizations: University of Illinois Urbana-Champaign. · National University of Singapore. · Zhejiang University. · University of Pennsylvania. ·
Abstract
Reinforcement learning (RL) has improved the capabilities of large language models (LLMs) across a wide range of tasks. However, token-level RL assigns an action's advantage to the entire generated response, including tokens specifying the parsed action, reasoning chains, and malformed content. Although the environment executes only the parsed action, variation in how the action is expressed in natural language can introduce additional variance into policy-gradient estimates. We propose RLiD (Reinforcement Learning in the Decision plane), a framework that directly optimizes the probabilities of executable action options. RLiD compiles observations, an optional compact memory, and a labeled action menu into the model input, and uses a large decision model (LDM) to select an action in a single forward pass. By eliminating autoregressive token generation and bypassing action parsing, RLiD avoids invalid action outputs and improves RL rollout efficiency. Our analysis shows that, within a fixed text policy, marginalizing over alternative realizations of the same action removes a zero-mean gradient component without changing the expected update. We further distill trained LDM policies into generative LLMs through supervision on their action trajectories. Experiments on ARC, Countdown, and Sokoban demonstrate improved performance or lower computational cost across several settings. On ALFWorld, the complete training and distillation pipeline reaches 99.3% success in 12.7 H100 GPU-hours, compared with 86.6% in 17.3 GPU-hours for token-level RL of the same LLM. With compiled memory, our strongest Qwen2.5-1.5B LDM and distilled generative LLM achieve 100% success on all 134 unseen and 140 seen ALFWorld evaluation games.
Figures & tables
| Task, backbone | Token-plane comparator | Episodes (k) | Decision | Token | GPU-h |
| One-step decisions (ARC-C, all 1,172 test questions) | |||||
| Qwen2.5-1.5B | GRPO (TRL), short rationale | 61.4 / 26.7 | 0.12 / 1.17 | ||
| Qwen3.5-2B | GRPO (TRL), 20-word rationale | 61.4 / 17.8 | 0.15–0.18 / 0.76 | ||
| Countdown-after, Qwen2.5-1.5B, frozen protocol c : test (1,024) / untouched problems (2,048, mean) | |||||
| ldm | answer only | 25.6 | / 42.2 | / 30.3 | 0.71 / 1.06 |
| ldm | 30-word rationale | 25.6 | / 42.2 | / 28.4 | 0.71 / 1.53 |
| Policy | final | GPU-h | at 2.5 GPU-h |
|---|---|---|---|
| decision plane ( ldm ) | 2.3 | ||
| ldm without letter regulariser | 5.9 | 2.9 | 6.2 |
| token plane, answer only | 2.9 | ||
| token plane, 30-word rationale | 5.9 | 5.6 | 6.2 |
| token plane, rationale + decision | 5.9 |
| Method (backbone) | success | budget |
|---|---|---|
| ALFWorld, token plane (Qwen2.5-1.5B / 7B) | ||
| PPO | / | 150 it |
| RLOO | / | 150 it |
| GRPO | / | 150 it |
| GiGPO | / | 150 it, 2 / 4 GPU |
| GiGPO | / | 500 it |
| advantage | sign | dec. share | ||
| Decision plane ( ) | ||||
| oracle | 0.88 | 1.00 | 1.000 | |
| GRPO | 0.54 | 0.52 | 1.000 | |
| GiGPO | 0.52 | 0.52 | 1.000 | |
| Token, answer only ( ) | ||||
| oracle | 0.67 † | 1.00 | / | 0.610 / 0.628 |
| Task (student) | teacher | pure RL | distill | distill+RL |
|---|---|---|---|---|
| ALFWorld (1.5B) | 98.5 [8.0] | 79.9 / 85.1 | 100 / 99.3 | 100 / 100 |
| GPU-h | 11.2 / 14.3 | – | 2.1 / 2.6 | |
| ALFWorld (7B) | 98.5 [8.0] | 90.3 100 | 100 | 100 |
| GPU-h | 17 § | – | 5.1 | |
| Sokoban (1.5B) | 74.2 [2.9] | 70.3 | ||
| GPU-h | 3.5–4 e | – | 0.8–2.5 § |
| Student | unseen | seen |
| Memory prompt | 100 | 100 |
| Memory prompt, no action list | 99.3 | 100 |
| No memory | 90.5 | 97.1 |
| Granite-3.3-2B, memory prompt | 100 | 100 |
| SmolLM2-1.7B, memory prompt | 96.3 | 99.3 |
| Qwen2.5-0.5B, memory prompt | 98.5 | 100 |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| no memory | compiled memory | |
|---|---|---|
| 2-step history | 77.6 / 70.9 (45.5) | 97.8 / 98.5 ( 79.9 ) |
| 6-step history | 93.3 / 92.5 (45.5) | 98.5 / 97.8 (44.0) |
| Env | steps | menu source | |
|---|---|---|---|
| ARC-C | 1 | 3–5 | answer options |
| Countdown | 3 | 36 | LLM-written compiler |
| Sokoban | 10 | 4 | native moves |
| FrozenLake | 10 | 4 | native moves |
| ALFWorld | 50 | 5–62 ‡ | admissible commands |
| Issue | Setting | Without fix | With fix |
|---|---|---|---|
| Too few updates per iteration | ALFWorld, Qwen3.5-2B, it. 40 | 27.6 (4 minibatches) | 45.5 (24 minibatches) |
| ALFWorld, Qwen2.5-1.5B, it. 40 | 8.2 (4) | 14.2 (24) | |
| Letter bias on fixed 4-action menus | Sokoban, Qwen2.5-0.5B, final | 17.2 (calibration) | 37.9 / 55.3 / 42.2 (marginal regulariser, 3 seeds) |
| Greedy evaluation of a stochastic policy | ALFWorld, Qwen3.5-2B, no memory, final | 70.9 (greedy) | 77.6 (sampled, ) |
| Entry | Runs |
|---|---|
| ARC ldm Qwen2.5-1.5B / Qwen3.5-2B | arc_q15_lr1e5, arc_q15_s{1,2} / arc_q35_2b_lr1e5, arc_q35_s{1,2} |
| ARC token GRPO (corrected parser) | tokrl_arcfix_q25_1.5b_s{17,1,2}, tokrl_arcfix_q35_2b_s{17,1} |
| Countdown plain / afterstate | cd_q35_2b_m, cd_q15_lr5e5 / cda_q15_lr5e5, replicates cdaR_q15_s{0..3} ( --marg-mode micro --marg-rows adv , our intended reconstruction of the original run’s regulariser), cdaX_q35_{2b,4b}; tokrl_cd_*; token Countdown-after (aligned sampling) tokrl_cdafix_q25_1.5b_s{17,1,2} (superseded: tokrl_cda_q25_1.5b); same-runner token arms mC_{ans,cot,rcd}_s{0,1} (cdaR hyperparameters plus --policy , --fmt-pen 0 ; the label regulariser acts only in the rcd path; mC_rcd_s1 run after the evaluation-RNG fix); ldm without label regulariser cdaN_q15_s{0,1} (cdaR hyperparameters with --marg 0 , policy=ldm ) |
| Countdown frozen protocol (Appendix L ) | LR grid pr_{ldm,hyb,ans,cot}lr{2e5,5e5,1e4}s0; seeds pr{arm}sel_s{1,2,3}; stop-scored hybrid --rcd-stop scored --rcd-early-stop 1 ; exclusive timing tm*; sealed split se{arm}_s{0..3}sh{0,1}; plain Countdown cdP_q15_s{0,1}, cdP_q35_2b_s0, mP{ans,rcd}s0; earlier stop-corrected hybrid mCx_rcd_s{0,1}, mCx2_rcd_s{2,3}; strict re-evaluations strict_mC*; aggregation scripts/protocol_report.py |
| FrozenLake protocol | flP_{ldm,hyb}_lr{1e5,1e4}s0, flP{ldm,hyb}_sel_s{1,2} |
| Distill (Countdown-after) | teacher dump + BC cdD24k (results/bc/cdD24k), RL cdD24k_rl; second student seed results/bc/cdD24k_seed1, cdD24k_rl_s1; teacher-KL variant cdD24k_T |
| Run | final | val-selected | selected iteration |
|---|---|---|---|
| ARC Qwen2.5-1.5B s1 | 78.4 | 78.4 | 100 |
| ARC Qwen2.5-1.5B s2 | 79.1 | 78.8 | 90 |
| ARC Qwen3.5-4B | 92.9 | 92.6 | 110 |
| ARC Qwen3.5-9B | 94.6 | 93.9 | 20 |
| Countdown-after Qwen2.5-1.5B | 60.4 | 60.4 | 400 |
| Countdown-after Qwen3.5-4B | 67.4 | 67.9 | 325 |
| Method (backbone) | success | budget |
|---|---|---|
| ldm , Qwen3.5-2B | / | 150 it, 8–9 h |
| ldm , Qwen3.5-4B | 99.3 / 99.3 | 150 it, 11 h |
| Method | selection | benchmark | budget |
| LDM continuation control | 180+200 it, 7.0+4.3 h | ||
| LDM + SPEAR replay | 180+200 it, 7.0+4.6 h | ||
| LDM + outcome curriculum | 180+200 it, 7.0+4.7 h | ||
| LDM + replay + curriculum | 180+200 it, 7.0+5.3 h | ||
| LDM + replay, from base | 200 it, 7.4 h | ||
| LDM + curriculum, from base | 200 it, 5.7 h |
| Method | holdout |
|---|---|
| LDM, pure RL | |
| LDM, pure RL (seed 0) | |
| LDM + continuation control | |
| LDM + SPEAR replay | |
| LDM + outcome curriculum | |
| LDM + SPEAR replay + curriculum |
| Student training | it. 0 | it. 50 | it. 150 | final | first | last |
|---|---|---|---|---|---|---|
| pure RL | 11.3 | 30.7 | 49.0 | 60.4 (400) | 1.68 | 1.53 |
| hard BC RL | 51.8 | 56.2 | 54.7 | 56.2 (150) | 0.84 | 0.12 |
| soft BC RL | 53.5 | 56.6 | 55.1 | 56.5 (150) | 0.74 | 0.11 |
| hard BC RL + entropy floor | 51.8 | 56.4 | 55.7 | 55.9 (150) | 0.95 | 0.13 |
| soft BC RL + entropy floor | 53.5 | 56.8 | 55.5 | 57.0 (150) | 0.85 | 0.23 |
| Backbone | policy | gen. | roll (s) | train (s) | dec/s | ldm speed-up |
|---|---|---|---|---|---|---|
| Qwen2.5-0.5B | decision plane ( ldm ) | 1 (label) | 0.5 | 1.7 | 119.2 | – |
| token, answer only | 4.3 | 1.9 | 1.8 | 69.6 | 1.7 | |
| token, 30-word rationale | 45.9 | 8.3 | 2.3 | 24.2 | 4.9 | |
| Qwen2.5-1.5B | decision plane ( ldm ) | 1 (label) | 1.1 | 3.5 | 56.2 | – |
| token, answer only | 4.3 | 3.0 | 3.7 | 37.9 | 1.5 | |
| token, 30-word rationale | 20.7 | 10.9 | 4.5 | 16.6 | 3.4 |
| Policy | cosine | rel. error | (M W) | M better |
|---|---|---|---|---|
| base, | 0.980 / 0.985 | 0.196 / 0.238 | 5/12 | |
| base, | 0.970 / 0.990 | 0.122 / 0.064 | 6/14 | |
| base, | 0.945 / 0.993 | 0.150 / 0.048 | 10/16 | |
| random init 1, | 0.848 / 0.912 | 0.319 / 0.209 | 8/16 | |
| random init 2, | 0.922 / 0.945 | 0.253 / 0.154 | 8/14 | |
| trained FrozenLake, | 0.956 / 0.975 | 0.181 / 0.137 | 8/15 |
| Advantages, rationale | seeds | final | final | area | matched steps |
|---|---|---|---|---|---|
| GRPO adv., 3-token rationale | 8 | 0.291 / 0.300 | (6) | (7) | 214/240, 215/240 |
| GRPO adv., 1-token rationale | 8 | 0.284 / 0.295 | (4) | (6) | 221/240, 214/240 |
| oracle adv., 3-token rationale | 4 | 0.304 / 0.319 | (3) | (4) | 94/120, 107/120 |
| Invalid labels | init. mass | seeds | area | final | KL | matched | skipped |
|---|---|---|---|---|---|---|---|
| none | 0 | 8 | (6) | (4) | 10.0 / 10.1 | 221 / 214 | 0 / 0 |
| 4 tokens, low bias | 20 | 8 | (6) | (7) | 10.1 / 10.0 | 222 / 221 | 0 / 0 |
| 1 token, low bias | 25 | 8 | (6) | (5) | 10.1 / 10.0 | 203 / 214 | 0 / 0 |
| 4 tokens, high bias | 61 | 8 | (6) | (5) | 7.5 / 10.1 | 163 / 204 | 35 / 0 |
| 1 token, high bias | 64 | 8 | (6) | (6) | 10.2 / 10.0 | 203 / 200 | 0 / 0 |
| Policy | valid (%) | raw | adv., valid [95% CI] | states | adv., valid, mean |
|---|---|---|---|---|---|
| Sokoban 0.5B, base, answer only | 99.1 | 70 | 68 [57, 76] | 32 | 49 |
| Sokoban 0.5B, trained, answer only | 100.0 | 0 | 0 [0, 0] | 2 | 0 |
| Sokoban 0.5B, base, rationale | 16.1 | 94 | 53 [39, 67] | 32 | 58 |
| Sokoban 0.5B, trained, rationale | 99.7 | 96 | 94 [91, 96] | 29 | 94 |
| Countdown-after 1.5B, base, answer only | 99.5 | 29 | 30 [19, 41] | 30 | 18 |
| Countdown-after 1.5B, trained, answer only | 93.7 | 2 | 0 [0, 0] | 7 | 0 |
| Arm | test | untouched | contrast | test | untouched |
|---|---|---|---|---|---|
| ldm | |||||
| hybrid | ldm hybrid | , | , | ||
| answer only | ldm answer | , | , | ||
| 30-word rationale | ldm rationale | , | , | ||
| ldm , no label reg. | no reg. answer | , | , | ||
| no reg. rationale | , | , |
| Backbone | token comparator | episodes | decision | token | GPU-h |
|---|---|---|---|---|---|
| Qwen2.5-1.5B | answer only / rationale + label (same runner) | 25.6 | (2) | 2.5 / 10.0 (1) | 0.31 / 1.25–1.75 |
| Qwen2.5-1.5B | GRPO (TRL), full expression | 102.4 / 38.4 | (2) | 14.3 (1) | 2.1 / 2.25 |
| Qwen3.5-2B | GRPO (TRL), full expression | 102.4 / 38.4 | 47.4 (1) | 50.0 (1) | 1.7 / 2.51 |
| Arm | solved | viable mass | chosen viable | (solved / failed) | rollout (s) |
|---|---|---|---|---|---|
| ldm | 108 | 0.374 | 0.421 | 0.21 / 0.74 | 11.8 |
| hybrid | 107 | 0.367 | 0.421 | 0.20 / 0.64 | 77.5 |
| Prompt (init.) | selection | unseen | seen | unseen, |
|---|---|---|---|---|
| Memory ( ldm ) | 100.0 | 100.0 (0.0) | 100.0 (0.0) | 100.0 |
| Memory (base), seed 0 | 99.2 | 100.0 | 100.0 | 100.0 |
| No memory (base) | 96.2 | 89.3 (3.4) | 95.7 | 90.5 |
| Published GiGPO | 86.7 (1.7) | 86.7 |
| Stage | GPU-h | unseen |
|---|---|---|
| Qwen3.5-2B ldm -RL from scratch, 150 it | 9.21 | |
| Teacher dump and BC into the 1.5B ldm | 0.62 | |
| 1.5B ldm -RL, 100 it | 2.29 | |
| Rollouts, 100-step SFT, evaluation | 0.59 | |
| RLiD route, total | 12.70 | 99.3 |
| Token-level GiGPO, same LLM and runner, 150 it | 17.34 | 86.6 |
| Initialisation | 25 steps | 50 steps | 100 steps | 200 steps |
|---|---|---|---|---|
| Base | 15.7 | 26.9 | 67.9 | – |
| ldm adapter | 97.8 | 97.8 | 100.0 | 100.0 |
| Teacher | student prompt | unseen | seen |
|---|---|---|---|
| Qwen3.5-2B ldm , with memory | memory | 99.3 | 100.0 |
| Refined 1.5B ldm , with memory | memory | 100.0 | 100.0 |
| Qwen3.5-2B ldm , trained without memory | no memory | 78.4 | 82.9 |
| Refined 1.5B ldm , with memory | no memory | 91.8 | 96.4 |
| Selector | selection | unseen |
|---|---|---|
| None (greedy, one sample) | 96.1 | 96.3 |
| First admissible proposal | 96.5 | 94.0 |
| Most frequent admissible proposal | 98.0 | 97.8 |
| ldm ’s highest-probability proposal | 99.6 | 100.0 |
| Run | it 10 | it 20 | it 30 | GPU-h at 30 |
|---|---|---|---|---|
| Bonus, seed 0 | 2.7 | 9.8 | 26.6 | 4.81 |
| Bonus, seed 1 | 3.1 | 8.2 | 20.7 | 5.11 |
| GiGPO, seed 0 | 2.7 | 5.1 | 10.9 | 4.16 |
| GiGPO, seed 1 | 0.8 | 0.0 | 0.4 | 4.15 |
| Iteration | train success | malformed | non-admissible | episode length |
|---|---|---|---|---|
| 1 | 0.0 | 94.7 | 91.2 | 50.0 |
| 4 | 1.6 | 0.6 | 72.4 | 49.4 |
| 10 | 7.8 | 0.0 | 53.4 | 46.8 |
| 30 | 28.9 | 0.0 | 38.8 | 39.3 |
| 70 | 73.4 | 0.0 | 12.0 | 24.7 |
| 150 | 91.4 | 0.0 | 7.4 | 14.4 |
| Policy | GSM8K (500) | ARC-C (1,172) |
|---|---|---|
| Base | 74.6 | 75.3 |
| Memory prompt, ldm init. | 73.8 | 75.1 |
| Memory prompt, base init. | 73.4 | 75.6 |
| No memory, base init. | 75.8 | 75.8 |
| Policy | eval | OOD |
|---|---|---|
| ldm teacher | 82.8 | 35.9 |
| Generative SFT, base init. | 83.2 | 29.5 |
| Generative SFT, ldm init. | 83.4 | 34.4 |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,862 papers