SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Grounding

Renos Zabounidis; Yue Wu; Simon Stepputtis; Woojun Kim; Yuanzhi Li; Tom Mitchell; Katia Sycara

cs.LGMar 10, 2026

SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Grounding

Authors: Renos Zabounidis, Yue Wu, Simon Stepputtis, Woojun Kim, Yuanzhi Li, Tom Mitchell, Katia Sycara

Organizations: School of Computer Science, Carnegie Mellon University · Department of Mechanical Engineering, Virginia Tech

Abstract

LM-based agents excel when given high-level action APIs but struggle to ground language into low-level control. Prior work has LLMs generate skills or reward functions for RL, but these one-shot approaches lack feedback to correct specification errors. We introduce SCALAR, a bidirectional framework coupling LLM planning with RL through a learned skill library. The LLM proposes skills with preconditions and effects; RL trains policies for each skill and feeds back execution results to iteratively refine specifications, improving robustness to initial errors. Pivotal Trajectory Analysis corrects LLM priors by analyzing RL trajectories; Frontier Checkpointing optionally saves environment states at skill boundaries to improve sample efficiency. On Craftax, SCALAR achieves 88.2% diamond collection, a 1.9x improvement over the best baseline, and reaches the Gnomish Mines 9.1% of the time where prior methods fail entirely.

Figures & tables

Figure 1: Overview of SCALAR’s LLM-RL loop. The LLM reads the game manual and proposes candidate skills as symbolic operators with preconditions and effects ( 1 ). Skills whose preconditions are reachable from the current frontier are marked feasible and added to the skill graph; infeasible skills are deferred ( 2 ). Given a target skill, a planner sequences existing skills to reach a state satisfying its preconditions; an RL agent trains a policy for the target skill by executing these prerequisite skills and then learning from its own exploration ( 3 ). After training, trajectory analysis examines successful executions to verify or correct the LLM’s initial specification of preconditions and resource requirements ( 4 ). Revised specifications update the skill graph and inform subsequent LLM proposals. This loop repeats until the target skill achieves sufficient success rate.

Baselines
	Craftax-Classic (%)		Craftax (%)
Method	Eat Plant	Collect Diamond	Enter Dungeon	Unlock Gnomish Ladder	Enter Gnomish Mines
PPO-FC	23.9	35.4	86.6	1.8	0.0
PPO-RND	21.9	34.5	87.4	1.9	0.0
PPO-RNN	0.0	40.9	89.7	0.0	0.0
PPO-TRXL	0.0	31.5	96.5	0.4	0.0

Table 1: Success rates on Eat Plant and Collect Diamond, the two most difficult long-horizon tasks in Craftax-Classic, and Enter Dungeon, Unlock Gnomish Ladder, and Enter Gnomish Mines, the three deepest achievements in Craftax. Results at 1B frames, averaged over 5 seeds. Ablation rows show cumulative removal of components.

Figure 2: Training curves on Craftax-Classic (left two panels) and Craftax (right three panels). X-axis: training frames in millions. Y-axis: success rate (%) over 256 evaluation episodes. Lines show mean over 5 seeds; shaded regions show standard error. SCALAR (black solid) and SCALAR-TRXL (blue solid) are our method; dashed lines are PPO variants; dotted lines are PQN variants.

Figure 3: Resources collected before attempting diamond, comparing SCALAR with and without trajectory analysis. Bars show total wood, stone, and iron gathered. Left bars: LLM’s initial estimates. Right bars: after trajectory analysis refines requirements.

Modification	PPO	Scalar	+ Traj.
Baseline	34.8	88.1	88.1
Iron Pick (2 Iron)	5.1	5.4	86.1
Iron Pick (2 Wood)	14.8	61.0	81.8
Iron Sword (3 Iron)	3.1	88.1	87.0
Table (3 Wood)	32.8	0.7	86.7

Table 2: Diamond collection success (%) under modified crafting recipes. Each row changes one recipe from the default (e.g., Iron Pickaxe requires 2 iron instead of 1).

Figure 4: Fraction of training frames spent on the target skill (y-axis) for different reset probabilities

p\in\{0,0.5,0.9,0.99\}

. Four panels: Craftax-Classic (Collect Diamond) and Craftax (Enter Dungeon, Kill 8 Orcs, Enter Gnomish Mines). Higher

p

means more frequent resets to saved frontier states.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Comparison of Craftax-Classic and Craftax environments. Craftax-Classic is shown on the left; Craftax on the right.

Figure 6: Comparison of SCALAR-FC and SCALAR-TRXL across four tasks. Row 1: Overall task success rate. Row 2: Number of skill terminations per training batch. Row 3: Frames spent at the target skill per episode. Row 4: Success rate conditioned on reaching the target skill.

Figure 7: Fraction of training frames spent on the target skill over training. Four panels show Craftax-Classic Collect Diamond and Craftax Enter Dungeon, Kill 8 Orcs, and Enter Gnomish Mines. Solid lines show results with trajectory analysis; dashed lines show results without trajectory analysis.

Figure 8: Effect of reset probability $\alpha_{\mathrm{reset}}$ on training across four tasks. Row 1: Overall success rate. Row 2: Success rate conditioned on reaching the target skill. Row 3: Frames per episode at the target skill. Row 4: Skill terminations per batch. Higher reset values are shown in red with $\alpha=0.999$ and are essential for tasks with long prerequisites such as Kill 8 Orcs and Enter Gnomish Mines, while shorter tasks like Collect Diamond are insensitive to reset probability.

Figure 9: Train-eval gap, measured as training accuracy minus evaluation accuracy, across reset probabilities. At

\alpha=0.999

shown in red, a gap of 12–18% emerges across all tasks; lower reset probabilities maintain near-zero gaps.

Figure 10: Effect of graduation threshold $\alpha_{\mathrm{grad}}$ across four tasks. Lower thresholds shown in red with $\alpha=0.1$ advance sooner; higher thresholds shown in blue with $\alpha=0.5$ require more mastery. Row 1: Overall success. Row 2: Transitions per batch. Row 3: Skill duration. Row 4: Eval skill success.

Figure 11: Effect of network sharing across four tasks. Red with 0 shared layers uses separate networks per skill; blue with 3 shared layers uses a single network; intermediate colors share 1–2 layers. Row 1: Skill success. Row 2: Transitions. Row 3: Skill duration.

Figure 12: All achievement rates on Craftax-Classic with goal Collect Diamond. SCALAR achieves high rates on task-relevant achievements while avoiding orthogonal ones like sword crafting.

Figure 13: All achievement rates on Craftax with goal Enter Gnomish Mines. SCALAR skips iron tool crafting entirely, using stone tools and arrows instead.

Figure 14: Success rate and episode length across all tasks. Top: Craftax-Classic. Bottom: Craftax. SCALAR’s longer episodes reflect task completion; baselines have shorter episodes because they fail early.

Hyperparameter	PPO-FC/RNN/RND	PPO-TRXL
Learning rate	$2\times 10^{-4}$	$2\times 10^{-4}$
Discount factor ( $\gamma$ )	0.99	0.999
GAE lambda ( $\lambda$ )	0.8	0.8
Clip coefficient ( $\epsilon$ )	0.2	0.2
Entropy coefficient	0.01	0.002
Value function coefficient	0.5	0.5

Appendix

Table 3: PPO hyperparameters. All methods use linear learning rate annealing and optimistic resets with ratio 16.

Hyperparameter	PQN-FC	PQN-RNN
Learning rate	$1\times 10^{-4}$	$3\times 10^{-4}$
Discount factor ( $\gamma$ )	0.99	0.99
TD-lambda ( $\lambda$ )	0.0	0.5
$\epsilon$ -greedy (start $\rightarrow$ end)	$0.1\rightarrow 0.005$	$1.0\rightarrow 0.005$
Maximum gradient norm	1.0	0.5
Update epochs	1	4

Appendix

Table 4: PQN hyperparameters. Both use 1024 parallel environments,

10^{9}

total timesteps, and linear LR annealing.

LLM-Guided RL

Large language model-guided reinforcement learning uses language models to shape rewards, exploration, planning, and policy learning.

See all 140 papers

LeafSkill: Efficient Leaf-Level Action Grounding for Recursive LLM Agents
Oct 23, 2026Shen Liu, Ze Zhou, Zhenguo Xu +3LLM Agent Skill Retrieval LLM Grounding
North China Institute of Computer System Engineering, Beijing, China · Beijing University of Aeronautics and Astronautics, Beijing, China · University of Science and Technology of China, Hefei, China +1
SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
Feb 9, 2026Peng Xia, Jianwen Chen, Hanyang Wang +10LLM Agent Skill Learning RL for LLM Agents
UNC-Chapel Hill · University of Chicago · University of California San Diego +3
SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinement
Jun 9, 2026Srishti Gautam, Arjun Radhakrishna, Sumit GulwaniLLM Self-Refinement LLM Agent Self-Improvement
Microsoft

Oct 23, 2026J/K move · Enter open · S save

LeafSkill: Efficient Leaf-Level Action Grounding for Recursive LLM Agents

Shen Liu, Ze Zhou, Zhenguo Xu, Shaopu Wang+2

North China Institute of Computer System Engineering, Beijing, China · Beijing University of Aeronautics and Astronautics, Beijing, China · University of Science and Technology of China, Hefei, China+1

LLM Agent Skill Retrieval LLM Grounding Long-Horizon LLM Agents

Open paper AskPDF arXiv

Recursive large language model (LLM) agents solve long-horizon tasks through decomposition, execution, and environmental feedback. Yet after producing an atomic subtask, they often invoke the same large model again to translate a familiar instruction into an executable action. We observe that high-level planning is task-specific, whereas leaf-level action grounding recurs across tasks and can be checked against environment-admitted actions. We introduce LeafSkill, an external reuse layer that retrieves semantically related grounding experience, binds action templates to current objects and admissible actions, and uses a lightweight 1.7B selector to choose a bound action or defer the unchanged subtask to the original agent. Skills remain outside the model parameters, allowing a frozen selector to use post-training additions. Under lexical and syntactic paraphrases, exact matching retrieves none of the reference actions, while semantic retrieval preserves 96.0% and 97.0% candidate coverage; the frozen selector adopts 97.34% of unseen skill types. On an independent confirmation set of 200 Robotouille tasks, LeafSkill completes 137 tasks versus 121 for ReCAP (68.5% vs. 60.5%, exact McNemar p=.0259). On the 106 tasks completed by both, it reduces Qwen3.5-27B tokens by 14.42% and calls by 14.08%. A separate held-out repeat yields a consistent 9.0-point success gain and 18.98% token reduction. Reusable leaf grounding can thus improve task completion while reducing host-model inference without replacing high-level planning.

Abstract

LLM-Guided RL

Explore similar work

LeafSkill: Efficient Leaf-Level Action Grounding for Recursive LLM Agents

SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinement