Reinforcement Learning (RL) agents often struggle with efficiency and performance in complex environments. We propose a novel framework that uses a Large Language Model (LLM) to dynamically generate a curriculum over available actions, enabling the agent to incorporate each action individually. We apply this framework to the game of Blackjack, where the LLM creates a multi-stage training path that progressively introduces complex actions to a Tabular Q-Learning and a Deep Q-Network (DQN) agent. Our evaluation in a realistic 8-deck simulation over 10 independent runs demonstrates significant performance gains over standard training methods. The curriculum-based approach increases the DQN agent's average win rate from 43.97% to 47.41%, reduces the average bust rate from 32.9% to 28.0%, and accelerates the overall workflow by over 74%, with the agent's full training completing faster than the baseline's evaluation phase alone. These results validate that LLM-guided curricula can build more effective, robust, and efficient RL agents.
Figures & tables
Figure 1. The adaptive, LLM-guided curriculum learning framework. (1) Generation & Adaptation: An LLM generates an initial curriculum stage. After a training phase, the agent’s performance summary is fed back to the LLM, which adapts the curriculum by deciding whether to advance the agent to the next stage. (2) Training Loop: The agent’s training is constrained by the current curriculum stage, which masks unavailable actions in the environment to focus exploration. A flowchart showing an adaptive loop. In phase 1, an LLM generates a curriculum. In phase 2, an RL agent trains in an environment constrained by that curriculum. A feedback arrow shows that the agent's performance summary from phase 2 is sent back to the LLM in phase 1, which then updates the curriculum.
Figure 2. A statistical summary of agent performance over 10 independent runs in the 8-deck curriculum environment. The results highlight the DQN agent’s significantly higher peak performance distribution (left) and confirm that this peak is most frequently achieved at Stage 4 (right). The high stage completion rate for both agents (middle) indicates the curriculum was well-paced and successfully navigated in most runs. Statistical summary charts of agent performance across 10 runs.
Figure 3. Win rate progression for baseline DQN and Tabular agents trained without a curriculum. The DQN agent’s performance is highly volatile, illustrating the instability and exploration challenges that arise when the full action space is introduced at once. Win rate progression chart for baseline agents showing high volatility.
Figure 4. Win rate progression for DQN and Tabular agents trained with the LLM-guided curriculum. The curriculum provides a stable learning trajectory for the DQN agent, preventing the performance volatility seen in baseline training and allowing it to converge to a high and consistent win rate. Win rate progression chart for curriculum-trained agents showing a stable learning trajectory.
Figure 5. Deeper analysis of agent performance, showing a positive correlation between the number of curriculum stages completed and the best achieved win rate for the DQN agent (left). However, the agent’s final performance is often lower than its peak (middle), reinforcing the finding, quantified in the summary (right), that the optimal policy is achieved at an intermediate stage. Charts showing correlation between completed stages and win rates.
Stage
Available Actions
DQN (%)
Tabular (%)
1: Hit/Stand
0,1
42.24
39.74
2: +Double
0,1,2
42.37
38.72
3: +Split (no Double)
0,1,3
45.54
38.78
4: Full Basic (0–3)
0,1,2,3
49.42
38.15
5: +Insurance
0,1,2,3,5
48.66
40.00
6: +Surrender
0,1,2,3,4
46.57
37.78
Table 1. Win rate by curriculum stage (8-deck). The values represent the best observed win rate for a single run at each stage.
Figure 6. A strategy heatmap for an agent trained without a curriculum. Rows are the player’s sum, columns are the dealer’s up card. Cell colors represent the chosen action. The strategy appears less structured than the curriculum-trained agent. Strategy heatmap showing unstructured decision boundaries for the baseline agent.
Figure 7. A strategy heatmap for an agent trained with a curriculum. Rows are the player’s sum, columns are the dealer’s up card. Cell colors represent the chosen action. The strategy appears more structured than the no curriculum-trained agent. Strategy heatmap showing structured decision boundaries for the curriculum agent.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8. Performance progression of the DQN agent through a 5-stage curriculum in an 8-deck game. DQN agent performance progression through 5 curriculum stages.
Figure 9. Comparative analysis in a 1-deck environment. Comparative analysis chart for 1-deck environment.
Figure 10. Comparative analysis in a 4-deck environment. Comparative analysis chart for 4-deck environment.
Figure 11. Comparative analysis in an 8-deck environment. Comparative analysis chart for 8-deck environment.
Figure 12. Comparative analysis in an infinite-deck environment. Comparative analysis chart for infinite-deck environment.
Figure 13. Heatmap visualization of the DQN agent’s learned policy at Stage 3 ("Introducing Splits"). Heatmap of learned policy at Stage 3.
Automatic curriculum learning can improve the effectiveness of reinforcement learning by selecting the training experiences presented to the agent over time. Predicting the consequences of such decisions can, however, be difficult. We analyze automatic curriculum learning as a sequential decision-making problem, highlighting a gap between the quantities that determine the value of curriculum decisions and the information captured by local learning signals commonly used to guide them. We then investigate whether Large Language Models (LLMs) can exploit richer information about the learning problem to better anticipate the consequences of curriculum decisions. We introduce a method that combines online learning-progress estimates with LLM-informed estimates of (i) the potential downstream benefits of learning on each task and (ii) whether direct training on a task is currently likely to produce progress. We evaluate the approach on a custom benchmark of 256 textual goals in Craftax under different curriculum objectives. We observe the strongest gains when optimizing for individual target tasks. When optimizing across the full task set, the benefits vary across learners with different mechanisms for cross-task transfer, ranging from modest improvements in learning speed to larger gains that persist through the end of training.
Reinforcement learning (RL) has emerged as a viable recipe for training LLM agents to reason over external memory banks in multi-session dialogue. Existing work trains exclusively on a single benchmark, leaving open how the composition of training data shapes the skills a memory agent acquires. We present a controlled empirical study that holds architecture, RL algorithm, and all hyperparameters fixed and varies only the training curriculum across three conditions: in-domain (LoCoMo), mixed-benchmark (LoCoMo + LongMemEval), and out-of-domain (LongMemEval only). Across two benchmarks and ten question types, curriculum composition acts as a fine-grained lever on specialization rather than a uniform scaling factor on performance. The mixed curriculum yields the strongest overall F1 on both evaluation sets. Training on a narrow out-of-domain set transfers a targeted skill - temporal reasoning - despite weak aggregate performance. Per-type differences substantially exceed aggregate differences, indicating that single-number benchmark comparisons systematically underreport curriculum effects. We further report two practical lessons from adapting GRPO to a single-GPU regime: cross-benchmark mixing requires filtering format-specific noise from memory banks to preserve training signal, and binary exact-match reward produces no learning signal at the small group sizes (G = 4) required on one GPU, motivating continuous reward functions in this regime.
Xinjie He, Zhiyuan Lin, Su Liu +4
Columbia University · Independent Researcher · Johns Hopkins University +1
Post-training large foundation models with reinforcement learning typically relies on massive and heterogeneous datasets, making effective curriculum learning both critical and challenging. In this work, we propose ACTOR-CURATOR, a scalable and fully automated curriculum learning framework for reinforcement learning post-training of large language models (LLMs). ACTOR-CURATOR learns a neural curator that dynamically selects training problems from large problem banks by directly optimizing for expected policy performance improvement. We formulate problem selection as a non-stationary stochastic bandit problem, derive a principled loss function based on online stochastic mirror descent, and establish regret guarantees under partial feedback. Empirically, ACTOR-CURATOR consistently outperforms uniform sampling and strong curriculum baselines across a wide range of challenging reasoning benchmarks, demonstrating improved training stability and efficiency. Notably, it achieves relative gains of 28.6% on AIME2024 and 30.5% on ARC-1D over the strongest baseline and up to 80% speedup. These results suggest that ACTOR-CURATOR is a powerful and practical approach for scalable LLM post-training.