While Large Language Models (LLMs) form the cornerstone of sequential decision-making agent development, they have inherent limitations in high-frequency decision tasks. Existing research mainly focuses on discrete embodied decision scenarios with low-frequency and significant semantic differences in state space (e.g., household planning). These methods suffer from limited performance in high-frequency decision-making tasks, since high-precision numerical state information in such tasks undergoes frequent updates with minimal fluctuations, and exhibiting policy misalignment between the learned sub-tasks and composite tasks. To address these issues, this paper proposes Normalized Action Reward guided Consistency Policy Optimization (NAR-CP). 1) Our method first acquires predefined dense rewards from environmental feedback of candidate actions via reward functions, then completes reward shaping through normalization, and theoretically verifies action reward normalization does not impair optimal policy. 2) To reduce policy misalignment in composite tasks, we use LLMs to infer sub-observation candidate actions and generate joint policies, with consistency loss ensuring precise alignment between global semantic policies and sub-semantic policies. Experiments on UAV pursuit, a typical high-frequency task, show our method delivers superior performance on independent and composite tasks with excellent generalization to unseen tasks.
Figures & tables
Figure 1: Method Overview. (a) Reduce the deviation between the sub-tasks joint policy πJ and the composite policy πF through the consistency loss. (b) By applying Action Reward Normalization to the original dense reward, we dynamically amplify the reward variance and restore the effectiveness of gradient signals.
Figure 2: Details Of The Consistency Policy. Atop-k represents the action with the highest probability extracted from the joint policy πJ . πFull-topk obtained by inputting the combination of Atop-k and composite observation prompt into the LLM. Finally employ the KL divergence to construct the consistency loss Lconsistency
Figure 3: Figure from left to right in turn shows: The reward curve in the direction-tracking task. The reward curve in the distance-keeping task. The reward curve in the Integrated-tracking task.
Figure 4: Error distribution characteristics: Direction tracking : x -axis = root sum of squared deviations, y -axis = density; red zone = 0 – 0.141 (precision), yellow zone = 0 – 0.707 (general); accuracy increases as curve approaches 0 in positive zone. Distance keeping : x -axis = current − target distance, y -axis = density; red zone = ±0.1 (precision), yellow zone = ±1.0 (general); accuracy increases as curve approaches 0. Integrated Tracking : x -axis = direction + distance error (+ = distance ¿ target, − = distance ¡ target), y -axis = density; red zone = ±0.241 (precision), yellow zone = ±1.707 (general); coordinated control increases as curve approaches 0.
Model
Direction Tracking
Distance Keeping
Integrated Tracking
General Rate %
Precise Rate %
General Rate %
Precise Rate %
General Rate %
Precise Rate %
Policy Bias
Qwen2.5-3B
31.62
4.62
25.04
0.04
2.83
0.02
0.05
Twosome
1.38
0.08
19.00
1.08
0.46
0.06
0.186
Twosome-DR
51.58
17.38
15.12
0.13
24.58
0.10
1.22
NAR-CP
96.79
91.33
76.67
60.83
82.00
61.46
0.006
Table 1: Performance comparison across different tracking scenarios with additional policy bias metric for integrated tracking.
Model
Distance Generalization
Strategy Generalization
Complex Composite Task
General Rate %
Precise Rate %
General Rate %
Precise Rate %
General Rate %
Precise Rate %
Qwen2.5-3B
9.11
0.44
3.04
0.42
0.88
0.02
NAR-CP
68.00
48.05
74.50
22.38
19.62
1.08
Table 2: Conducting generalization experiments for verification in unseen scenario.
Large Language Models (LLMs) have recently shown strong capabilities in reasoning, planning, and tool-use, enabling new forms of autonomous agents. However, LLM-based agents struggle with long-horizon sequential decision tasks that require precise action optimization and environment interaction. Reinforcement Learning (RL), while effective for sequential control, often lacks the high-level abstraction and task decomposition abilities needed for complex scenarios. This paper introduces an LLM-Augmented Reinforcement Learning Agent that integrates LLM-driven planning with RL-based action optimization. The proposed architecture leverages the LLM to generate subgoals, structured plans, and contextual guidance, while the RL agent refines low-level actions through interaction with the environment. Experiments on sequential decision tasks demonstrate improved sample efficiency, higher success rates, and more coherent action trajectories compared to RL-only and LLM-only baselines. This hybrid paradigm highlights a promising direction for building more capable autonomous systems.
Christophe D. Hounwanou, John Emeka Eze, Yaé Ulrich Gaba
Department of Mathematics, AIMS, Kigali, Rwanda · African Institute for Mathematical Sciences (AIMS), Kigali, Rwanda · African Center for Advanced Studies, Pretoria, South Africa
Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities, yet their potential for sequential decision-making remains underexplored. In this paper, we study the ICL capabilities of LLMs in sequential decision-making settings, including Markov Decision Processes (MDPs), Partially Observable MDPs (POMDPs), and Ambiguous POMDPs (APOMDPs). We fine-tune pretrained LLMs to perform few-shot decision-making directly from offline, oracle-labeled trajectories. Our framework enables flexible imitation of policies through supervised fine-tuning (SFT). Theoretically, we focus on linear MDPs and interpret a fine-tuned attention layer as implicitly estimating optimal Q-functions from in-context data. Building on this interpretation, we derive an end-to-end suboptimality bound for the induced policy that separates the in-context estimation error from the training-length bias. Empirically, across synthetic MDP, POMDP, and APOMDP settings, we find that fine-tuned LLMs achieve substantially smaller optimality gaps than in-context-only and random baselines, with especially large gains in longer-horizon, partially observed, and model-ambiguous environments. Together, these results show that supervised fine-tuning provides an effective route to endowing pretrained LLMs with sequential decision-making capabilities from offline data, which is an important advantage in domains such as healthcare where offline data are abundant.
We study when large language models (LLMs) can serve as effective black-box policy optimizers for reinforcement learning (RL) tasks, i.e., when can we replace classical RL algorithms with an LLM? We explore this question by introducing Prompted Policy Optimization (PromptPO), an iterative method that prompts an LLM with Python descriptions of the state space, action space, and reward function, then has it generate and refine executable policies based on rollout feedback. Across hard exploration environments, Meta-World robotics tasks, and several real-world control problems, PromptPO often matches or exceeds the performance of standard RL baselines while using substantially fewer environment interactions. To maximize expected return, and without further explicit prompting, the policies PromptPO outputs range from tuned proportional controllers or rule-based plans to policies that run planning algorithms like value iteration. Our results demonstrate that LLM-based policy optimization is sufficient when the LLM can leverage prior knowledge about the environment or optimization strategy. PromptPO underperforms standard RL baselines in MuJoCo domains. This demonstrates possible limitations of LLM-based policy optimization to settings that requiring fine-grained continuous control.
Stephane Hatgis-Kessell, Emma Brunskill
Department of Computer Science, Stanford University