SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewards

Dengjia Zhang; Xiaoou Liu; Lu Cheng; Yaqing Wang; Kenton Murray; Hua Wei

cs.LGFeb 24, 2026

SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewards

Authors: Dengjia Zhang, Xiaoou Liu, Lu Cheng, Yaqing Wang, Kenton Murray, Hua Wei

Organizations: Johns Hopkins University, Baltimore MD, USA · Arizona State University, Tempe AZ, USA · University of Illinois Chicago, Chicago IL, USA · Purdue University, West Lafayette IN, USA

Abstract

Large language models (LLMs) are increasingly deployed as multi-step decision-making agents, where effective reward design is essential for guiding learning. Although recent work explores various forms of reward shaping and step-level credit assignment, a key signal remains largely overlooked: the intrinsic uncertainty of LLMs. Uncertainty reflects model confidence, reveals where exploration is needed, and offers valuable learning cues even in failed trajectories. We introduce SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewards, a reinforcement learning framework that incorporates uncertainty directly into the reward design. SELAUR integrates entropy-, least-confidence-, and margin-based metrics into a combined token-level uncertainty estimate, providing dense confidence-aligned supervision, and employs a failure-aware reward reshaping mechanism that injects these uncertainty signals into step- and trajectory-level rewards to improve exploration efficiency and learning stability. Experiments on two benchmarks, ALFWorld and WebShop, show that our method consistently improves success rates over strong baselines. Ablation studies further demonstrate how uncertainty signals enhance exploration and robustness.

Figures & tables

Figure 1: Overview of our method, SELAUR. (a) Uncertainty Estimation : We employ three approaches to estimate model uncertainty: entropy, confidence, and margin, derived from token probability distributions. (b) Step and Trajectory Aggregation : Throughout the model’s process, we compute two types of rewards that contribute to the uncertainty reward. At the step level, token-level uncertainties are aggregated. At the trajectory level, step-level uncertainties are combined with varying weights, where later steps receive higher weights in the overall trajectory reward. (c) Failure-aware Reward Reshaping : For successful cases, the model is trained with the standard reward. For failed cases, we incorporate the uncertainty reward into the training loop to improve learning robustness.

Base Model	Method	AlfWorld							WebShop
Base Model	Method	Pick	Look	Clean	Heat	Cool	Pick2	All	Score	Succ.
Qwen2.5-1.5B	prompting	0.1071	0.0000	0.0000	0.071	0.0000	0.0000	0.0312	0.2180	0.0429
	PPO	0.9142	0.3750	0.8333	0.8333	0.7368	0.7916	0.8046	0.7126	0.4921
	RLOO	0.9000	0.7000	0.9200	0.7142	0.6875	0.5000	0.7812	0.8022	0.6394
	GRPO	0.6756	0.5000	0.8000	0.9285	0.6190	0.5263	0.6796	0.8359	0.6718
	GiGPO	0.9143	0.7500	1.0000	0.8889	0.8421	0.7916	0.8828	0.8017	0.6757

Table 1: Main experimental results on ALFWorld and WebShop . For ALFWorld , we report success rates (%) on individual tasks and their overall average (All). For WebShop , we report both the task score and success rate (%). Bold numbers mean the best performance, and italic numbers mean the second-best. Overall, our method achieves the strongest performance across both benchmarks.

Setting			Task Score	Success Rate
entropy	least confidence	margin	Task Score	Success Rate
✗	✗	✗	0.8017	0.6757
✓	✗	✗	0.8504	0.7265
✗	✓	✗	0.8465	0.7226
✗	✗	✓	0.7834	0.6328
✓	✓	✗	0.8263	0.6953

Table 2: Each row evaluates a specific combination of metrics: ✓ indicates the inclusion of an uncertainty metric in the selection process, while ✗ denotes its exclusion. Results show that while individual or pairwise metrics provide partial gains, the full integration of Entropy, Least Confidence, and Margin ( ✓ across all) yields the optimal performance in both Task Score and Success Rate.

Figure 2: Entropy change during training under different uncertainty strategies. SELAUR maintains higher entropy throughout training, indicating stronger exploration compared to other methods.

Reward Method	AlfWorld	WebShop
Baseline	0.8828	0.6757
Negative	0.8516	0.7539
Exponential	0.8750	0.7031
Ours (SELAUR)	0.8906	0.7656

Table 3: Comparison of different reward functions on WebShop and AlfWorld . We report average of success rate(%) on AlfWorld and WebShop. Our proposed SELAUR achieves the best performance, demonstrating a better trade-off between exploration and exploitation.

Figure 3: Comparison of action traces in the WebShop environment. Top : SELAUR leverages uncertainty to explore alternative paths rather than being trapped in a single incorrect trajectory, eventually discovering the correct solution. Bottom : GiGPO exhibits low uncertainty, leading to repetitive and fixed behaviors.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 608 papers

Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning
Jun 5, 2026Yijin Zhou, Linqian Zeng, Xiaoya Lu +4Reinforcement Learning LLM Uncertainty Estimation
Shanghai Jiao Tong University, China · Shanghai Artificial Intelligence Laboratory, China · Shanghai Innovation Institute, China
Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning
Jun 4, 2026Ujjwal Bhatta, Utsabi Dangol, Sumaly Bajracharya +2Reinforcement Learning LLM-Guided RL
USD AI Research, Department of Computer Science University of South Dakota Vermillion, SD 57069, USA
RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models
Feb 27, 2026Daniel Yang, Samuel Stante, Florian Redhardt +5Reward Modeling Reward Model Evaluation
ETH Zurich · ETH AI Center

Jun 5, 2026J/K move · Enter open · S save

Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning

Yijin Zhou, Linqian Zeng, Xiaoya Lu, Wenyuan Xie+3

Shanghai Jiao Tong University, China · Shanghai Artificial Intelligence Laboratory, China · Shanghai Innovation Institute, China

Reinforcement Learning LLM Uncertainty Estimation LLM Agent Reliability RL for Tool Use Uncertainty Estimation in RL

Open paper AskPDF arXiv

Large language model (LLM)-based agents often make suboptimal tool-use decisions, including unsupported tool invocation and hallucinated direct responses, which may accumulate errors throughout multi-step interactions. Existing approaches mainly improve these behaviors through inference-time correction or coarse-grained reward signals based on decision outcomes and structured checklists, leaving the uncertainty characteristics of agent decisions underexplored. We observe that decision-oriented reinforcement learning tends to weaken the uncertainty separation between correct and incorrect actions, resulting in overconfident mistakes and weaker exploration signals. Therefore, we propose TRUST, which incorporates uncertainty quantification into reward design as a repulsive force for maintaining uncertainty separation, and labels lightweight key-turn annotations for unified post-training of multi-turn trajectories. Experimental results across diverse tool-use benchmarks show that TRUST consistently enhances both decision quality and agent performance while maintaining more reliable uncertainty estimates during optimization.

Abstract

RL for LLM Agents

Explore similar work

Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning

Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning

RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models