Reinforcement Learning (RL) for Large Language Model (LLM) agents in multi-turn, long-horizon environments remains challenging due to inefficient action selection and repetitive behavioral patterns. In particular, agents often become trapped in high-confidence yet unproductive behaviors, repeatedly executing invalid actions or entering action loops that hinder task completion. We observe that these frequently violate environment preconditions, revealing a mismatch between the agent's representation of the internal state and the dynamics of the underlying environment. In addition, repetitive high-confidence actions lead to cyclical behaviors that provide little to no progress towards the goal. We argue that addressing these challenges requires more productive task-aligned exploration to improve the agent's understanding of environment dynamics. Motivated by intrinsic curiosity-driven exploration, we propose a conditional state-space exploration framework that balances task completion with intrinsic information-seeking behavior. Specifically, we introduce an intrinsic reward that penalizes actions whose observation duplicates prior state memory. This provides the benefit of reducing redundant behaviors and encouraging state-space coverage without relying on a general information-theoretic measure. Strong experimental results demonstrate that our approach substantially improves policy learning in multi-turn environments, mitigating invalid actions and repetitive loops.
Figures & tables
Figure 1: Failure modes of an RL-trained LLM agent in ALFWorld . (a) Repetitive action loop. The agent repeats an invalid action and receives the same failure feedback until the step budget runs out. (b) Repetitive reasoning loop. The agent’s reasoning ignores the current observation, and the agent cycles between the fridge and the microwave without completing the task. In both cases the agent’s internal state diverges from the environment dynamics, and the looping steps consume interaction budget while revealing no new observation. These are the steps our intrinsic reward penalizes.
Type
Method
ALFWorld
WebShop
Pick
Look
Clean
Heat
Cool
Pick2
All
Score
Succ.
Closed-Source Model
Prompting
GPT-4o
75.3
60.8
31.2
56.7
21.6
49.8
48.0
31.8
23.7
Prompting
Gemini-2.5-Pro
92.8
63.3
62.1
69.0
26.6
58.7
60.3
42.5
35.9
Qwen2.5-1.5B-Instruct
Prompting
Qwen2.5
5.9
5.5
3.3
9.7
4.2
0.0
4.1
23.1
5.2
Table 1: Performance on ALFWorld and WebShop . Following GiGPO, for ALFWorld environment we report the average success rate (%) for each subtask along with the overall result. For WebShop environment, we report the average score and the average success rate (%). Prompting/PPO/RLOO/GRPO/GiGPO rows are quoted from Feng et al. (2025) ; ProxMO from Fang et al. (2026) .
Method
ALFWorld
WebShop
Mean Episode Length
Invalid Action Rate
Repetitive Action Rate
Mean Episode Length
Invalid Action Rate
Repetitive Action Rate
GiGPO
32.27
4.08
45.31
5.53
0.14
43.25
Ours
22.02
2.79
11.53
5.18
0.09
40.83
Table 2: Exploration efficiency on ALFWorld and WebShop . We compare our method with GiGPO using three metrics: mean episode length, invalid action rate, and repetitive action rate. Both GiGPO and our method are run by us under the same setup. Lower values indicate more productive exploration.
Figure 2: A successful ALFWorld episode of the agent trained with our intrinsic reward, shown in two halves. The agent searches eight locations, revisiting only two, then takes, cleans and places the butterknife in 18 steps without a single invalid action.
Split
Method
Pick
Look
Clean
Heat
Cool
Pick2
All
Seen
GiGPO
92.3
84.6
92.0
100.0
81.2
83.3
89.1
Ours
100.0
100.0
100.0
90.9
81.2
91.7
95.3
Unseen
GiGPO
82.6
91.3
79.2
73.7
87.0
81.2
82.8
Ours
95.7
100.0
91.7
89.5
100.0
87.5
94.5
Table 3: Success rate (%) on seen and unseen ALFWorld scenes for our agent and the re-trained GiGPO.
Figure 3: Controlled comparison of our model with and without the intrinsic penalty during the first 100 training steps (identical settings). Top: validation success and duplicate-observation rate in the training rollouts. Bottom: distinct observations per episode in the training rollouts and the cumulative number of distinct (game, observation) states visited during training. Rollout curves are 10-step moving averages.
Seen scenes
Unseen scenes
Metric
γ=0
Ours
γ=0
Ours
Exploration
Duplicate-observation rate
59.3%
37.0%
61.8%
36.6%
Location revisit rate
68.6%
41.3%
68.7%
42.3%
New-information rate of info. actions
43.5%
67.4%
40.4%
67.1%
Distinct locations visited
4.68
6.17
4.87
6.88
Table 4: Effect of the intrinsic penalty: PPO with γ=0 vs. ours (identical settings, 100 training steps), same 128 episodes per split.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
GiGPO
Ours
Success rate
89.1%
95.3%
Episode length (steps)
14.7
12.5
Distinct locations visited
3.66
5.16
Location revisit rate
31.5%
18.9%
New-information rate of info. actions
81.2%
89.2%
Duplicate-observation rate
16.7%
8.7%
Appendix
Table 5: Test-time information gathering (per-episode means). † Episodes in which both agents find the target.
GiGPO
Ours
Action
Per ep.
n
Satisfied
Per ep.
n
Satisfied
go to
10.24
1311
98.6%
7.26
929
98.3%
take
1.58
202
93.6%
1.43
183
100.0%
put
1.38
176
99.4%
1.40
179
92.2%
open
0.74
95
98.9%
1.34
171
99.4%
examine
0.07
9
66.7%
0.37
47
80.9%
Appendix
Table 6: Actions per episode and precondition satisfaction by action type. ‡ Includes close and look .
Metric
GiGPO
Ours
Violations after the first (total)
26
23
per episode
0.20
0.18
share of all violations
70.3%
56.1%
Exact repeats of a failed action (total)
19
10
share of all violations
51.4%
24.4%
Back-to-back violations
4
3
Appendix
Table 7: Repeated precondition violations (shares of each agent’s violations).
Metric
GiGPO
Ours
Information gathering
Episode length (steps)
17.4
14.7
Distinct locations visited
4.22
5.80
Location revisit rate
35.4%
22.1%
New-information rate of info. actions
76.7%
88.7%
Duplicate-observation rate
20.8%
9.4%
Appendix
Table 8: Results on unseen scenes (out-of-distribution split). † Episodes in which both agents find the target.
Step 25
Step 50
Step 100
Metric
γ=0
Ours
γ=0
Ours
γ=0
Ours
Duplicate-observation rate
69.6%
63.2%
66.1%
51.5%
56.9%
40.7%
Distinct observations / episode
14.5
17.6
14.7
21.6
15.1
18.8
Location revisit rate
67.4%
64.5%
65.6%
58.9%
68.9%
49.8%
New-information rate of info. actions
33.0%
38.2%
37.8%
48.8%
42.0%
59.7%
"Nothing happens." rate
45.7%
40.2%
44.4%
31.1%
31.3%
20.2%
Appendix
Table 9: Training rollouts with γ=0 vs. ours, averaged over the 10 steps ending at each step.
Seen scenes
Unseen scenes
Metric
γ=0
Ours
γ=0
Ours
Success rate
39.1%
57.8%
34.4%
52.3%
Episode length (steps)
37.9
31.0
40.5
32.7
Actions per episode
33.8
30.3
34.4
32.1
Distinct locations visited
4.68
6.17
4.87
6.88
Location revisit rate
68.6%
41.3%
68.7%
42.3%
Appendix
Table 10: Test-time behavior of the controlled agents after 100 training steps.
Figure 4: Visitation entropy H(PL) against the number of duplicate observations D for all full-length episodes ( L=51 observations) of the controlled agents after 100 training steps, seen and unseen splits combined. The dotted and dashed curves are the upper and lower bounds of Proposition 2 .
Task: cool some mug and put it in coffeemachine (unseen scene).
Penalty ( γ=401 ): success in 8 steps
1
go to sinkbasin 1
arrive; sinkbasin empty
2
go to countertop 1
arrive; see mug 1
3
take mug 1 from countertop 1
pick up mug 1
4
go to fridge 1
arrive; fridge closed
5–6
cool mug 1 with fridge 1
cool mug 1
Appendix
Figure 5: The two controlled agents after 100 training steps on the same unseen game (actions and observations from the evaluation logs, abridged). In total, 31 of the 50 steps of the agent without the penalty return "Nothing happens." .
Figure 6: Additional example outputs of the multi-turn interaction of the LLM agent on the ALFWorld dataset after training with conditional state-space exploration with intrinsic rewards.
Figure 7: Prompts used for both ALFWorld and WebShop environments.