PEARL: Process-Experience Aware Reinforcement Learning for Multi-Turn Agents

Jie Yang; Mianqiu Huang; Yiwei Zhao; Bo Wang; Yang Liu; Yuxin Wang

cs.AIOct 10, 2026

PEARL: Process-Experience Aware Reinforcement Learning for Multi-Turn Agents

Authors: Jie Yang, Mianqiu Huang, Yiwei Zhao, Bo Wang, Yang Liu, Yuxin Wang

Organizations: Fudan University · Shanghai Innovation Institute

Abstract

Terminal rewards make multi-turn agents trainable without step-level annotations, but group-relative advantages provide little guidance about which intermediate decisions to adjust. Failed trajectories also contain reasonable actions, while unfavorable decision patterns can recur as training proceeds. We introduce PEARL (Process-Experience Aware Reinforcement Learning), which improves local credit assignment without an additional critic. Given two continuations from the same history with different outcomes, PEARL locates their first divergence and builds a feature representation of each branch's local decision. It clusters these decision features in recent failure and success memories to identify recurring local patterns. Using this pattern memory, PEARL selectively strengthens negative advantages at lower-reward decisions, while leaving positive advantages and the base policy optimizer unchanged. On AppWorld, PEARL improves over ARPO on Normal and Challenge tasks at both 4B and 8B scales. On ALFWorld, it achieves 97.9% mean success at 4B, 2.7 points above the strongest baseline, while remaining competitive at 8B. Ablations of shaping location, success-bank filtering, and memory window support the value of selective, recurring-pattern-based updates.

Figures & tables

Figure 1: Overview of PEARL. Unequal-outcome branches provide local decision contrasts. Recurring decision patterns are stored in recent failure and success memories, success-side evidence filters ambiguous failure patterns, and retained failure patterns guide stronger negative updates.

Figure 2: Decision representation at the first divergence. Each branch scores its selected token across the latter half of the network, producing a compact layer-logit feature for clustering.

Qwen3-4B-Instruct-2507
	AppWorld			ALFWorld
Method	Normal	Challenge	Mean	Seen	Unseen	Mean
Base Model	8.93	5.60	7.26	38.1	33.1	35.6
GRPO	52.38 $\pm$ 0.79	24.38 $\pm$ 0.16	38.38 $\pm$ 0.48	94.9 $\pm$ 2.8	94.3 $\pm$ 5.7	94.6 $\pm$ 4.2
ARPO	53.37 $\pm$ 2.18	29.02 $\pm$ 2.56	41.19 $\pm$ 2.37	95.1 $\pm$ 0.4	92.3 $\pm$ 0.8	93.7 $\pm$ 0.2
GiGPO	44.84 $\pm$ 4.76	24.02 $\pm$ 0.52	34.43 $\pm$ 2.64	96.0 $\pm$ 2.0	94.4 $\pm$ 3.4	95.2 $\pm$ 2.7

Table 1: Final task success rates (%) after 400 AppWorld or 300 ALFWorld updates. Each “Mean” equally weights the two splits. Trained-method values are mean

\pm

standard deviation over three runs; Base Model is the untrained model. Bold and underline mark the best and second-best distinct values within each model scale, with ties formatted alike.

Figure 3: Qwen3-4B-Instruct-2507 evaluation success rates (%) on AppWorld (top, through 400 updates) and ALFWorld (bottom, through 300 updates). Step 0 is the shared pre-training evaluation; subsequent points, spaced every 25 updates, are averages across the available training runs. Orange endpoint labels show PEARL’s final scores reported in Table 1 .

Panel of Figure 4 — Figure 4: Single-run AppWorld success rates (%) at step 400 with Qwen3-4B-Instruct-2507. (a) Failure-only, success-only, and bidirectional shaping without cross-bank filtering. (b) With versus without success-bank filtering, holding failure-only shaping and MCS10/MS5 fixed. (c) Four HDBSCAN input representations, holding all other settings fixed. Light and dark bars denote Normal and Challenge, respectively.

Figure 5: Failure-shaping activity with a 25-update window and unlimited history (

W=\infty

) on AppWorld (Qwen3-4B-Instruct-2507, hard filtering, MCS10/MS5; one run per setting). Left: number of updates with active failure shaping in each consecutive block of 10 updates. Right: cumulative number of updates with active failure shaping, reaching 88 for

W=25

and 3 for unlimited history.

Cluster	Trajectory behavior and outcome
Success $S_{1}$	Behavior 1: friends and roommates. Inspect the contact-relationship API specification; receive a valid schema. Later, collect the matching requests before denying them, and reject all 14. The trajectory succeeds. Behavior 2: siblings. Inspect the same API specification; receive a valid schema. Later, check both transaction senders and receivers and like all 8 matching transactions. The trajectory succeeds.
Failure $F_{1}$	Behavior 1: friends and roommates. Inspect the same specification; receive a valid schema. Later failure: Deny requests while paging through the pending list; reject only 13 of 14, missing request 1844. Modifying the list during traversal is consistent with a pagination skip. Behavior 2: siblings. Inspect the same specification; receive a valid schema. Later failure: Check only whether the sender is a sibling, ignoring sibling receivers. Like only 2 of the 8 matching transactions.
Shared pattern. Inspect the contact-relationship API specification before identifying people by relationship.
Memory decision. $S_{1}$ filters $F_{1}$ . The same specification-inspection behavior occurs in successful trajectories. The corresponding failure cluster is excluded from additional negative shaping.

Table 2: AppWorld, Qwen3-4B-Instruct-2507, training step 400. Two behaviors from each cluster exhibit the same API-inspection pattern; success cluster

S_{1}

filters failure cluster

F_{1}

Cluster	Trajectory behavior and outcome
Success $S_{1}$	Behavior 1: finding a cup. Observe that cabinet 1 is closed; open it to check for the cup. Action: open cabinet 1 . The cabinet is empty. The trajectory succeeds. Behavior 2: finding bread. Observe that cabinet 1 is closed; open it to check for bread. Action: open cabinet 1 . The cabinet contains a plate. The trajectory succeeds.
Failure $F_{0}$	Behavior 1: finding a cup. Open the closed cabinet to check for the cup. Action: open cabinet 1 . Later failure: Cool and place a mug instead of the required cup ; then repeatedly look around or check inventory until the 50-turn limit. Behavior 2: finding a pot. Open the closed cabinet to check for the pot. Action: open cabinet 1 . Later failure: Revisit searched locations without checking stoveburner 2, where a successful trajectory finds the pot. Exhaust 50 turns without acquiring or cooling it.
Shared pattern. When a candidate cabinet is closed, open it to inspect its contents for the target object.
Memory decision. $S_{1}$ filters $F_{0}$ . The same closed-cabinet opening behavior occurs in successful trajectories. The corresponding failure cluster is excluded from additional negative shaping.

Table 3: ALFWorld, Qwen3-4B-Instruct-2507, training step 200. Both clusters contain the displayed closed-cabinet opening behavior; success cluster

S_{1}

filters failure cluster

F_{0}

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Step	Failure entries	Success entries	Retained failure clusters	Retained entries	Filtered clusters	Success clusters
100	297	347	0	0	2	2
200	716	798	1	19	2	2
300	1106	1195	0	0	5	4
400	1368	1509	1	1200	1	15

Appendix

Table 4: Failure- and success-bank snapshots from the AppWorld hard-filter MCS10/MS5 run with unlimited history (

W=\infty

). “Retained entries” counts entries in retained failure clusters, not the number of clusters.

Environment and cluster

Audited decisions, observed responses, and bank decision

AppWorld step 200

F_{1}

Delay alarm by one hour: call apis.phone.show_alarms() without an access token; the API returns HTTP 401 (missing, invalid, or expired authorization). Delay alarm by twenty minutes: make the same unauthenticated call; the API again returns HTTP 401. Matched pattern: invoke a protected API without satisfying its authentication requirement. Bank decision: retained because the success bank does not veto

F_{1}

ALFWorld step 300

F_{0}

After turning on the lamp: issue examine desklamp 1 , which is outside the admissible set; the environment replies “Invalid action.” After a rejected inspection: following a rejected look under desklamp 1 , issue examine desklamp 1 and again receive “Invalid action.” Matched pattern: use an unavailable lamp-inspection command. Bank decision: retained because no success cluster has formed to veto

F_{0}

Appendix

Table 5: Retained failure-side patterns in Qwen3-4B-Instruct-2507 runs. The two rows report separate bank snapshots and two audited decisions per environment.

Multi-Turn RL

See all 20 papers

What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents
May 19, 2026Xiaozhe Li, Tianyi Lyu, Yang Li +6Reinforcement Learning Credit Assignment in RL
1Tongji University · 2Shanghai AI Laboratory · 5Independent +2
GAGPO: Generalized Advantage Grouped Policy Optimization
May 13, 2026Siyuan Zhu, Chao Yu, Rongxin Yang +4Reinforcement Learning Group Relative Policy Optimization
School of Computer Science and Engineering, Sun Yat-sen University · Meituan
DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter
Oct 13, 2026Xuyan Ye, Xiyan Fu, Shengda Fan +3RL for LLM Agents Reinforcement Learning
College of Computing and Data Science, Nanyang Technological University · Gaoling School of Artificial Intelligence, Renmin University of China

May 19, 2026J/K move · Enter open · S save

What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents

Xiaozhe Li, Tianyi Lyu, Yang Li, Yichuan Ma+5

1Tongji University · 2Shanghai AI Laboratory · 5Independent+2

Reinforcement Learning Credit Assignment in RL Step-Level Credit Assignment in RL RL for LLM Agents Multi-Turn RL

Open paper AskPDF arXiv

Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging: a single success-or-failure signal must be distributed across many actions. Existing methods rely on trajectory-level rewards or proxy signals, without fully leveraging per-step environmental feedback. Multi-turn agent settings are underexplored, where feedback can include error messages, page changes, observations, or reference trajectories. We systematically study five feedback sources and two insertion granularities and introduce SERL, a selective environment-reweighted learning framework. SERL uses the task reward to determine update direction, while environment feedback adjusts placement and magnitude, focusing on critical actions. On ALFWorld and WebShop, SERL achieves 90.0% and 80.1% success, outperforming strong RL and distillation baselines. Analysis shows that grounded, action-relevant feedback at meaningful points consistently outperforms indiscriminate use of longer or richer context.

Abstract

Multi-Turn RL

Explore similar work

What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents

GAGPO: Generalized Advantage Grouped Policy Optimization

DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter