Multi-turn LLM agents are becoming pivotal to production systems, spanning customer service automation, e-commerce assistance, and interactive task management, where accurately distinguishing high-value informative signals from stochastic noise is critical for sample-efficient training. In real-world scenarios, a failure in a trivial task may reflect random instability, whereas success in a high-difficulty task signifies a genuine capability breakthrough. Yet, existing group-based policy optimization methods rigidly rely on statistical deviation within discrete batches, frequently misallocating credit when task difficulty fluctuates. To address this issue, we propose Proximity-based Multi-turn Optimization (ProxMO), a practical and robust framework engineered specifically for the constraints of real-world deployment. ProxMO integrates global context via two lightweight mechanisms: success-rate-aware modulation dynamically adapts gradient intensity based on episode-level difficulty, while proximity-based soft aggregation derives baselines through continuous semantic weighting at the step level. Extensive evaluations on ALFWorld and WebShop benchmarks demonstrate that ProxMO yields substantial performance gains over existing baselines with negligible computational cost. Ablation studies further validate the independent and synergistic efficacy of both mechanisms. Crucially, ProxMO offers plug-and-play compatibility with standard GRPO frameworks, facilitating immediate, low-friction adoption in existing industrial training pipelines. Our implementation is available at: https://anonymous.4open.science/r/proxmo.
Figures & tables
Figure 1: Motivating challenges in multi-turn policy optimization. (a) Context-blind normalization: identical z-score magnitudes yield uniform advantage intensities across high-success (e.g., 75%) and low-success (e.g., 25%) groups, ignoring informational heterogeneity. (b) Hard boundary partitioning: binary participation (in/out based on threshold) with equal intra-group weighting, causing singleton degeneracy under strict criteria or incorrect equal weighting under loose criteria.
Figure 2: The overview of ProxMO. Episode-level: success-rate-aware modulation adapts credit to task difficulty (i.e., p ). Step-level: proximity-based soft aggregation eliminates discrete boundaries for robust baseline estimation.
Method
ALFWorld
WebShop
Pick
Look
Clean
Heat
Cool
Pick2
All
Score
Succ.
Closed-Source LLMs
GPT-4o
75.3
60.8
31.2
56.7
21.6
49.8
48.0
31.8
23.7
Gemini-2.5-Pro
92.8
63.3
62.1
69.0
26.6
58.7
60.3
42.5
35.9
Qwen2.5-1.5B-Instruct
Base
5.9
5.5
3.3
9.7
4.2
0.0
4.1
25.1
6.3
Table 1: Comparison of benchmark results on unseen test instances (averaged over 3 random seeds) across Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. Best results are bold , second-best are underlined . Δ vs GRPO denotes relative improvement (%) over GRPO. Key metrics (All, Succ.) are highlighted.
Figure 3: Hyperparameter sensitivity analysis on ALFWorld and WebShop, with ProxMO maintaining stable high performance across broad parameter configurations. For clarity, temperature τ is visualized on a logarithmic scale.
Figure 4: Ablation study on ALFWorld, where ProxMO outperforms all variants and the strong baseline GiGPO.
Figure 5: Training time comparison on ALFWorld (shaded regions denote confidence intervals) reveals ProxMO incurs a minimal additional overhead (+1.09%) versus GRPO across training iterations, confirming its computational efficiency for scalable pipelines.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
Training Iteration
Group Size 1
Group Size 2
Group Size 3
Group Size >3
Iteration 40
36.2%
15.6%
11.2%
37.0%
Iteration 80
34.2%
14.2%
12.3%
39.3%
Iteration 120
30.2%
16.2%
13.6%
40.0%
Appendix
Table 2: Distribution of step-level group sizes during training on ALFWorld. Singleton groups (size 1) consistently account for 30-36% of all steps, depriving these steps of meaningful baseline comparisons and credit signals.
Parameter
Value
ProxMO-specific
Episode steepness ( α )
4.0
Episode strength ( β )
0.1
Step temperature ( τ )
0.1
Following GiGPO Feng et al. (2025b)
Step discount ( γ )
0.95
Appendix
Table 8
Setting
ALFWorld
WebShop
Maximum episode length
50 steps
15 steps
Maximum prompt length
2048 tokens
4096 tokens
Training iterations
150
150
Appendix
Table 9
Category
Description
Pick & Place (Pick)
Locate and move objects to target locations
Examine in Light (Look)
Pick up objects and examine under lamps
Clean & Place (Clean)
Clean objects and place them appropriately
Heat & Place (Heat)
Heat objects in microwaves before placing
Cool & Place (Cool)
Cool objects in fridges before placing
Pick Two & Place (Pick2)
Manipulate two objects sequentially
Appendix
Table 3: ALFWorld task categories and descriptions.
Component
Details
Product catalog
Over 1.1 million real products from Amazon
User instructions
12,000 diverse instructions across categories
Observations
Rich HTML requiring semi-structured parsing
Action space
Search queries, navigation, attribute filtering
Evaluation metrics
Score (attribute matching), Success (completion)
Episode constraint
15-step limit (simulating user patience)
Appendix
Table 4: WebShop environment specifications.
Figure 6: The prompt template for ALFWorld agents.
Figure 7: The prompt template for WebShop agents.
Figure 8: Step 1: Task understanding and exploration strategy.
Figure 9: Step 2: First remote control discovered and acquired.
Figure 10: Step 3: Navigating with first object to destination.
Figure 11: Step 4: Placing first remote control in armchair.
Figure 12: Step 5: First object stored, searching for second remote.
Figure 13: Step 6: Dresser checked, no remote control found.
Figure 14: Step 7: Sidetable 5 checked, moving to sofa.
Figure 15: Step 8: Second remote control located on sofa.
Figure 16: Step 9: Second object acquired, returning to armchair.
Figure 17: Step 10: Final remote control placed, task complete.