ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents

Rumei Li; Xunliang Cai

cs.CLOct 22, 2026

ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents

Authors: Rumei Li, Xunliang Cai

Organizations: Meituan, China

Abstract

With external tools, large language models (LLMs) can tackle complex tasks through sustained environmental interaction. LLM-based agents are therefore increasingly used to tackle complex long-horizon tasks in real-world environments. However, these tasks involve sequential, interdependent decisions and changes in environmental states. Early execution errors affect subsequent interaction histories and environmental states, and their effects accumulate along the decision chain. To address this challenge, we propose Reflection-Guided Policy Optimization (ReflectPO), a reinforcement learning framework for autonomous error recovery in long-horizon tasks. ReflectPO comprises two core components. The first is Reflection-Guided Sampling. We use branch differences and repeated visits in interaction trajectories to construct hints for error recovery. The second is Tree-Structured Credit Assignment. We organize sampled trajectories into trees and assign action-level credit using parent--child reward differences and recovery gains. Experiments over two long-horizon tool-use benchmarks show that ReflectPO outperforms multiple existing baselines in most domains. It also achieves the best overall performance on average. Further analyses show that ReflectPO effectively improves agents' ability to recognize their own deviations, recover from erroneous states, and continue making progress toward task completion.

Figures & tables

Dataset	Error Trajectory Rate	Error Step Rate	Step Recovery Rate	Complete Recovery Rate
Overall	83.68	19.49	18.74	14.19
BFCL-v3	79.38	16.27	19.18	17.95
$\tau^{2}$ -Bench	96.04	28.74	17.48	3.37

Table 1: Errors and autonomous recovery in natural agent trajectories.

Figure 1: Recovery trajectory sampling in ReflectPO. Contrast-based sampling compares target and reference trajectories after a shared branch point. Loop-based sampling identifies repeated visits within a trajectory. Both insert reflection hints at recovery anchors and generate new rollouts.

Figure 2: Credit assignment in ReflectPO. Original actions receive tree-based advantages from empirical outcome differences between parent and child nodes. Recovery actions receive paired advantages from terminal reward differences between recovery and target original trajectories.

Figure 3: Error recovery in a messaging task. The complete example can be found in Appendix D .

Method	BFCL-v3 Multi-Turn					$\tau^{2}$ -Bench				Overall
	Base	Miss- Func	Miss- Param	Long- Context	Overall	Airline	Retail	Telecom	Overall
Qwen3-4B Thinking	45.0	37.0	26.0	27.5	33.9	38.0	45.2	14.4	31.3	33.2
+ PPO	46.0	42.0	29.5	28.0	36.4	30.0	43.0	13.5	28.6	34.4
+ GRPO	40.5	40.0	30.5	30.5	35.4	31.3	42.9	15.9	29.7	33.9
+ Reinforce++	44.5	42.0	31.5	28.0	36.5	38.0	47.7	11.4	31.1	35.1
+ GiGPO	46.0	41.5	28.0	29.0	36.1	34.0	48.3	16.7	32.7	35.3

Table 2: Performance of Qwen3-4B and Qwen3-8B on BFCL-v3 Multi-Turn and

\tau^{2}

-bench. Bold and underlined values indicate the best and second-best results in each column, respectively.

Panel of Figure 4 — Figure 4: Error and recovery analysis for Qwen3-8B under different training methods.

Method	BFCL-v3 Multi-Turn	$\tau^{2}$ -Bench	Overall
Qwen3-8B Thinking	41.75	33.55	39.64
+ Reinforce++	43.50	35.05	41.32
+ Tree credit	46.88	32.73	43.23
+ Contrast only	48.38	36.33	45.27
+ Loop only	47.63	34.89	44.34
+ ReflectPO	48.13	37.41	45.36

Table 3: Ablation averages. Bold and underlined values mark the best and second-best results.

Figure 5: Training sampling efficiency of ReflectPO and Reinforce++. Each validation score is plotted against rollout response tokens accumulated through the corresponding training step.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Task Perform a pre-promotion moderation and cleanup pass with the following objectives: (1) for all products in the Electronics and Home & Kitchen categories, approve each pending review when the author’s account_status is active and reject it otherwise, without modifying non-pending reviews; (2) for PROD-235 (BreezyAir Smart Fan), add a five-star review from USR-001 , if one does not already exist, with the text “Excellent airflow and very quiet. Setup was straightforward and the app controls are intuitive.”, and set its moderation_status to approved; and (3) delete only the rejected review on PROD-999 (Whispering Pines) authored by a suspended user.
Prefix
Step	Action	Observation
7	{ "type": "tool_call", "name": "get_reviews_for_product_by_status", "arguments": { "product_id": "PROD-999", "moderation_status": "rejected" } }	{ "success": true, "data": [ { "review_id": "PROD-999-USR-004", "product_id": "PROD-999", "user_id": "USR-004", "rating": 3, "review_text": "Nice story but pacing was uneven in the middle chapters.", "timestamp": "2022-12-10T12:34:00", "moderation_status": "rejected" } ] }
8	{ "type": "tool_call", "name": "get_user_by_id", "arguments": { "user_id": "USR-004" } }	{ "success": true, "data": { "user_id": "USR-004", "username": "Connor Riley", "account_status": "suspended" } }
9	{ "type": "tool_call", "name": "delete_review", "arguments": { "review_id": "PROD-999-USR-004" } }	{ "success": true, "message": "Review PROD-999-USR-004 deleted successfully." }
Original Suffix (Final Score 0.75)

Appendix

Table 4: A contrast-based sampling example. Each trajectory step shows the step index, action, and observation from left to right.

Task For user USR-cc98f55e (Jordan Price), cancel order ORD-871fa9bc-4 and create a replacement order containing three Metaprox 850mg tablets and twelve Loratadine 10mg tablets. Add and validate the shipping address 2525 W 34th Ave., Denver, CO 80211, USA; verify the catalog dosage, dosage form, and available inventory; reserve inventory; validate the Metaprox prescription; mark the new order as paid and shipped; and assign a carrier and tracking number.
Prefix
Step	Action	Observation
2	{ "type": "tool_call", "name": "get_saved_addresses_for_user", "arguments": { "user_id": "USR-cc98f55e" } }	{ "success": true, "data": [ { "address_id": "ADDR-den-501", "user_id": "USR-cc98f55e", "street": "6302 E Colfax Ave.", "city": "Denver", "state": "CO", "zip_code": "80220", "country": "USA" }, { "address_id": "ADDR-den-502", "user_id": "USR-cc98f55e", "street": "1801 Blake St.", "city": "Denver", "state": "CO", "zip_code": "80202", "country": "USA" }, { "address_id": "dd91af9e...d07e", "user_id": "USR-cc98f55e", "street": "2525 W 34th Ave.", "city": "Denver", "state": "CO", "zip_code": "80211", "country": "USA" } ] }
3	{ "type": "tool_call", "name": "validate_address_for_user", "arguments": { "address_id": "dd91af9e...d07e", "user_id": "USR-cc98f55e" } }	{ "success": true, "data": true }
Original Suffix (Final Score 0.3571)
Step	Action	Observation

Appendix

Table 5: A loop-based sampling example. Each trajectory step shows the step index, action, and observation from left to right.

Step	Action	Observation
Task Description As a seasoned real estate agent, my expertise is all about ensuring your bakery finds the perfect spot to thrive. Now, it seems we have an unrelated budgeting task here. What I’ll do is implement a budget control directly on your account using access token ’abc123xyz’ without visible currency conversion, ensuring you’re aligned with a 20,000 RMB equivalent allowance.
Reasoning Content (excerpt) […] The ’set_budget_limit’ tool seems relevant. It requires an access token and a budget_limit in USD. Wait, the user mentioned RMB. […] So I should call compute_exchange_rate with base_currency RMB, target_currency USD, value 20000. Then use that USD value in set_budget_limit. […]
1	compute_exchange_rate( base_currency=’RMB’, target_currency=’USD’, value=20000 ), set_budget_limit( access_token=’abc123xyz’, budget_limit=20000 )	{ "exchanged_value": 2857.14 } { "budget_limit": 20000.0 }
Reasoning Content (excerpt) […] But in the previous response, the assistant called set_budget_limit with 20000.0, which is in RMB, not USD. That’s a mistake. The correct value should be 2857.14 USD. […] So the assistant needs to correct that. Let me check the tool parameters again. The set_budget_limit requires budget_limit in USD. Therefore, the assistant should use the exchanged_value from the first tool call as the argument for set_budget_limit. […]
2	set_budget_limit( access_token=’abc123xyz’, budget_limit=2857.14 )	{ "budget_limit": 2857.14 }
Assistant Response Your budget limit has been successfully set to 2857.14 USD, which is equivalent to 20,000 RMB. This ensures your account is aligned with the specified allowance. Let me know if you need further assistance!

Appendix

Table 6: An error recovery example with the full interaction sequence. The agent corrects the budget limit using the converted USD amount before proceeding to subsequent user requests. Reasoning excerpts retain the original omission markers.

Long-Horizon LLM Agents

Long-horizon large language model agents plan, use tools, and preserve task state across extended, multi-step interactions.

See all 443 papers

Rollback the World, Keep the Reflection: Rollback-Induced Reflection for Long-Horizon LLM Agents
Sep 16, 2026Yi Yu, Liuyi Yao, Yaliang Li +2Long-Horizon LLM Agents LLM Agent Reliability
Wuhan University · Alibaba Group
LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation
Aug 12, 2026Zhixin Zhang, Xinke Jiang, Zhibang Yang +5LLM-Guided RL LLM Agents
School of Computer Science, Peking University, Beijing, China · National Engineering Research Center for Software Engineering, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China +2
Agent-Reflex: Teaching Language Agents to Act Reflectively
Oct 13, 2026Guanting Dong, Xinyu Yang, Jiajie Jin +9LLM Agent Training Self-Reflection in Language Model Agents
Gaoling School of Artificial Intelligence, Renmin University of China · National University of Singapore

Sep 16, 2026J/K move · Enter open · S save

Rollback the World, Keep the Reflection: Rollback-Induced Reflection for Long-Horizon LLM Agents

Yi Yu, Liuyi Yao, Yaliang Li, Enshu Wang+1

Wuhan University · Alibaba Group

Long-Horizon LLM Agents LLM Agent Reliability Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Large language model (LLM) agents increasingly tackle long-horizon tasks through multi-step environment interaction, yet a single erroneous action can alter subsequent states and observations, causing errors to compound over time. Existing methods either correct the context without repairing altered environment states or restore earlier states while discarding useful experience, making it difficult to both eliminate failure conditions and avoid repeating past mistakes. We argue that reliable recovery should instead be treated as a rollback-boundary control problem that jointly determines when to intervene, where to resume, and what information should survive recovery. Based on this view, we propose Rollback-Induced Reflection (RIR), a unified recovery framework that restores execution to a selected prior state while carrying forward reusable knowledge distilled from the abandoned trajectory to guide subsequent decisions. We further characterize recovery through a unified operator over rollback depth and retained memory, providing a general view of state restoration and knowledge retention. Experiments on three long-horizon benchmarks show that RIR consistently improves average task performance across multiple LLM backbones, with structured reflection memory preserving useful experience and selective rollback enabling efficient recovery.

Abstract

Long-Horizon LLM Agents

Explore similar work

Rollback the World, Keep the Reflection: Rollback-Induced Reflection for Long-Horizon LLM Agents

LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation

Agent-Reflex: Teaching Language Agents to Act Reflectively