ClueFlow: Learning Long-Horizon Search through Attributed Research-State Transitions

Yicheng Wang; Wentao Tan; Yidan Huang; Yifei Liu; Qiong Cao

cs.AIOct 19, 2026

ClueFlow: Learning Long-Horizon Search through Attributed Research-State Transitions

Authors: Yicheng Wang, Wentao Tan, Yidan Huang, Yifei Liu, Qiong Cao

Organizations: Joy Future Academy

Abstract

Long-horizon search requires agents to acquire relevant evidence and integrate it into an evolving research state. Existing methods maintain research states to guide exploration or assign turn-level progress signals, yet grounding these signals in explicit, attributable state changes remains underexplored. We introduce ClueFlow, which learns long-horizon search from attributed research-state transitions. ClueFlow constructs Source-State-Final attribution chains that connect evidence-acquiring browser actions, Evidence and Candidate updates, and the final answer supported by those updates. During search, ClueFlow records typed state changes and potentially supporting browser actions, then applies gold-conditioned semantic validation and source tracing after execution. We use this attribution to guide both supervised fine-tuning and reinforcement learning. During SFT, attribution selects which decisions receive supervision with their original interaction contexts. During Turn-PPO, the same attribution augments outcome-based turn advantages with credit assigned along the Source-State-Final chains. Transition-level validation also recovers useful supervision from gold-consistent state changes in trajectories with incorrect final answers. We evaluate ClueFlow across two Qwen backbones and four search benchmarks spanning live-web and fixed-corpus settings, observing consistent gains from SFT and further improvements from PPO. With Qwen3.5-35B-A3B, ClueFlow achieves 60.0% accuracy on BrowseComp and 72.0% on BrowseComp-Plus. In controlled BrowseComp-Plus experiments, attribution-guided SFT matches all-turn exact-context SFT using about 30% of the training tokens, while attribution-guided PPO outperforms outcome-only PPO by 4.1 pp. These results demonstrate that state-transition attribution enables efficient supervision and effective credit assignment for long-horizon search agents.

Figures & tables

Figure 1: Learning from attributed research-state transitions. (a) Textual-state agents maintain evolving reports that summarize accumulated findings (e.g., IterResearch; Chen et al., 2026a ). (b) Tabular-state agents maintain tables tracking candidates, constraints, and collected information (e.g., Table-as-Search; Lan et al., 2026 ). Both update and consult these states to guide subsequent search. (c) ClueFlow additionally learns from record-level transitions through post-execution validation and source attribution.

Figure 2: Overview of ClueFlow. (a) Stateful trajectory generation maintains Evidence and Candidate records across search turns. (b) Post-execution validation against the reference answer and source tracing identify attributed Source, State, and Final turns. Validated Source–State progress can also be retained from incorrect trajectories. (c) Attribution-guided SFT uses one-turn examples with original per-turn inputs for attributed turns and multi-target chunks for other supervised turns from correct trajectories. (d) Attribution-guided Turn-PPO adds attribution bonuses to outcome advantages for actor updates. Here, $\Delta^{E/C}_{t,k}$ denotes the $k$ -th Evidence/Candidate state transition at assistant turn $t$ .

Representative deep-research systems
		Live-web			Fixed-corpus
Method	Post-training	BC	BC-ZH	xBench-2510	BCP †
IterResearch	RFT + RL	37.3	45.2	–	–
Tongyi DeepResearch	MT + SFT + RL	43.4	46.7	55.0	44.5
RE-TRAC	SFT	53.0	57.3	–	–
REDSearcher	MT + SFT + RL	57.4	58.2	–	–

Table 1: Comparison with representative deep-research systems on three live-web benchmarks and BrowseComp-Plus (BCP). Scores are accuracy (%).

SFT variant	Random turns	Attributed turns	All turns	Acc. (%) $\uparrow$	Rel. tokens $\downarrow$
Chunk only	$\times$	$\times$	$\times$	42.89	$0.69\times$
Chunk + random turns	$\checkmark$	$\times$	$\times$	43.01	$1.06\times$
Chunk + attributed turns	$\times$	$\checkmark$	$\times$	46.27	$1.00\times$
+ Failure recovery	$\times$	$\checkmark$	$\times$	48.31	$1.07\times$
All-turn exact-context	$\times$	$\times$	$\checkmark$	46.14	$3.33\times$

Table 2: Controlled ablations of attribution-guided SFT supervision construction and PPO credit assignment on BrowseComp-Plus.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Fixed corpus	Live web
QA source	OpenResearcher QA	REDSearcher QA
Training questions	3,657	7,551
Correct / incorrect teacher trajectories	2,927 / 730	6,300 / 1,251
SFT examples (one-turn / multi-target chunk)	34,783 / 44,031	38,962 / 75,557
Chunk size $K$	5	10
Global batch size	64	64

Appendix

Table 4: SFT data and training for the two environments.

Setting	Fixed 30B / 35B	Live 30B / 35B
QA source	OpenResearcher QA	REDSearcher QA
Training questions sampled	800	1,000
Trajectories per rollout batch	20	100
Rollout batches	40	10
Optimization batch size	20	100
History window (turns)	10	20

Appendix

Table 5: PPO data and training.

Training-data property	ReAct SFT	Keep-5 SFT	ClueFlow SFT
Teacher rollout configuration	ReAct	Keep-5	ClueFlow
Distinct training questions	2,927	2,927	3,657
Successful teacher trajectories	2,927	2,927	2,927
Incorrect trajectories used	0	0	730 †
SFT serialization	Full trajectory	One-turn	One-turn + chunks
Serialized training examples	2,927	117,877	78,814

Appendix

Table 6: SFT data and trajectory construction for the context-management comparison.

SFT model	Inference-time context strategy	Acc. (%)
ReAct	Full history	28.55
ReAct	Keep-5	29.64
ReAct	HCM	35.06
ReAct	Discard-all	41.81
Keep-5	Keep-5	32.89
Keep-5	HCM	37.11

Appendix

Table 7: Additional context-management baselines on BrowseComp-Plus. The four ReAct rows share the same SFT checkpoint and differ only in inference-time context management. HCM and Discard-all use a 64K-token context-management threshold. All evaluations use a maximum budget of 800 browsing calls per question.

Variant	Correct	Non-correct	Unfinished (#)
	Avg. browse calls
Outcome-only	82.21	319.69	20
Source-only attribution	106.91	415.37	64
Source+Final attribution	52.37	212.04	19
Full-chain attribution	88.76	396.30	24
+ Failure recovery	87.26	428.76	15

Appendix

Table 8: Additional rollout, behavioral, and post-hoc attribution statistics for the BrowseComp-Plus PPO ablations.

Long-Horizon LLM Agents

Long-horizon large language model agents plan, use tools, and preserve task state across extended, multi-step interactions.

See all 448 papers

Aug 5, 2026

ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment

Long-horizon search agents must make multiple sequential actions (steps) to search, retrieve, verify, and integrate evidence to reach a final answer. However, existing methods for training these agents typically treat all steps within a trajectory uniformly during both supervised fine-tuning (SFT) and reinforcement learning (RL), failing to distinguish useful actions from erroneous or redundant ones. In this paper, we propose Answer-Backtracked Credit Assignment (ABC), a fine-grained credit assignment framework for training long-horizon search agents by converting sparse trajectory-level outcomes into dense step-level supervision that rewards useful actions (even in failed trajectories) while suppressing erroneous or redundant actions. Specifically, given a potentially obscure query and its corresponding ground-truth answer, ABC first performs Answer-Backtracked Clue Recovery, which traces back from the answer to recover intermediate clues required to solve the question. It then applies Clue-Anchored Step Scoring to evaluate each search step against these clues, converting sparse binary outcome supervision into dense step-level rewards. Based on these rewards, we develop ABC-SFT, which reweights the loss of each turn, and ABC-GRPO, which uses the step-level scores as rewards in GRPO. Building on this framework, we train ABSeeker based on Qwen3.5-4B with only 8.5k examples. ABSeeker achieves 37.3% on BrowseComp and 39.1% on BrowseComp-ZH. With context management, the scores further improve to 55.3% and 52.9%, respectively, significantly outperforming same-scale (4B) agents and even matching the performance of larger ones (approximately 30B). These results demonstrate the effectiveness of answer-backtracked step-level credit assignment for training long-horizon search agents.

Yijun Lu, Rui Ye, Jiajun Wang +4

Shanghai Jiao Tong University

Credit Assignment in RL Step-Level Credit Assignment in RL LLM Agent Training RL for LLM Agents

Ask PDF

Sep 29, 2026

Traverse: Learning When to Remember, Reset, and Redirect for Long-Horizon Web Search

Long-horizon information-seeking agents often accumulate noisy or misleading context, causing early mistakes to persist and making recovery increasingly difficult. We introduce an autonomous search harness in which the agent manages its own search process through three states: Rubric, Answer, and Verify. The agent first defines criteria for a valid answer, searches under these criteria, and then independently verifies the result before deciding whether to terminate or continue searching. It is further equipped with a Seal Memory tool that enables active context management. Training this behavior with reinforcement learning, however, can induce Seal Collapse, resulting in unstable training and preventing the agent from reliably learning when and how to use its memory tools. We solve this with a simple strategy that trains only the final segment after context management. Our 35B model achieves 72.83 on BrowseComp, outperforming comparable open-source systems, and consistently improves over the base model across BrowseComp-ZH, xbench, DeepSearchQA, WideSearch, financial investigation, and product search. Ablations show that autonomous compression outperforms automatic compaction and validate our RL design.

Jingyuan Ma, Lynx Aster, He Zhang +5

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · ByteDance

Long-Horizon Agent Evaluation Web Search Agents RL for LLM Agents Memory-Augmented Language Model Agents

Ask PDF

Oct 4, 2026

LexiHorizon: Stabilizing Reinforcement Learning for Long-Horizon Deep Search

Deep search agents tackle complex knowledge tasks through iterative retrieval, multi-hop reasoning, and evidence synthesis across multiple sources. Existing approaches typically assume relatively stable retrieval systems and operate over short-horizon tool interaction. However, when retrieval is sensitive to query formulation, even a semantically appropriate query may fail to surface critical evidence because of mismatched entity names, aliases, or keyword combinations. Recovering from such failures requires repeated query reformulation and longer interaction trajectories. This setting poses a distinct training challenge, as the policy must sustain long-horizon query exploration while managing an expanding volume of retrieved content. We propose LexiHorizon, a framework for training search agents over long horizons that expands the trajectory context budget, manages accumulated retrieval content using a window over recent tool observations while preserving the reasoning history, and introduces an outcome-gated search-effort reward that provides a bounded bonus for tool invocations to trajectories with nonzero answer reward. Experiments on XBench, WebWalkerQA, and BrowseComp-ZH show that the resulting 9B model consistently outperforms both its base model and MiroThinker-1.7-mini, with maximum absolute gains of 8.7 and 23.8 percentage points, respectively. These results suggest that combining an extended context budget with reasoning-preserving context management benefits long-horizon deep search agents.

Zhiqing Nong, Liang Wen, Chao-Hsuan Liu

Long-Horizon Agent Evaluation RL for IR Long-Horizon LLM Agents Memory-Augmented Language Model Agents

Ask PDF

Abstract

Long-Horizon LLM Agents

Explore similar work

ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment

Traverse: Learning When to Remember, Reset, and Redirect for Long-Horizon Web Search

LexiHorizon: Stabilizing Reinforcement Learning for Long-Horizon Deep Search