TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs

Yutao Xie; Nathaniel Thomas; Nicklas Hansen; Yang Fu; Li Erran Li; Xiaolong Wang

cs.CLMar 11, 2026

TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs

Authors: Yutao Xie, Nathaniel Thomas, Nicklas Hansen, Yang Fu, Li Erran Li, Xiaolong Wang

Organizations: UC San Diego · AWS AI

Abstract

Search-augmented large language models (LLMs) trained with reinforcement learning (RL) have achieved strong results on open-domain question answering (QA), but training still remains a significant challenge. The optimization is often unstable due to sparse rewards and difficult credit assignments across reasoning and tool calls. To address this, we introduce Turn-Level Information Potential Reward Shaping (TIPS), a simple framework that assigns dense, turn-level rewards to each reasoning + tool-call segment based on the increased likelihood of the correct answer under a teacher model. By leveraging the potential-based reward shaping, TIPS offers fine-grained and policy-invariant guidance that overcomes the limitations of outcome-only optimization. Evaluated on seven QA benchmarks, TIPS consistently outperforms GRPO/PPO baselines and substantially improves training stability. For instance, with a Qwen-2.5 7B Instruct model, TIPS improves the average Exact Match score by 11.8% and F1 by 13.6% relative to PPO. Our results demonstrate that turn-level information-potential reward shaping provides an effective and general solution to sparse-reward credit assignment for multi-turn LLM reasoning.

Figures & tables

Figure 1: Overview of our training framework. The policy model interacts with the environment by conducting multi-turn conversations: each turn consists of reasoning, issuing a query, and receiving search results, until a final answer is produced. Two dense reward signals are then derived: ( i ) an outcome reward , obtained by verifying whether the final answer matches the ground truth; and ( ii ) an information reward , provided by a teacher model that measures the information gain each turn contributes toward the ground truth. Both rewards are combined and optimized with PPO.

Figure 2: Turn-level information reward pipeline. At each turn, retrieved evidence updates the answer likelihood, yielding a turn-level reward

\Delta_{k}

. These rewards are then injected at turn boundaries.

Figure 3: EM accuracy on the training set. TIPS converges to a high accuracy; PPO drifts late; GRPO collapses.

Model	In-domain		Out-of-domain					Avg
Model	NQ	HotpotQA	TriviaQA	PopQA	2Wiki	MuSiQue	Bamboogle	Avg
Qwen-2.5-7B Instruct
PPO	41.95	34.46	63.71	43.55	32.94	8.94	35.40	37.28
GRPO	37.15	26.54	56.39	37.11	19.41	7.20	16.00	28.54
MT-GRPO*	37.17	29.28	58.29	38.37	22.62	7.99	19.20	30.42
MT-PPO	42.37	26.45	55.12	41.59	22.81	6.91	11.20	29.49

Table 1: Exact Match (EM) for 7B and 3B models. In-domain: {NQ, HotpotQA}. Out-of-domain: {TriviaQA, PopQA, 2Wiki, MuSiQue, Bamboogle}. The last column is the average over all 7 tasks.

Model	In-domain		Out-of-domain					Avg
Model	NQ	HotpotQA	TriviaQA	PopQA	2Wiki	MuSiQue	Bamboogle	Avg
Qwen-2.5-7B Instruct
PPO	50.53	44.65	70.88	47.62	38.70	17.79	45.33	45.07
GRPO	47.85	35.29	63.58	42.51	24.15	12.12	22.93	35.49
MT-GRPO*	48.10	38.36	66.05	44.11	27.94	14.29	28.25	38.16
MT-PPO	50.60	35.29	62.08	46.45	28.56	12.13	20.87	36.57

Table 2: F1 for 7B and 3B models. In-domain: {NQ, HotpotQA}. Out-of-domain: {TriviaQA, PopQA, 2Wiki, MuSiQue, Bamboogle}. The last column is the average over all 7 tasks.

Model	EM	F1	FLOPs overhead (%)
Qwen2.5-3B-Instruct	33.6 (+11.4%)	41.1 (+10.2%)	11.761
Qwen3-4B-Instruct-2507	48.4 (+7.3%)	57.1 (+6.1%)	11.846
Qwen2.5-7B-Instruct	41.7 (+11.9%)	51.2 (+13.7%)	11.810
Qwen2.5-14B-Instruct	45.4 (+12.7%)	53.1 (+10.6%)	11.813
Llama3.1-8B	40.3 (+34.0%)	49.0 (+29.3%)	11.659

Table 3: Generalization of TIPS across model families and scales. EM/F1 are TIPS scores; percentages in parentheses indicate relative improvement over the outcome-only PPO baseline. FLOPs overhead is the relative per-step increase due to teacher scoring.

Figure 4: Training dynamics of PPO vs. TIPS. Blue curves denote PPO and orange curves denote TIPS with teacher refresh every 200 steps. Overall, TIPS climbs steadily to higher and more stable plateaus, while PPO often suffers mid-training drift or collapse, especially on multi-hop datasets.

Base model	Small (0.001–0.05)	Medium (0.05–0.3)	Large (0.3–1.0)
Qwen3-4B	$+1.4\%$ (stable)	$+7.3\%$ (stable)	$-3.4\%$ (stable)
Qwen2.5-7B	$0\%$ (crashed)	$+11.9\%$ (stable)	$+3.1\%$ (stable)
Qwen2.5-3B	$0\%$ (stable)	$+11.4\%$ (stable)	$0\%$ (stable)

Table 5: EM gains over PPO for different target information-reward ranges under dynamic

\alpha

. The medium band yields stable and consistently positive gains across backbones; very small targets effectively turn shaping off, while very large targets let shaping compete with the terminal reward and can hurt performance.

Panel of Figure 6 — Figure 6: Effect of the teacher refresh interval on EM for TIPS.

Policy	Frozen policy	Qwen3-4B-TIPS	Llama3.1-8B
Qwen2.5-7B	41.7 (+11.9%)	30.0 (-19.5%)	29.0 (-22.2%)
Qwen3-4B	48.4 (+7.3%)	45.88 (+1.7%)	43.0 (-4.7%)

Table 6: Ablation on teacher selection. Rows fix the policy backbone and vary the teacher.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: BrowseComp-Plus performance versus model size. We plot PPO (blue circles), GRPO (orange squares), and TIPS (green triangles) for Qwen2.5-3B/7B and Qwen3-4B, together with the 32B Search-R1 PPO agent. TIPS consistently outperforms outcome-only PPO/GRPO at each scale, and a 4B TIPS agent even surpasses the 32B Search-R1 baseline.

Model	PPO	GRPO	TIPS
Qwen2.5-3B-Instruct	1.20	1.20	1.57
Qwen2.5-7B-Instruct	1.24	3.61	4.10
Qwen3-4B-Instruct	6.75	2.65	9.40
Search-R1-32B	4.11	–	–

Appendix

Table 7: BrowseComp-Plus accuracy (%). TIPS consistently improves over PPO/GRPO, and our Qwen3-4B TIPS agent surpasses a 32B Search-R1 baseline.

Parameter	Value
topk (–topk)	3
faiss_gpu (–faiss_gpu)	True
retrieval_method	e5 (dense)
retrieval_pooling_method	mean
retrieval_query_max_length	256
retrieval_use_fp16	True

Appendix

Table 8: Retrieval server configuration

Parameter	Value
algorithm.adv_estimator	grpo
algorithm.gamma	1.0
algorithm.lam	1.0
algorithm.use_kl_in_reward	False
actor_rollout_ref.actor.use_kl_loss	True
actor_rollout_ref.actor.kl_loss_type	low_var_kl

Appendix

Table 9: Hyperparameter settings for GRPO-family algorithms

Parameter	Value
algorithm.adv_estimator	gae
algorithm.use_kl_in_reward	False
data.train_batch_size	256
data.val_batch_size	256
data.max_prompt_length	4096
data.max_response_length	4096

Appendix

Table 10: Hyperparameter settings for PPO-family algorithms

Figure 8: Sample failed rollout generated with PPO with advantage heatmap

Figure 9: Sample failed rollout generated with TIPS with advantage heatmap

Figure 10: Sample successful rollout generated with TIPS with advantage heatmap

Figure 11: Sample successful rollout generated with PPO with advantage heatmap

Figure 12: A failure case for TIPS’ reward. We see that the policy executed a good, detailed tool call, whose results returned the correct answer Watchmen . However, the tool call was assigned a negative

\Delta

, indicating the answer was less likely after seeing the tool results, compared to after only seeing the prompt. We do note that this may just be noise, as the absolute scale of the reward is rather small.

\begin{array}[]{lccc}\hline\cr\text{Model}&\text{PPO (TFLOPs/step)}&\text{Teacher scoring (TFLOPs/step)}&\text{Relative increase (\%)}\\ \hline\cr\text{Qwen2.5 3B}&64661.474&7604.648&11.761\\ \text{Qwen2.5 7B}&136219.934&16136.034&11.846\\ \text{Qwen2.5 14B}&271071.084&32013.051&11.810\\ \text{Llama 3 8B}&147038.119&17369.665&11.813\\ \text{Qwen3 4B}&90932.211&10602.213&11.659\\ \hline\cr\end{array}

Appendix

Table 11: Comparison of PPO step FLOPs and teacher-scoring FLOPs per model.

\begin{array}[]{lrrrr}\text{Model}&q_{\text{size}}&k_{\text{size}}&v_{\text{size}}&N_{\text{dense}}\\ \hline\cr\text{Qwen2.5 3B}&2048&256&256&3.397\times 10^{9}\\ \text{Qwen2.5 7B}&3584&512&512&7.615\times 10^{9}\\ \text{Qwen2.5 14B}&5120&1024&1024&1.477\times 10^{10}\\ \text{Llama 3 8B}&4096&1024&1024&8.030\times 10^{9}\\ \text{Qwen3 4B}&4096&1024&1024&4.411\times 10^{9}\\ \end{array}

Appendix

Table 12: Model-specific inputs used in the teacher-scoring FLOP equations.

\begin{array}[]{ll}\text{Parameter}&\text{Value}\\ \hline\cr B&256\\ S&5\\ L_{\max}&3676.8\\ L_{a}&10.0\\ A&2.0\\ \{L_{i}\}&(400.0,1219.2,2038.4,2857.6,3676.8)\\ \end{array}

Appendix

Table 13: Shared constants for the teacher-scoring FLOP setup.

Model	Without Reward (s/step)	With Reward (s/step)	Overhead (%)
Qwen 2.5 3B	30.56	35.95	17.64
Qwen 2.5 7B	93.00	108.22	16.37

Appendix

Table 14: Wall-time overhead of teacher scoring during TIPS training.

Model	PPO (s/step)	TIPS (s/step)
Qwen 2.5 3B	67.62	35.95
Qwen 2.5 7B	63.39	108.22

Appendix

Table 15: Raw per-step training time comparison between TIPS and PPO.

Model	Variant	$\alpha\,(\text{s}/\text{tok})$	$\beta\,(\text{s})$	$R^{2}$	$\bar{r}\,(\text{tok})$
3 B	PPO	0.035988	18.427	0.8948	1367
3 B	TIPS	0.037113	15.463	0.0869	552
7 B	PPO	0.039683	31.235	0.1401	810
7 B	TIPS	0.050097	27.694	0.6539	1607

Appendix

Table 16: Per-step regression coefficients and response-length statistics.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning
Apr 16, 2026Zihan Liang, Yufei Ma, Ben Chen +6Retrieval-Augmented Generation RL for Language Model Reasoning
Kuaishou Technology
LOTAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning
Jul 15, 2026Qiang Zhu, Jiajun Wu, Longyi WangRL for Language Model Reasoning RL for Tool Use
Zhejiang University
Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning
May 21, 2026Zihan Liang, Yufei Ma, Ben Chen +4Tool-Augmented Language Model Agents On-Policy Self-Distillation

Apr 16, 2026J/K move · Enter open · S save

IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning

Zihan Liang, Yufei Ma, Ben Chen, Zhipeng Qian+5

Kuaishou Technology

Retrieval-Augmented Generation RL for Language Model Reasoning Credit Assignment in RL Step-Level Credit Assignment LLM Reasoning

Open paper AskPDF arXiv

Reinforcement learning has emerged as an effective paradigm for training large language models to perform search-augmented reasoning. However, existing approaches rely on trajectory-level rewards that cannot distinguish precise search queries from vague or redundant ones within a rollout group, and collapse to a near-zero gradient signal whenever every sampled trajectory fails. In this paper, we propose IG-Search, a reinforcement learning framework that introduces a step-level reward based on Information Gain (IG). For each search step, IG measures how much the retrieved documents improve the model's confidence in the gold answer relative to a counterfactual baseline of random documents, thereby reflecting the effectiveness of the underlying search query. This signal is fed back to the corresponding search-query tokens via per-token advantage modulation in GRPO, enabling fine-grained, step-level credit assignment within a rollout. Unlike prior step-level methods that require either externally annotated intermediate supervision or shared environment states across trajectories, IG-Search derives its signals from the policy's own generation probabilities, requiring no intermediate annotations beyond standard question-answer pairs. Experiments on seven single-hop and multi-hop QA benchmarks demonstrate that IG-Search achieves an average EM of 0.430 with Qwen2.5-3B, outperforming the strongest trajectory-level baseline (MR-Search) by 1.6 points and the step-level method GiGPO by 0.9 points on average across benchmarks, with particularly pronounced gains on multi-hop reasoning tasks. Despite introducing a dense step-level signal, IG-Search adds only ~6.4% to per-step training wall-clock time over the trajectory-level baseline and leaves inference latency unchanged, while still providing a meaningful gradient signal even when every sampled trajectory answers incorrectly.

Abstract

RL for Language Model Reasoning

Explore similar work

IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning

LOTAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning

Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning