RL-IF: Tracking Non-local Training Data Influence for Online Reinforcement Learning

Shixuan Liu; Cheng Tang; Yuzheng Hu; Fan Wu; Han Zhao; Jiaqi W. Ma

cs.LGOct 20, 2026

RL-IF: Tracking Non-local Training Data Influence for Online Reinforcement Learning

Authors: Shixuan Liu, Cheng Tang, Yuzheng Hu, Fan Wu, Han Zhao, Jiaqi W. Ma

Organizations: University of Illinois Urbana-Champaign

Abstract

Online reinforcement learning (RL) has achieved significant success in decision-making tasks, but modern online RL remains highly sensitive to the quality of training experience. Understanding the contribution of individual training instances is therefore central to data-centric RL, with applications ranging from experience selection to policy interpretation. Existing data attribution methods for online RL primarily focus on local influence within a single training round, overlooking the non-local data influence across multiple training rounds. In this work, we formalize data attribution in online RL through a trajectory-level leave-one-out quantity, RL-LOO, which measures how one training sample influences a downstream target after subsequent online updates have taken place. We then derive RL-IF, a first-order estimator that propagates data influence through both optimization effects and policy-induced sampling effects, and provide theoretical guarantees on its approximation accuracy under smoothness assumptions. Our analysis further disentangles these two effects and shows both theoretically and empirically that the full RL-IF estimator can often be well approximated using the optimization effect alone. Building on this observation, we develop RINSE, a practical data filtering method for online RL. Experiments on standard control benchmarks and an RLHF-style toxicity-mitigation setting show that RINSE improves training efficiency and final performance over standard PPO and local attribution baselines. These results suggest that RL-IF provides a practical and principled framework for understanding and improving online RL.

Figures & tables

Figure 1: Two-fold data influence in RL. A training occurrence $z^{*}\in D_{k}^{(l)}$ affects learning through (i) the optimization path (blue): propagating via parameter updates to $\theta_{T}^{(M)}$ , and (ii) the sampling path (orange): altering future rollout buffers $D_{k+1},D_{k+2},\dots$ . These effects are coupled through the policy-induced data distribution.

Figure 2: Left: two LunarLander samples that appear harmful locally but are beneficial in the long run. Right: fraction of late-episode samples labeled harmful vs. attribution horizon

H

Score	Ant-v4 LDS $\uparrow$	HalfCheetah-v4 LDS $\uparrow$
Random	$-0.031\pm 0.058$	$-0.063\pm 0.032$
TracIn	$0.055\pm 0.102$	$0.138\pm 0.037$
RL-IF-opt	$0.191\pm 0.037$	$0.218\pm 0.027$
RL-IF-full	$0.192\pm 0.033$	$0.218\pm 0.026$

Table 1: Linear data modeling scores of attribution methods on Ant-v4 and HalfCheetah-v4 .

Panel of Figure 3 — Figure 3: Data-selection results under rollout filtering. (a) Test returns over training rounds. (b) Relative time efficiency, sample efficiency, and peak mean return across environments. RINSE largely improves return and efficiency over standard training and local baselines.

Figure 4: Toxicity-mitigation results in the RLHF setting.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Reference-construction ablation on Ant and HalfCheetah. Curves show mean return at the reported training rounds; shaded regions indicate one standard error.

Env	Env ID	Type	Action Space
Ant	Ant-v4	MuJoCo locomotion	Continuous
HalfCheetah	HalfCheetah-v4	MuJoCo locomotion	Continuous
Hopper	Hopper-v4	MuJoCo locomotion	Continuous
LunarLander	LunarLander-v2	Box2D control	Discrete
BipedalWalker	BipedalWalker-v3	Box2D locomotion	Continuous
FrozenLake	FrozenLake-v1	Tabular navigation	Discrete

Appendix

Table 2: Summary of RL environments used in experiments. MuJoCo environments are from Todorov et al. [2012] , while Box2D and FrozenLake environments are from Gym/Gymnasium [ Brockman et al., 2016 , Towers et al., 2025 ] .

Hyperparameter	Ant	HalfCheetah	LunarLander	BipedalWalker	Hopper	FrozenLake
$N_{\text{envs}}$	8	8	1	1	8	1
Total rollouts	500	500	500	100	100	50
$n_{\text{steps}}$	512	512	1024	8192	512	2048
$n_{\text{epochs}}$	20	20	4	10	20	10
Learning rate	1e-4	1e-4	5e-3	1e-4	1e-4	5e-3
Optimizer	Adam	Adam	SGD	Adam	SGD	SGD

Appendix

Table 3: PPO/training hyperparameters per environment.

	Ant-v4		HalfCheetah-v4
Score	Pearson $\uparrow$	Spearman $\uparrow$	Pearson $\uparrow$	Spearman $\uparrow$
Random	$-0.020\pm 0.041$	$-0.031\pm 0.058$	$-0.069\pm 0.037$	$-0.063\pm 0.032$
TracIn	$0.029\pm 0.081$	$0.055\pm 0.102$	$0.166\pm 0.043$	$0.138\pm 0.037$
RL-IF-opt	$0.179\pm 0.017$	$0.191\pm 0.037$	$0.266\pm 0.015$	$0.218\pm 0.027$
RL-IF-full	$0.180\pm 0.023$	$0.192\pm 0.033$	$0.267\pm 0.016$	$0.218\pm 0.026$

Appendix

Table 4: Linear data modeling scores of attribution methods on Ant-v4 and HalfCheetah-v4 .

Method	Pearson $\rho$ (mean $\pm$ SE)
TracIn	$+0.538\pm 0.165$
RL-IF-opt	$+0.461\pm 0.161$
RL-IF-full	$\mathbf{+0.687\pm 0.041}$

Appendix

Table 5: Direct LOO attribution evaluation on FrozenLake with

H=2

. Correlations are reported as per-seed means with standard errors.

Environment	$\Lambda$	$\sqrt{L_{\max}}$	$\eta$	$\eta\sqrt{L_{\max}}/\Lambda$
FrozenLake	$1.3$	$2.4$	$5\times 10^{-3}$	$9.2\times 10^{-3}$
HalfCheetah	$65$	$12.0$	$1\times 10^{-4}$	$1.8\times 10^{-5}$
Ant	$2.4\times 10^{3}$	$12.2$	$1\times 10^{-4}$	$5.0\times 10^{-7}$

Appendix

Table 6: Empirical diagnostics of the sampling contribution.

Panel of Figure 6 — Figure 6: Comparison between TracIn and RL-IF. The top row shows the comparison between the TracIn method and the ground-truth LOO values, while the bottom row shows the comparison between RL-IF and the ground-truth LOO values under the same held-out settings. From left to right, the panels show the attribution horizon $H$ from 1 to 5. The colorbar (point index) is shared across each row and shown once on the right. Each panel keeps its own axes because the attribution and LOO magnitudes differ across horizons.

Panel of Figure 7 — Figure 7: Dynamic-horizon ablation. Fixed horizon vs. the adaptive (dynamic) horizon. The dynamic horizon matches or outperforms the fixed variant on all four environments, with the largest gains on the harder-exploration tasks.

Figure 8: Horizon ablation. Effect of the different horizons

H\in\{1,2,4,5,8,10\}

. Longer horizons speed up early learning, but the benefit saturates around

H\!\approx\!4

–

5

Figure 9: Discard-ratio ablation. Effect of the discard ratio

\rho

on Ant-v4 and HalfCheetah-v4 . Moderate ratios (

\rho{\approx}10\%

–

20\%

) are best, while very aggressive filtering can degrade performance.

Setting	Metric	Method	Ant	HalfCheetah	LunarLander	BipedalWalker	Hopper	FrozenLake
100% only	TE $\downarrow$	IIF	109.7 $\pm$ 2.0	77.5 $\pm$ 8.5	80.3 $\pm$ 6.0	93.9 $\pm$ 3.3	114.7 $\pm$ 4.0	105.2 $\pm$ 7.9
	TE $\downarrow$	RINSE	102.9 $\pm$ 2.4	60.2 $\pm$ 6.7	88.6 $\pm$ 3.9	71.1 $\pm$ 9.5	107.5 $\pm$ 4.5	104.2 $\pm$ 2.7
	SE $\downarrow$	IIF	85.8 $\pm$ 1.8	59.5 $\pm$ 6.5	55.4 $\pm$ 4.3	74.1 $\pm$ 2.8	87.9 $\pm$ 3.0	51.0 $\pm$ 3.9
	SE $\downarrow$	RINSE	74.7 $\pm$ 1.8	42.8 $\pm$ 4.7	49.1 $\pm$ 2.1	49.7 $\pm$ 7.0	77.2 $\pm$ 3.4	48.0 $\pm$ 1.6
90/95/100%	TE $\downarrow$	IIF	110.2 $\pm$ 3.7	75.0 $\pm$ 11.0	76.1 $\pm$ 8.5	94.9 $\pm$ 6.0	94.5 $\pm$ 18.8	133.4 $\pm$ 26.0
	TE $\downarrow$	RINSE	103.0 $\pm$ 4.9	63.9 $\pm$ 10.4	89.9 $\pm$ 6.8	67.9 $\pm$ 13.6	88.9 $\pm$ 16.7	116.3 $\pm$ 12.5

Appendix

Table 7: Relative time efficiency (TE) and sample efficiency (SE) across environments. All values are normalized to the PPO baseline (

100\%

); lower is better. Best results per column are in bold.

Environment	Collect	Train	Attribution	Replay	Base	Overhead	Overhead / Base
Ant-v4	1.46	17.66	15.90	5.51	19.12	21.41	112%
HalfCheetah-v4	0.86	17.13	13.85	5.36	17.99	19.21	107%
Hopper-v4	1.06	26.45	14.28	5.66	27.51	19.94	72%
BipedalWalker-v3	7.24	26.95	15.56	5.66	34.19	21.22	62%
LunarLander-v2	0.75	0.97	0.62	1.44	1.72	2.06	120%
FrozenLake-v1	1.43	3.20	3.07	2.87	4.63	5.94	128%

Appendix

Table 8: Wall-clock runtime in seconds per round on the traditional RL environments.

Figure 10: Training curves of Standard, IIF, and RINSE on RLHF toxicity reduction.

Metric	RINSE	Unperturbed baseline
Median perplexity $\downarrow$	$\mathbf{43.4\pm 2.6}$	$50.5\pm 5.3$
Distinct-2 $\uparrow$	$\mathbf{0.79\pm 0.01}$	$0.72\pm 0.03$
Distinct-3 $\uparrow$	$\mathbf{0.96\pm 0.00}$	$0.92\pm 0.02$
Seq-rep-4 $\downarrow$	$\mathbf{0.0037\pm 0.0013}$	$0.0046\pm 0.0010$

Appendix

Table 9: Generation quality on held-out Wiki-Toxic prompts at matched toxicity.

Training Data Attribution

Training data attribution identifies which examples, sources, or groups shape a model’s predictions, capabilities, and failures.

See all 81 papers

Which Rollout Taught It That? BehaviorTrace and the Limits of Training-Data Attribution in Online RL
Oct 7, 2026Amit NautiyalGradient-Based Attribution Training Data Attribution
Independent Researcher
Efficient RLVR Training via Weighted Mutual Information Data Selection
Mar 2, 2026Xinyu Zhou, Boyu Zhu, Haotian Zhang +2Reinforcement Learning RL for Language Model Reasoning
HKUST(GZ) · UCL · Kuaishou Technology +1
Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training
Apr 2, 2026Dong Shu, Denghui Zhang, Jessica HullmanRL Post-Training RL for Language Model Reasoning
Northwestern University · Stevens Institute of Technology

Oct 7, 2026J/K move · Enter open · S save

Which Rollout Taught It That? BehaviorTrace and the Limits of Training-Data Attribution in Online RL

Amit Nautiyal

Independent Researcher

Gradient-Based Attribution Training Data Attribution RL Fine-Tuning

Open paper AskPDF arXiv

When reinforcement learning teaches a language model a new behavior, can we find the training rollouts that taught it? And when an attribution method says it can, how do we know the answer is real? We study both questions on online RL fine-tuning with GRPO, using a planted behavior with a known cause. We release BehaviorTrace, an open evaluation harness that combines full-gradient sketching, the planted-behavior setup, and controls for gradient magnitude, fluency, headroom, and variation across seeds and generation draws. Across three seeds on Qwen2.5-1.5B, much of the apparent attribution signal comes from confounds. A control that ranks training steps by gradient size alone, with no behavior target, reaches 4.2 to 4.5 times chance and matches or beats the best targeted estimator on two of three seeds. At saturated checkpoints, model fluency predicts the behavior label at least as well as every gradient method we compared it with. Once fluency is controlled, the per-rollout results change from seed to seed and from one generation draw to the next, so a single run cannot settle the question. One signal does hold on all three seeds. The gradient of the trigger tokens aligns with a target built where the behavior actually occurs. We turn these findings into a checklist for evaluating attribution in RL. We test existing estimators, including GAS (renormalized TracInCP) and a TRAK-style estimator, and do not propose a new one.

Abstract

Training Data Attribution

Explore similar work

Which Rollout Taught It That? BehaviorTrace and the Limits of Training-Data Attribution in Online RL

Efficient RLVR Training via Weighted Mutual Information Data Selection

Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training