Online reinforcement learning (RL) has achieved significant success in decision-making tasks, but modern online RL remains highly sensitive to the quality of training experience. Understanding the contribution of individual training instances is therefore central to data-centric RL, with applications ranging from experience selection to policy interpretation. Existing data attribution methods for online RL primarily focus on local influence within a single training round, overlooking the non-local data influence across multiple training rounds. In this work, we formalize data attribution in online RL through a trajectory-level leave-one-out quantity, RL-LOO, which measures how one training sample influences a downstream target after subsequent online updates have taken place. We then derive RL-IF, a first-order estimator that propagates data influence through both optimization effects and policy-induced sampling effects, and provide theoretical guarantees on its approximation accuracy under smoothness assumptions. Our analysis further disentangles these two effects and shows both theoretically and empirically that the full RL-IF estimator can often be well approximated using the optimization effect alone. Building on this observation, we develop RINSE, a practical data filtering method for online RL. Experiments on standard control benchmarks and an RLHF-style toxicity-mitigation setting show that RINSE improves training efficiency and final performance over standard PPO and local attribution baselines. These results suggest that RL-IF provides a practical and principled framework for understanding and improving online RL.
Figures & tables
Figure 1: Two-fold data influence in RL. A training occurrence z∗∈Dk(l) affects learning through (i) the optimization path (blue): propagating via parameter updates to θT(M) , and (ii) the sampling path (orange): altering future rollout buffers Dk+1,Dk+2,… . These effects are coupled through the policy-induced data distribution.
Figure 2: Left: two LunarLander samples that appear harmful locally but are beneficial in the long run. Right: fraction of late-episode samples labeled harmful vs. attribution horizon H .
Score
Ant-v4 LDS ↑
HalfCheetah-v4 LDS ↑
Random
−0.031±0.058
−0.063±0.032
TracIn
0.055±0.102
0.138±0.037
RL-IF-opt
0.191±0.037
0.218±0.027
RL-IF-full
0.192±0.033
0.218±0.026
Table 1: Linear data modeling scores of attribution methods on Ant-v4 and HalfCheetah-v4 .
Figure 3: Data-selection results under rollout filtering. (a) Test returns over training rounds. (b) Relative time efficiency, sample efficiency, and peak mean return across environments. RINSE largely improves return and efficiency over standard training and local baselines.
Figure 4: Toxicity-mitigation results in the RLHF setting.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Reference-construction ablation on Ant and HalfCheetah. Curves show mean return at the reported training rounds; shaded regions indicate one standard error.
Env
Env ID
Type
Action Space
Ant
Ant-v4
MuJoCo locomotion
Continuous
HalfCheetah
HalfCheetah-v4
MuJoCo locomotion
Continuous
Hopper
Hopper-v4
MuJoCo locomotion
Continuous
LunarLander
LunarLander-v2
Box2D control
Discrete
BipedalWalker
BipedalWalker-v3
Box2D locomotion
Continuous
FrozenLake
FrozenLake-v1
Tabular navigation
Discrete
Appendix
Table 2: Summary of RL environments used in experiments. MuJoCo environments are from Todorov et al. [2012] , while Box2D and FrozenLake environments are from Gym/Gymnasium [ Brockman et al., 2016 , Towers et al., 2025 ] .
Hyperparameter
Ant
HalfCheetah
LunarLander
BipedalWalker
Hopper
FrozenLake
Nenvs
8
8
1
1
8
1
Total rollouts
500
500
500
100
100
50
nsteps
512
512
1024
8192
512
2048
nepochs
20
20
4
10
20
10
Learning rate
1e-4
1e-4
5e-3
1e-4
1e-4
5e-3
Optimizer
Adam
Adam
SGD
Adam
SGD
SGD
Appendix
Table 3: PPO/training hyperparameters per environment.
Ant-v4
HalfCheetah-v4
Score
Pearson ↑
Spearman ↑
Pearson ↑
Spearman ↑
Random
−0.020±0.041
−0.031±0.058
−0.069±0.037
−0.063±0.032
TracIn
0.029±0.081
0.055±0.102
0.166±0.043
0.138±0.037
RL-IF-opt
0.179±0.017
0.191±0.037
0.266±0.015
0.218±0.027
RL-IF-full
0.180±0.023
0.192±0.033
0.267±0.016
0.218±0.026
Appendix
Table 4: Linear data modeling scores of attribution methods on Ant-v4 and HalfCheetah-v4 .
Method
Pearson ρ (mean ± SE)
TracIn
+0.538±0.165
RL-IF-opt
+0.461±0.161
RL-IF-full
+0.687±0.041
Appendix
Table 5: Direct LOO attribution evaluation on FrozenLake with H=2 . Correlations are reported as per-seed means with standard errors.
Environment
Λ
Lmax
η
ηLmax/Λ
FrozenLake
1.3
2.4
5×10−3
9.2×10−3
HalfCheetah
65
12.0
1×10−4
1.8×10−5
Ant
2.4×103
12.2
1×10−4
5.0×10−7
Appendix
Table 6: Empirical diagnostics of the sampling contribution.
Figure 6: Comparison between TracIn and RL-IF. The top row shows the comparison between the TracIn method and the ground-truth LOO values, while the bottom row shows the comparison between RL-IF and the ground-truth LOO values under the same held-out settings. From left to right, the panels show the attribution horizon H from 1 to 5. The colorbar (point index) is shared across each row and shown once on the right. Each panel keeps its own axes because the attribution and LOO magnitudes differ across horizons.
Figure 7: Dynamic-horizon ablation. Fixed horizon vs. the adaptive (dynamic) horizon. The dynamic horizon matches or outperforms the fixed variant on all four environments, with the largest gains on the harder-exploration tasks.
Figure 8: Horizon ablation. Effect of the different horizons H∈{1,2,4,5,8,10} . Longer horizons speed up early learning, but the benefit saturates around H≈4 – 5 .
Figure 9: Discard-ratio ablation. Effect of the discard ratio ρ on Ant-v4 and HalfCheetah-v4 . Moderate ratios ( ρ≈10% – 20% ) are best, while very aggressive filtering can degrade performance.
Setting
Metric
Method
Ant
HalfCheetah
LunarLander
BipedalWalker
Hopper
FrozenLake
100% only
TE ↓
IIF
109.7 ± 2.0
77.5 ± 8.5
80.3 ± 6.0
93.9 ± 3.3
114.7 ± 4.0
105.2 ± 7.9
RINSE
102.9 ± 2.4
60.2 ± 6.7
88.6 ± 3.9
71.1 ± 9.5
107.5 ± 4.5
104.2 ± 2.7
SE ↓
IIF
85.8 ± 1.8
59.5 ± 6.5
55.4 ± 4.3
74.1 ± 2.8
87.9 ± 3.0
51.0 ± 3.9
RINSE
74.7 ± 1.8
42.8 ± 4.7
49.1 ± 2.1
49.7 ± 7.0
77.2 ± 3.4
48.0 ± 1.6
90/95/100%
TE ↓
IIF
110.2 ± 3.7
75.0 ± 11.0
76.1 ± 8.5
94.9 ± 6.0
94.5 ± 18.8
133.4 ± 26.0
RINSE
103.0 ± 4.9
63.9 ± 10.4
89.9 ± 6.8
67.9 ± 13.6
88.9 ± 16.7
116.3 ± 12.5
Appendix
Table 7: Relative time efficiency (TE) and sample efficiency (SE) across environments. All values are normalized to the PPO baseline ( 100% ); lower is better. Best results per column are in bold.
Environment
Collect
Train
Attribution
Replay
Base
Overhead
Overhead / Base
Ant-v4
1.46
17.66
15.90
5.51
19.12
21.41
112%
HalfCheetah-v4
0.86
17.13
13.85
5.36
17.99
19.21
107%
Hopper-v4
1.06
26.45
14.28
5.66
27.51
19.94
72%
BipedalWalker-v3
7.24
26.95
15.56
5.66
34.19
21.22
62%
LunarLander-v2
0.75
0.97
0.62
1.44
1.72
2.06
120%
FrozenLake-v1
1.43
3.20
3.07
2.87
4.63
5.94
128%
Appendix
Table 8: Wall-clock runtime in seconds per round on the traditional RL environments.
Figure 10: Training curves of Standard, IIF, and RINSE on RLHF toxicity reduction.
Figure 19
Metric
RINSE
Unperturbed baseline
Median perplexity ↓
43.4±2.6
50.5±5.3
Distinct-2 ↑
0.79±0.01
0.72±0.03
Distinct-3 ↑
0.96±0.00
0.92±0.02
Seq-rep-4 ↓
0.0037±0.0013
0.0046±0.0010
Appendix
Table 9: Generation quality on held-out Wiki-Toxic prompts at matched toxicity.
When reinforcement learning teaches a language model a new behavior, can we find the training rollouts that taught it? And when an attribution method says it can, how do we know the answer is real? We study both questions on online RL fine-tuning with GRPO, using a planted behavior with a known cause. We release BehaviorTrace, an open evaluation harness that combines full-gradient sketching, the planted-behavior setup, and controls for gradient magnitude, fluency, headroom, and variation across seeds and generation draws. Across three seeds on Qwen2.5-1.5B, much of the apparent attribution signal comes from confounds. A control that ranks training steps by gradient size alone, with no behavior target, reaches 4.2 to 4.5 times chance and matches or beats the best targeted estimator on two of three seeds. At saturated checkpoints, model fluency predicts the behavior label at least as well as every gradient method we compared it with. Once fluency is controlled, the per-rollout results change from seed to seed and from one generation draw to the next, so a single run cannot settle the question. One signal does hold on all three seeds. The gradient of the trigger tokens aligns with a target built where the behavior actually occurs. We turn these findings into a checklist for evaluating attribution in RL. We test existing estimators, including GAS (renormalized TracInCP) and a TRAK-style estimator, and do not propose a new one.
Reinforcement learning (RL) plays a central role in improving the reasoning and alignment of large language models, yet its efficiency critically depends on how training data are selected. Existing online selection strategies predominantly rely on difficulty-based heuristics, favouring datapoints with intermediate success rates, implicitly equating difficulty with informativeness and neglecting epistemic uncertainty arising from limited evidence. We introduce InSight, an INformation-guided data SamplInG metHod for RL Training, grounded in a weighted mutual information objective. By modeling data outcomes with Bayesian latent success rates, we show that expected uncertainty reduction decomposes into complementary difficulty- and evidence-dependent components, revealing a fundamental limitation of difficulty-only selection. Leveraging this observation, InSight constructs a stable acquisition score based on the mean belief of datapoints' success rather than noisy sampled outcomes, and naturally extends to multi-rollout settings common in reinforcement learning with verifiable rewards (RLVR). Extensive experiments demonstrate that InSight consistently achieves state-of-the-art performance and improves training efficiency, including a +1.41 average gain on Planning & Mathmatics benchmarks, +1.01 improvement on general reasoning, and up to ~2.2x acceleration, with negligible additional computational overhead.
Traditional RL algorithms like Proximal Policy Optimization (PPO) typically train on the entire rollout buffer, operating under the assumption that all generated episodes provide a beneficial optimization signal. However, these episodes frequently contain noisy or unfaithful reasoning, which can degrade model performance and slow down training. In this paper, we propose Influence-Guided PPO (I-PPO), a novel framework that integrates data attribution into the RL post-training loop. By calculating an influence score for each episode using a gradient-based approximation, I-PPO identifies and eliminates episodes that are anti-aligned with a validation gradient. Our experiments demonstrate that I-PPO consistently outperforms SFT and PPO baselines. We show that our filtering process acts as an intrinsic early stopping mechanism, accelerating training efficiency while effectively reducing unfaithful CoT reasoning.
Dong Shu, Denghui Zhang, Jessica Hullman
Northwestern University · Stevens Institute of Technology