Probe the Harness: Setup Checks for Stale-Data RL Comparisons in Language Models

Taiheng Pan

cs.LGOct 2, 2026

Probe the Harness: Setup Checks for Stale-Data RL Comparisons in Language Models

Authors: Taiheng Pan

Organizations: The University of Melbourne

Abstract

Methods for training language models on stale samples are judged by comparisons against importance-corrected baselines. We show that details of the experimental harness can reverse the observed ranking of methods, and we introduce PTH (Probe The Harness), a set of checks that makes the harness visible. Our case is a comparison between SAN, a behaviour-free method, and truncated importance sampling (TIS) on verl and in a single-GPU trainer, in which SAN first finished ahead in both stacks. Four details of the harness changed this comparison: the PPO ratio was taken against the learner's own recomputed probabilities, the data seed did not reach the TIS arm, the replay queue reused its first batch for 33 updates, and two loss normalisers differed from their description. In each case the logged quantity looked consistent with a working setup, while the quantity that defines the comparison went unchecked. With the harness checked, TIS matches SAN on verl, and in the trainer TIS learns steadily while SAN keeps a margin. We contribute the signature of each detail and its effect on the comparison, reference results for TIS and uncorrected GRPO under sampler lag, and the PTH checklist.

Figures & tables

Layer	Logged quantity	Defining quantity	Sec.	Check
Policy in the PPO ratio	Clip fraction, zero throughout	Ratio to the sampler’s recorded probability	4.1	1
Configuration of each arm	Run labels, three distinct results	Data seed each arm received	4.2	2
Data for each update	Data age, as designed	Identity of the training batch	4.3	3
Implemented loss	Short run trains and logs	Loss equals its written equation	4.4	4

Table 1: Four layers of a stale-data harness. In each layer the logged quantity looked consistent with a working setup, and the defining quantity determined the comparison. Each row has a section and a PTH check.

Figure 1: The PPO-clip arm trains without an active clip. verl, Qwen2.5-Math-1.5B, GSM8K, refresh interval 96. (a) Greedy validation accuracy, one line per data seed; the dotted line marks the refresh. The PPO-clip arm (orange) drops after update 80 in every seed;

\text{SAN}^{+}

(blue) and TIS (green) end level. (b) For the PPO-clip runs, the per-token KL between sampler and learner grows by four orders of magnitude before the refresh, while the clip fraction stays at zero on every update.

Stack	Setting	Baseline	SAN family
verl	refresh interval 64	PPO clip: .743 .778 .777	SAN : .805 .806 .787
		TIS : .763 .776 .770	$\text{SAN}^{+}$ : .791 .792 .784
			$\text{SAN}^{\mathrm{ref}}$ : .788 .803 .794
verl	refresh interval 96	PPO clip: .114 .280 .099	$\text{SAN}^{+}$ : .773 .752 .772
verl	interval 64, 300 updates	PPO clip: .723 (peak .822)	$\text{SAN}^{+}$ : .869
trainer	1.5B, replay age 32	TIS : .002 .000 .002 .002	SAN : .858 .731 .781

Table 2: Initial comparison. Final greedy accuracy on verl (one entry per run) and last-10-update reward in the trainer.

Figure 2: The replay warm-up accounts for the TIS drop. Single-GPU trainer, Qwen2.5-Math-1.5B, GSM8K, replay age 32. (a) Index of the batch each update trains on. Both queues give every update the same data age; the warm-up queue trains on batch 0 for the first 33 updates. (b, c) Reward on freshly generated batches, 10-update running mean, under the unique-prefill queue (solid, two runs per method) and the warm-up queue (dashed, four TIS and three SAN runs). Under the warm-up queue every TIS run falls to zero after update 50; under unique prefill both keep learning. SAN is similar under both queues.

	interval 64, lr $2{\times}10^{-6}$			interval 96, lr $2{\times}10^{-6}$			interval 96, lr $4{\times}10^{-6}$
	default	43	44	default	43	44	default
Uncorrected GRPO	.743	.778	.777	.114	.280	.099
TIS	.763	.785	.808	.764	.781	.785	.837
SAN	.805	.806	.787	.738
$\text{SAN}^{+}$	.791	.792	.784	.773	.752	.772	.733
$\text{SAN}^{\mathrm{ref}}$	.788	.803	.794

Table 3: verl with the harness checked. Final greedy GSM8K accuracy at update 100, paired by data seed (default order, 43, 44). At refresh interval 64, the TIS default-order entry is the original first run and the seed-43 and seed-44 entries are reruns with those seeds. Blank cells were not run.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Coefficient	404	505	Mean
TIS	$\hat{A}\min(\pi_{\theta}/q,2)$	68.4	67.2	67.8
ASymPO, stabilised loss		69.9	67.2	68.6
Plain sequence normalisation	$\hat{A}/S$	71.5	70.7	71.1
Sequence gain only	$\hat{A}/D$	74.6	74.2	74.4
SAN	$\hat{A}w_{t}/D$	75.4	77.0	76.2
Token weights only	$\hat{A}w_{t}/S$	77.3	78.1	77.7

Appendix

Table 4: Trainer, unique-prefill queue. Replay age 32, Qwen2.5-Math-1.5B, no KL penalty. Greedy accuracy (%) on 256 GSM8K test problems at update 100, seeds 404 and 505; 21.5% at update 0.

Model	lr	TIS	SAN
Qwen2.5-Math-1.5B	$10^{-5}$	.002 (404), .000 (505), .002 (404), .002 (505)	.858 (404), .731 (505), .781 (606)
later code version	$10^{-5}$	.273 (606), .108 (707), .004 (808), .150 (909)	.833 (707)
Qwen2.5-3B-Instruct	$5{\times}10^{-6}$	.821 (404)	.860 (404)
Qwen2.5-Math-7B	$3{\times}10^{-6}$	.569 (404)	.616 (404)
Llama-3.2-3B-Instruct	$5{\times}10^{-6}$	.800 (404), .717 (505)	.771 (404)
Qwen2.5-Math-1.5B, LoRA 64	$10^{-4}$	.817 (404), .619 (505)	.721 (404), .815 (505)

Appendix

Table 7

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 438 papers

ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning
Oct 17, 2026Diyi Hu, Ruizhong Qiu, Lin Qiu +2RL for Language Models RL for Language Model Reasoning
Independent Researcher · University of Illinois Urbana-Champaign · Meta
Cool the Sampler, Not the Learner: Sampling Temperature Moves the Staleness Cliff of Importance-Corrected GRPO
Sep 29, 2026Taiheng PanRL for Language Models Reinforcement Learning
The University of Melbourne
Probe with Participation Trophies: Random-Reward RL as a Probe of LLM Capability
Oct 1, 2026Yu Mao, Lei Yu, Zining Zhu +6RL for Language Models Reinforcement Learning
University of Toronto · Stevens Institute of Technology · University of California, Santa Cruz +3

Oct 17, 2026J/K move · Enter open · S save

ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning

Diyi Hu, Ruizhong Qiu, Lin Qiu, Hanghang Tong+1

Independent Researcher · University of Illinois Urbana-Champaign · Meta

RL for Language Models RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Group Relative Policy Optimization

Open paper AskPDF arXiv

Reinforcement learning from verifiable rewards assigns one terminal reward to an entire reasoning trajectory, obscuring how individual reasoning steps contributed to the outcome. GRPO broadcasts a group-normalized advantage to every response token, while finer-grained approaches often rely on dense auxiliary rollouts or branching schemes that can also alter the data used for policy updates. We cast the missing middle as budgeted measurement: with a limited rollout budget, which partial solutions should be evaluated, and how should their value estimates refine credit? ProbePO selects partial-solution states with a reward-free uncertainty score and estimates their expected verifier rewards under a capped continuation budget. ProbePO redistributes, rather than replaces, GRPO's trajectory-level advantage across tokens while preserving each trajectory's mean token-level advantage; probe continuations serve only as measurements. Across three DeepSeekMath and Qwen-family models spanning 1.5B-7B parameters, ProbePO consistently outperforms GRPO and achieves the highest accuracy among the four evaluated methods in 16 of 18 model-benchmark comparisons. Against dense-probing VinePPO on DeepSeekMath-7B, ProbePO achieves higher accuracy on all six benchmarks with 76% less end-to-end training time.

Abstract

RL for Language Models

Explore similar work

ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning

Cool the Sampler, Not the Learner: Sampling Temperature Moves the Staleness Cliff of Importance-Corrected GRPO

Probe with Participation Trophies: Random-Reward RL as a Probe of LLM Capability