LocusRL: Diagnosing LLM Reward and Policy Interventions in Competitive Games

Chengyu Luan; Bo Xin; Songyan Guo; Yuxiang Zuo; Ahmed Yazdan; Jiahang Li; Yicheng Liu

cs.LGOct 3, 2026

LocusRL: Diagnosing LLM Reward and Policy Interventions in Competitive Games

Authors: Chengyu Luan, Bo Xin, Songyan Guo, Yuxiang Zuo, Ahmed Yazdan, Jiahang Li, Yicheng Liu

Organizations: Harbin Institute of Technology

Abstract

Large language models can intervene in reinforcement learning through both reward design and action selection, yet aggregate performance offers an incomplete account of what these interventions actually do. Similar returns can conceal different learning mechanisms, while plausible rewards can induce undesirable behavior. We introduce LocusRL, a diagnostic framework that connects controlled reward-policy comparisons with audits of reward judgments, signal delivery, optimization objectives, and executed actions. The framework traces performance differences to testable explanations and checks targeted corrections through executable rules and counterfactual replay. Across two evaluation batches covering ten Connect Four training seeds, we uncover seed-dependent reversals in intervention effects and show how tracing actual updates changes their interpretation: historical Qwen training operates through reward-weighted teacher-action likelihood. A separate matched three-seed reward-direction experiment distinguishes sensitivity to a learning signal from its usefulness. With terminal rewards held fixed, a sign-reversed dense oracle yields a 2.8% aggregate win rate, compared with 57.2% for terminal-only training and 46.7% for the positive dense oracle. Thus, a reward can strongly influence learning without improving performance. At the decision level, counterfactual replay verifies a winning correction to a diagnosed action error. Complementary experiments in Leduc and reward-validation studies in Goofspiel extend the analysis to imperfect-information settings, revealing how reference-label definitions and validation-data exposure affect intervention assessment. Together, these findings show why evaluating LLM interventions requires tracing how their outputs become learning signals and actions. LocusRL turns aggregate outcomes into actionable diagnoses and verifiable corrections.

Figures & tables

Figure 1: Historical Connect Four design. Base–Reward and Policy–Hybrid are within-family contrasts; because the policy family also changes architecture, data, and update rule, Eq. 2 is descriptive rather than a single-factor causal decomposition. Dense reward is an exact frozen-cache lookup; optimization and execution are audited downstream.

Check	Quantified object / decision basis	Follow-up action
Reward semantics	Agreement count / total under a fixed reference; report category and denominator	Recheck the reference, then audit frozen outputs
Signal delivery	Cache hits / queries; optimization samples receiving nonzero dense reward	Separate queryability, realized delivery, and nonzero signal
Optimization objective	Numerical match between logged loss and declared formula; sampled–update log-probability discrepancy	Verify the implemented objective; distinguish teacher-action learning from on-policy updates
Action execution	Proposed, parsed, fallback, and executed actions; replayed terminal state and payoff	Localize the divergence and verify a concrete replacement
Evidence validity	Checkpoint–seed pairing; repeated trajectories; historical exposure overlap	Set the statistical unit, re-evaluate under a common contract, or revise validation status

Table 1: Operational LocusRL checks. Metrics retain their own denominators; a failed check determines the next diagnostic or corrective action rather than being collapsed into one score.

Figure 2: Execution diagnosis and terminal replay. The recorded Hybrid fallback misses an immediate win. Open diamonds mark 20 replay-certified repairs that share one first-repair board across two checkpoints; they are not 20 independent tactical situations.

Figure 3: Historical seed-wise contrasts, keeping the three-seed and seven-seed batches separate. S1–S10 denote training seeds 20260800–20260809; runtime-label exceptions belong to the historical evaluations. Common-runtime results are reported in Table 2 ; exact coordinates are in Table 26 .

Training batch	Games/arm	Base	Reward	Policy	Hybrid
Initial 3 seeds	60	37	37	34	29
Added 7 seeds	140	96	96	97	96
Descriptive total	200	133	133	131	125

Table 2: Common-runtime evaluation of historical checkpoints. Entries are wins; reusing trained checkpoints adds evaluation records, not training replicates.

Figure 4: Historical numerical objective check. The declared signed rule predicts a negative ratio for negative common advantage, but the archived loss/NLL ratio remains positive. These are logged ratios, not reconstructed per-example gradient coefficients; all 1,400 added-batch ratios lie in $[0.25,1.25]$ .

Training seed	Policy wins	Hybrid wins	Hybrid–Policy
20260800	12/20	10/20	$-10.0$ pp
20260801	15/20	11/20	$-20.0$ pp
20260802	12/20	12/20	$0.0$ pp
Descriptive total	39/60	33/60	$-10.0$ pp

Table 3: Formal evaluation of the new on-policy checkpoints. Each cell has 20 games; there are no draws. The replication blocks are three paired training seeds, with two negative Hybrid–Policy contrasts and one tie.

Training seed	Terminal only	Positive oracle	Sign-reversed oracle
20260800	38	30	1
20260801	27	27	1
20260802	38	27	3
Total	103/180	84/180	5/180
Win rate	57.2%	46.7%	2.8%

Table 4: Step-100 reward-direction control: wins out of 60 per seed. Positive denotes the original sign of the historical rule oracle; the oracle is not assumed tactically perfect under the revised reference.

Figure 5: Frozen-cache audit under the revised reference. Left: category-wise semantic fidelity; right: exact-match support under separate training/evaluation lookup denominators. Forced-loss agreement is $88.32\%$ and positive avoidable-error scores total $556/3{,}444$ ; Table 16 gives all counts.

Reward	Correct	Accuracy	Failures	Formal cache
Leduc v0.1	9/24	37.5%	1/48	Stopped
Goofspiel v0.1	7/29	24.1%	0/58	Stopped
Goofspiel v0.2	18/29	62.1%	2/58	Stopped

Table 5: Reward-release decisions under the recorded gate criteria. Accuracy retains every registered pair; failures count action requests. Historical freeze and exposure evidence is detailed separately in Appendices J and K .

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Audit object	Evidence or estimator	Intervention / validation
Alignment Auditor ( Bou et al., 2026 )	Implicit LLM objectives	Bayesian IRL and posterior uncertainty	Inferred-reward refinement and downstream RLHF validation
Hidden-objective audits ( Marks et al., 2025 )	Undesired internal objective	Behavioral, interpretability, and training-data investigations	Blind auditing exercise on a model with a hidden objective
IR 3 ( Beigi et al., 2026 )	Reward-hacking features of implicit objectives	Contrastive IRL and feature decomposition	Targeted mitigation with behavior/capability evaluation
Spurious Rewards ( Shao et al., 2025 )	RLVR reward informativeness	Alternative reward signals across model families	Measured training responses to spurious rewards
RATE ( Reber et al., 2025 )	Reward-score attribute sensitivity	Rewrites with counterfactual bias correction	Attribute-effect estimation under stated assumptions
LocusRL	Realized LLM interventions in composed learners	System contrasts, objective and support logs, action traces, and validation histories	Controlled follow-ups, terminal-replay correction, and reward-release decisions

Appendix

Table 6: Closest audit methods and their relation to the present contribution. A local terminal replay and a policy-level repair are different validation units.

Line of work	LLM locus	Primary question identified	Relation to system comparison
ELLM; Reward Design; Motif ( Du et al., 2023 ; Kwon et al., 2023 ; Klissarov et al., 2024 )	Goal proposal, proxy reward, or preference-derived intrinsic reward	Can language-model knowledge improve exploration or replace costly reward engineering?	Whether the same reward artifact retains its effect after replacing the downstream policy family.
Text2Reward; Eureka ( Xie et al., 2024 ; Ma et al., 2024 )	Executable dense-reward synthesis and refinement	Can an LLM generate reward code whose induced policy achieves high return?	Reward presence is not crossed with conventional versus LLM policy systems using one frozen artifact.
GLAM; RAGEN; SPIRAL ( Carta et al., 2023 ; Wang et al., 2025 ; Liu et al., 2025 )	LLM policy or self-play policy	Can language policies be grounded and stably optimized through interaction?	The reward channel is not independently toggled under matched controls, and generation failures become part of the policy treatment.
RLAnything ( Wang et al., 2026 )	Jointly adapting policy, reward model, and environment	Do mutually adapting components improve an end-to-end LLM-agent system?	Co-adaptation is the method; it does not estimate the transfer of one frozen reward across policy families.
Policy-invariant LLM shaping ( Hounwanou et al., 2026 )	LLM planner and potential-based shaping	Can LLM feedback be incorporated without changing the optimal-policy set?	The theoretical guarantee does not estimate empirical interaction with an LLM policy; reported training evidence is a small pipeline validation.
LocusRL	Traditional/LLM policy $\times$ absent/present frozen LLM reward	Does the identical reward compose across policy systems, and is the realized treatment semantically and operationally valid?	Connect Four supplies system comparisons and controls; Leduc adds training/evaluation evidence, and Goofspiel adds reward-reference and validation diagnostics.

Appendix

Table 7: Representative LLM-for-RL work organized by the intervention it identifies. The final column states what is still needed to test the composed-system diagnostic question studied by LocusRL.

Category	Seat 0	Seat 1	Overall	Failed
Immediate win	$8/10$	$9/10$	$17/20$	$0$
Required block	$7/8$	$7/8$	$14/16$	$4$
Threat creation	$7/8$	$7/9$	$14/17$	$3$
Good move	$10/10$	$10/10$	$20/20$	$0$
Overall	$32/36$	$33/37$	$65/73$	$7$

Appendix

Table 8: Historical reasoning-enabled DeepSeek confirmation. Correct counts are conditional on two valid scores; failed pairs remain in the conservative denominator.

Configuration	Correct/scorable	Scorable acc.	Conservative	Ties	Reversed	Failed
DeepSeek v4 Flash, low-token	$26/80$	$32.50\%$	$32.50\%$	25	29	0
Mimo 2.5 Pro, high, 8192	$35/80$	$43.75\%$	$43.75\%$	27	18	0
GLM-5.3-Flash, high	$71/76$	$93.42\%$	$88.75\%$	2	3	4

Appendix

Table 9: Real-API reward-judge configuration screen on the frozen 80-pair dual-seat set. “Scorable” excludes exhausted API pairs but treats ties as incorrect. Conservative accuracy counts all failures in the denominator.

	Base / Reward	Initial Policy / Hybrid	Added Policy / Hybrid
Backbone	Masked-action MLP	Qwen2.5-3B-Instruct	Qwen2.5-3B-Instruct
Training actions	Learner rollouts	Heuristic teacher actions	Heuristic teacher actions
Configured / observed budget	100 updates $\times$ 200 episodes	TRAIN_MAX_STEPS=10 configured; completed updates and consumed-example order unavailable	100 logged updates/run; 1,400 updates across 14 runs
Objective	Centered-return policy gradient	Weighted teacher-action NLL; exact historical weight mapping not recovered	Positive bounded loss/NLL ratios verified numerically
Dense intervention	Reward adds frozen score	Hybrid adds frozen score to archived return signal	Same frozen reward mechanism

Appendix

Table 10: Implemented Connect Four learning systems and budget evidence. Historical Qwen rows separate what is verified from what remains recoverable.

Audit layer	Evidence object	Decision or next intervention
Execution identity	Train/eval source matrix; actual loss versus declared formula	Reconcile source and logging; hold weights fixed for common-runtime evaluation
Reward semantics	Reachable action pairs; executable reference counterexamples; full-cache labels	Validate the reference, then distinguish intact bytes from valid scores
Realized support	Lookup, nonzero signal, and direction-consistent exposure counts	Determine whether the nominal reward reaches the learner’s states
Action execution	Proposed/parsed/executed actions; first divergence; terminal certificate	Identify a concrete wrapper repair; freeze it for prospective testing
Multiplicity	Paired trajectories and training-seed contrasts	Separate repeated paths from independent retraining evidence
Exposure history	Historical opened-state union; frozen splits and stop record	Keep confirmation independent; stop before formal cache or training

Appendix

Table 11: Diagnostic workflow. Each audit produces an inspectable object and specifies the next action when its check does not close the question.

Environment	Protocol	Max steps	Return	Status
Connect Four	Perfect information; columns $0,\ldots,6$	42	$-1,0,+1$	Four-arm results complete
Leduc-style poker	Hidden cards; simplified betting; no re-raise	6	$[-7,+7]$	Reward gate stopped; terminal-reward Base/Policy training and 360-game evaluation complete
Goofspiel	Public remaining bids; pending bid and future prize order hidden	10	$-1,0,+1$	Validity gate failed; stopped before training

Appendix

Table 12: Environment and evidence scope. Adapter availability does not imply completed policy-performance experiments.

Batch	Training seed	Evaluation seed(s)	Games/checkpoint	Games/arm
Initial	20260800	20260810	60
	20260801	20260811	60
	20260802	20260812	60	180
Added	20260803–20260809	20260810, 20260811, 20260812	180	1,260

Appendix

Table 13: Training-to-evaluation correspondence. Every evaluation seed uses three opponents, two seats, and ten episodes per opponent–seat cell. All four arms share the indicated schedule within each training-seed block.

Agent	Random	Heur. v1	Heur. v2	Overall	Seat 0	Seat 1	Fallback
Base	$90.00\%$	$50.00\%$	$0.00\%$	$46.67\%$	$62.22\%$	$31.11\%$	$0.00\%$
Reward	$95.00\%$	$50.00\%$	$0.00\%$	$48.33\%$	$63.33\%$	$33.33\%$	$0.00\%$
Policy	$86.67\%$	$50.00\%$	$50.00\%$	$62.22\%$	$95.56\%$	$28.89\%$	$1.84\%$
Hybrid	$86.67\%$	$50.00\%$	$33.33\%$	$56.67\%$	$84.44\%$	$28.89\%$	$4.31\%$

Appendix

Table 14: Second 720-game run using low-token DeepSeek V2. Reward quality fails the later tactical audit, so these outcomes are historical negative evidence.

Condition	Wins/games	Win rate
Outcome only	$48/150$	$32.00\%$
Deterministic oracle dense	$56/150$	$37.33\%$
Sign-flipped deterministic oracle	$58/150$	$38.67\%$

Appendix

Table 15: Short deterministic reward controls. These are mechanism checks, not GLM v0.4 ablations.

Revised category	Entries	Direction agreement	Positive error/loss
Required block	255	$232$ ( $90.98\%$ )	–
Threat creation	757	$618$ ( $81.64\%$ )	–
Avoidable tactical error	3,444	$2{,}863$ ( $83.13\%$ )	$556$ ( $16.14\%$ )
Forced next-reply loss	471	$416$ ( $88.32\%$ )	$52$ ( $11.04\%$ )
Neutral	5,027	$3{,}813$ ( $75.85\%$ )	–
All	9,954	$7{,}942$ ( $79.79\%$ )	–

Appendix

Table 16: Revalidated deterministic audit of all 9,954 frozen GLM v0.4 scores. Forced loss denotes unavoidable loss on the opponent’s next reply; its negative direction evaluates the state, not blame for the candidate. Neutral agreement uses

|f|\leq 0.2

. These are finite-cache counts.

Figure 6: Score-sign composition under the revised reference. Each horizontal bar partitions one category into negative, zero, and positive frozen GLM scores. The taxonomy separates 471 forced next-reply losses from 3,444 avoidable tactical errors; no GLM score is recomputed. Exact direction agreement is reported in Table 16 ; neutral agreement uses $|f|\leq 0.2$ , not the zero-score share.

Training seed	Qwen train	Qwen eval	Reward–Base	Hybrid–Policy	Interaction
20260800	A/A	A/B	$+1.67$	$-16.67$	$-18.33$
20260801	B/B	B/B	$+1.67$	$0.00$	$-1.67$
20260802	B/B	B/B	$+1.67$	$0.00$	$-1.67$

Appendix

Table 17: Runtime and seed sensitivity of the recorded contrasts. Qwen columns give Policy/Hybrid labels; effects are percentage points on 60 games per arm and seed. A is fe1c288 ; B is f458d5d . Labels are provenance records, not a measured runtime treatment.

Training seed	Base	Reward	Policy	Hybrid
20260800	A / A	A / A	A / A	A / B
20260801	B / B	B / B	B / B	B / B
20260802	B / B	B / B	B / B	B / B

Appendix

Table 18: Recorded training/evaluation environment labels. Each cell is training / evaluation. A and B are defined below. The highlighted cell identifies the within-pair evaluation discrepancy; labels alone do not establish an executed code difference.

Batch	Contrast	Positive	Negative	Ties	Minimum $p$	Observed $p$
Initial (3)	Reward–Base	3	0	0	$.25$	$.25$
Initial (3)	Hybrid–Policy	0	1	2	$1$	$1$
Initial (3)	Interaction	0	3	0	$.25$	$.25$
Added (7)	Reward–Base	2	3	2	$.0625$	$1$
Added (7)	Hybrid–Policy	2	2	3	$.125$	$1$
Added (7)	Interaction	4	2	1	$.03125$	$.6875$

Appendix

Table 19: Exact training-seed sign-test resolution by batch. Ties are omitted from

k

; signs are computed from integer win counts, not rounded effects.

Contrast	Effect	Nominal 95% interval	Discordant	Nominal exact $p$	Evidence boundary
Reward–Base	$+1.67$ pp	$[-1.67,+5.00]$	$6$ vs. $3$	$.508$	Holm $p=.508$ ; seed-sign $p=.25$
Hybrid–Policy	$-5.56$ pp	$[-8.89,-2.78]$	$0$ vs. $10$	$.00195$	Holm $p=.00391$ ; seed-sign $p=1$ ; one paired path in one seed
Interaction	$-7.22$ pp	$[-11.67,-2.78]$	–	–	Seed-sign $p=.25$ ; not a game-level binary contrast

Appendix

Table 20: Nominal paired calculations for the formal 720-game evaluation. The previously computed intervals resample matched game records within seed blocks; repeated paths limit their inferential interpretation even with checkpoints fixed. Holm adjustment covers the two aggregate within-backbone McNemar tests (Reward–Base and Hybrid–Policy), with no other tests included.

Figure 7: Localization of the initial three-seed contrasts. Panel A gives recorded aggregate differences with nominal 95% game-resampling intervals; panels B and C give seed differences and win counts, and Panel D gives leave-one-seed-out estimates. Only the aggregate intervals have a documented resampling interpretation; the local per-seed bars are not used for inference. All ten Qwen win discordances repeat one paired path in seed 20260800. The aggregate intervals are not training-seed confidence intervals.

Arm	Exact support	Old consistent support	Old inconsistent exposure	Old hit agreement
Reward	$1040/2602$ ( $39.97\%$ )	$842/2602$ ( $32.36\%$ )	$198/2602$ ( $7.61\%$ )	$842/1040$ ( $80.96\%$ )
Hybrid	$1000/2821$ ( $35.45\%$ )	$802/2821$ ( $28.43\%$ )	$198/2821$ ( $7.02\%$ )	$802/1000$ ( $80.20\%$ )

Appendix

Table 21: Archived evaluation support under the original tactical reference. Exact support is label-independent; the remaining columns retain the old reference and are not revised-reference estimates. All columns describe evaluation visits rather than training exposure.

Agent	Random	Heur. v1	Heur. v2	Overall	Seat 0	Seat 1	Fallback
Base	$88.33\%$	$50.00\%$	$16.67\%$	$51.67\%$	$72.22\%$	$31.11\%$	$0.00\%$
Reward	$93.33\%$	$50.00\%$	$16.67\%$	$53.33\%$	$75.56\%$	$31.11\%$	$0.00\%$
Policy	$91.67\%$	$33.33\%$	$0.00\%$	$41.67\%$	$52.22\%$	$31.11\%$	$1.35\%$
Hybrid	$91.67\%$	$16.67\%$	$0.00\%$	$36.11\%$	$41.11\%$	$31.11\%$	$8.20\%$

Appendix

Table 22: GLM v0.4 Connect Four evaluation. Each arm has 60 games per opponent and 180 overall. Fallback is pooled over agent actions; all final actions are legal.

Figure 8: Seed-structured descriptive summary of the formal 720-game evaluation. Reward adds one win over Base in each of the three paired training-seed blocks. The aggregate Policy–Hybrid difference is confined to seed 20260800, whose Qwen pair records different evaluation environment labels. The other two blocks have matched labels. The recorded evaluation-label mismatch is marked in the figure; executed source equivalence remains unverified. Nominal matched-game calculations are reported in Appendix H ; method-level generalization remains bounded by the three training seeds and runtime mismatch. Exact aggregate rates are reported in Table 22 , and the per-seed counts are reproduced in Appendix I .

Figure 9: Seed-level action-reliability diagnostic for the two Qwen-policy arms. Hybrid has higher fallback in seeds 20260800 and 20260801, while both arms are at $20/498=4.02\%$ in seed 20260802. Aggregated across seeds, fallback is $20/1{,}486=1.35\%$ for Policy and $122/1{,}488=8.20\%$ for Hybrid, a $6.85$ -point difference. The first two Policy denominators are 503 and 485, both with zero fallback. The detailed panel includes the archived episode metrics and auxiliary execution diagnostics. Invalid-generation attempts are 60 and 366, respectively, while all executed actions are legal after fallback handling. This diagnostic is descriptive and does not identify fallback as the causal source of outcome differences.

Stage	Pairs	Correct	Failures	Decision
Dev v0.1	73	36 ( $49.3\%$ )	$0/146$	Use single registered revision
Dev v0.2	73	37 ( $50.7\%$ )	$0/146$	Freeze prompt and open confirmation
Confirmation	24	9 ( $37.5\%$ )	$1/48$ ( $2.08\%$ )	Gate L1 FAIL; STOP

Appendix

Table 23: Leduc reward calibration and the preregistered fail-closed decision. Pairwise accuracy always uses all registered pairs as its denominator.

Stage	Pairs	Correct	Failures	Leakage	Decision
Reward v0.1: separate action-scoring calls
Dev v0.1	91	32 ( $35.2\%$ )	$7/182$ ( $3.85\%$ )	0	Use single registered revision
Dev v0.2	91	21 ( $23.1\%$ )	$0/182$	0	Freeze prompt and open confirmation
Confirmation	29	7 ( $24.1\%$ )	$0/58$	0	Gate L1 FAIL; STOP
Reward v0.2: one pairwise-ranking call per state
Dev v0.1	91	52 ( $57.1\%$ )	$12/182$ ( $6.59\%$ )	0	Use single registered revision

Appendix

Table 24: Goofspiel reward calibration across two reward versions. Pairwise accuracy uses every registered pair; failures count action records. The versions use disjoint states and different scoring protocols.

Training seed	Policy recorded	Policy replay	Hybrid recorded	Hybrid replay
20260800	28	28	18	28
20260801	19	19	19	29
20260802	28	28	28	28
Total	75	75	65	85

Appendix

Table 25: Recorded outcomes and post-hoc terminal replay, each out of 60 games. The same fallback-only immediate-win rule is applied to both arms. Replay entries are certified counterfactual totals, not additional empirical policy evaluations.

Batch	Training seed	Reward–Base	Hybrid–Policy
Initial	20260800	$+1.6667$	$-16.6667$
	20260801	$+1.6667$	$0$
	20260802	$+1.6667$	$0$
Initial mean		$+1.6667$	$-5.5556$
Added	20260803	$+1.1111$	$+15.5556$
	20260804	$-1.1111$	$0$

Appendix

Table 26: Exact descriptive historical contrasts, in percentage points. Initial seeds use 60 games per arm; added seeds use 180. Means weight training seeds equally within each batch. This table gives the numeric values underlying Figure 3 .

Training seed	Base	Reward	Policy	Hybrid	$R-B$	$H-P$	$I$
20260803	86	88	87	115	$+1.11$	$+15.56$	$+14.44$
20260804	88	86	85	85	$-1.11$	$0.00$	$+1.11$
20260805	84	87	84	114	$+1.67$	$+16.67$	$+15.00$
20260806	85	85	115	85	$0.00$	$-16.67$	$-16.67$
20260807	88	86	115	84	$-1.11$	$-17.22$	$-16.11$
20260808	86	84	115	115	$-1.11$	$0.00$	$+1.11$

Appendix

Table 27: Added seven-seed batch. Counts are wins out of 180 games per arm and seed (no draws). Effects are percentage points;

I=(H-P)-(R-B)

. The final row uses 1,260 games per arm. These are descriptive seed contrasts, not an equivalence test.

Update	Outcome mean	Advantage mean	NLL mean	Loss	Loss/NLL
1	$+1$	$+0.300268$	$1.140075$	$1.425093$	$+1.25$
38	$-1$	$-1.699732$	$1.388736\times 10^{-5}$	$3.471841\times 10^{-6}$	$+0.25$

Appendix

Table 28: Numerical Qwen loss audit: seed 20260803 Policy. Under the historical signed

\mathrm{loss}=\operatorname{mean}(\widehat{A}\,\mathrm{NLL})

, a batch with constant outcome and a common centered advantage would have loss/NLL equal to that advantage. The logged ratio instead remains positive.

Slice	Policy wins/games	Hybrid wins/games
random_v1	24/30	19/30
heuristic_v1	15/30	14/30
Seat 0	27/30	24/30
Seat 1	12/30	9/30

Appendix

Table 29: Formal on-policy evaluation by opponent and seat. Every row is a slice of the same 120-game evaluation, not another experiment. There are no draws.

Object	Initial summary	Interpretation after the check
Reward semantics	9,954 structurally valid records	Reference revalidation separates avoidable error from forced loss; positive scores occur in both classes
Replication	Near-zero added-batch mean	Opposing seed-wise contrasts are hidden by the mean
Evaluation	Different historical runtime labels	Fixed checkpoints have directly measured outcomes under a common contract
Optimization	Declared signed teacher-action update	Positive logged loss/NLL ratios contradict that description; a separate normalized on-policy experiment is trained and evaluated
Reward influence	Dense shaping is present	Sign reversal strongly changes learning; original-sign shaping does not exceed terminal-only training
Execution	Final actions are legal	A repeated fallback misses an immediate win; terminal replay verifies a sufficient correction

Appendix

Table 30: How an additional diagnostic check changes interpretation in the inspected cases. These are case findings, not detector-accuracy trials.

Figure 10: Comparison-to-diagnosis workflow: seed localization, trajectory replay, reward-signal audit, and deployment checks. The leftmost $46.7\%$ versus $30.0\%$ comparison is Policy versus Hybrid for initial seed 20260800 ( $28/60$ versus $18/60$ ), not the whole batch. Small plots illustrate the checks and add no measured results. Table 1 gives the operational decision logic.

Figure 11: Distribution companion under the five-category revised reference. Violin, box, and scatter layers visualize frozen scores; exact category counts and agreement rates are given in Table 16 . Positive scores occur in $556/3{,}444$ avoidable-error records and $52/471$ forced-loss records. The latter are reference-direction disagreements in unavoidable-loss states, not evidence that a safe action was available. Plot smoothing and scatter placement are visualization choices, not additional observations.

Figure 12: Reference-revalidation accounting. Panel A shows legacy-to-revised category transitions; Panel B reports 525 final category changes, 333 reference-score changes, 471 forced-loss entries, and 2,012 direction disagreements. Stage A changes 203 labels before the final two-stage taxonomy reaches 525 changes. Final category totals are $255$ , $757$ , $3{,}444$ , $471$ , and $5{,}027$ ; $3{,}390$ is a transition count, not the final avoidable-error total. Frozen GLM scores are unchanged; Appendix R gives the accounting.

Figure 13: Companion effect map for the initial three-seed batch. Labelled contrasts and win counts describe the initial results. Diamonds are initial-batch aggregate estimates with nominal 95% game-resampling intervals; per-seed and leave-one-seed-out points are shown without local error bars. Leave-one-seed-out Hybrid–Policy estimates are $0$ , $-8.33$ , and $-8.33$ percentage points. Evaluation-label differences alone do not establish a runtime effect, and ten Qwen discordances repeat one paired trajectory. This view adds no independent experiment.

Trial	Player / action	Opponent wins before	Wins after	Old reference	GLM
03	$0/2$	$\{2,6\}$	$\{2,6\}$	Block $+0.3$	$-0.9$
04	$1/2$	$\{0,2\}$	$\{0\}$	Block $+0.3$	$-0.9$
07	$0/3$	$\{2,4\}$	$\{2,4\}$	Error $-0.9$	$+1.0$

Appendix

Table 31: Three distinct trial states verified by enumerating every legal actor action. Columns are zero-indexed. The actor has no immediate win and no safe action in any row. Cases 03 and 04 correct the reference’s reward direction; case 07 changes its category while retaining a negative direction.

Reference	Agreement	Disagreement	Threshold pass	Forced-loss class
Original	$7{,}867$	$2{,}087$	$8{,}498$	–
Stage A	$7{,}942$	$2{,}012$	$8{,}567$	–
Stage B	$7{,}942$	$2{,}012$	$8{,}567$	$471$

Appendix

Table 32: Exact reanalysis of fixed scores. Stage A corrects successor checks; Stage B additionally separates next-reply forced loss. Agreement counts use the same score-direction convention throughout.

Formal case	Pool key suffix	Player	Candidate	Own winning columns	GLM
006	9451	1	1	$\{4\}$	$+0.65$
015	1539	0	5	$\{2\}$	$+0.65$
030	8200	1	3	$\{2\}$	$+0.70$
032	3353	1	2	$\{4\}$	$+0.60$
060	2502	1	1	$\{3,4\}$	$+0.45$

Appendix

Table 33: Five protocol conflicts verified directly from the boards. All candidates completely block the opponent’s immediate threat, but fail to take an available immediate win. Columns are zero-indexed. Every submitted label is must_block ; both legacy and revised Oracle labels are obvious_error .

Category	Submitted	Legacy Oracle	Revised reference	Final review
Immediate win	0	0	0	0
Required block	15	20	10	10
Obvious error	17	22	22	22
Forced loss	10	0	10	10
Create threat	8	8	8	8
Neutral	10	10	10	10

Appendix

Table 34: Category accounting for the same 60 selected records. Submitted labels use the issued annotation protocol; reference columns are computed. Final labels come from completed AI-assisted review after the comparison was disclosed.

Training seed	Outcome only	Positive oracle	Negative oracle
20260800	$38/60$	$30/60$	$1/60$
20260801	$27/60$	$27/60$	$1/60$
20260802	$38/60$	$27/60$	$3/60$
Total wins	$103/180$	$84/180$	$5/180$
Win rate	$57.22\%$	$46.67\%$	$2.78\%$

Appendix

Table 35: Verified final-step historical-reward direction control, step 100. Each seed–condition cell has 60 evaluation games. This batch is separate from both the formal GLM four-arm runs and the historical five-update control.

Condition	Nonzero dense samples	Optimization samples	Fraction
Outcome only	0	406,029	$0.00\%$
Positive oracle	121,637	395,884	$30.73\%$
Negative oracle	300,588	680,933	$44.14\%$

Appendix

Table 36: Verified nonzero dense signals among samples used for optimization. Denominators differ across conditions and are not independent experimental replications.

Train seed	Eval seed	Base W/D/L	Policy W/D/L	Base sum	Policy sum	$\Delta$
20260800	30260800	43/6/11	45/8/7	80	84	$+0.0667$
20260801	30261800	37/6/17	19/17/24	18	$-11$	$-0.4833$
20260802	30262800	42/4/14	44/5/11	42	51	$+0.1500$
Total	—	122/16/42	108/30/42	140	124	$-0.0889$

Appendix

Table 37: Final fresh-schedule Leduc comparison. Each cell has 60 games, 30 per seat; W/D/L denotes positive/zero/negative terminal return. The last column is the Policy-minus-Base difference in mean chips/game.

Train seed	Agent decisions	Raise opportunities	Raises	Checks/calls
20260800	99	68	68	31
20260801	155	89	0	155
20260802	98	68	68	30

Appendix

Table 38: Policy action patterns on evaluated states. An opportunity is a visited agent decision at which bet_raise is legal. These counts describe endogenous visited-state distributions, not additional replicates.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,651 papers

Oct 1, 2026

Probe with Participation Trophies: Random-Reward RL as a Probe of LLM Capability

We connect the spurious-reward paradox to a model's reachability and propose random-reward reinforcement learning (RL) as a useful tool for the probing enterprise, addressing a decade-long debate over what probing performance actually reveals about a model. There are two prevailing explanations for the surprising finding that even random rewards can improve the performance of large language models (LLMs): one attributes the gains to particular mechanisms within RL training; the other to data contamination. Our results motivate a different view: spurious-reward RL can probe a model's reachability, or what further training can attain from its current state under specified constraints, beyond what is reflected in its current performance. Two OLMo checkpoints with the same accuracy on synthetic arithmetic (3.5%), for example, reach 8.5% and 55% in their best runs under the same correctness-rewarded RL. Examining OLMo checkpoints across pre-training and mid-training reveals three distinct regimes of training response: early on, RL produces little improvement even when correct answers are rewarded; later in pre-training, rewarding correct answers becomes effective while random rewards remain weak; and, upon entering mid-training, even random rewards can produce large gains. A similar ordering appears in a number-masked supervised fine-tuning (SFT) analysis of these checkpoints, suggesting that the pattern is not specific to a particular RL mechanism. Moreover, RL with random rewards offers a distinctive perspective on what training can make an LLM do, since its reward signal supplies no information about which answers are correct. By asking what training can attain without correctness feedback, it addresses the label-leakage side of a central problem in decodability-based probing: whether a successful probe reveals the model's capabilities or learns the task itself.

Yu Mao, Lei Yu, Zining Zhu +6

University of Toronto · Stevens Institute of Technology · University of California, Santa Cruz +3

RL for Language Models Reinforcement Learning Language Model Probing

Ask PDF

Oct 8, 2026

Do LLMs Learn from Rewards in Context? : Rethinking the role of reward in In-Context RL

LLM agents increasingly improve at inference time by accumulating experience in context rather than by updating parameters. This process is often described as in-context reinforcement learning (ICRL). Whether in-context learning (ICL) can actually play the role of RL, however, has not been tested. We study this question in its simplest form, direct ICRL, where the model conditions directly on raw trajectory-reward pairs, and ask whether the reward acts as a learning signal. Through controlled experiments on four benchmarks across six models, we find that the reward is read, but its effect is small: flipping, randomizing, or removing the reward leaves the improvement curve almost unchanged, and this holds even under meta-prompts that explicitly instruct the model to explore, exploit, or reason over rewards. Trajectories drive improvement, but not through their semantic content: shuffled or corrupted trajectories work as well as real ones. These patterns closely mirror those known in ICL, suggesting that direct ICRL is better understood as a special case of ICL than as inference-time RL. This reframing has implications for agent memory design: ICL factors such as input distribution and demonstrations may matter more than RL elements such as reward shaping and exploration.

Minchan Kwon, Seunghee Koh, Sunghyun Baek +2

Korea Advanced Institute of Science and Technology (KAIST)

In-Context RL Contextual RL RL for LLM Agents In-Context Learning

Ask PDF

Aug 5, 2026

Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learning

Critic-free group-based reinforcement learning has become a scalable approach for post-training large language models. However, most existing methods allocate the same number of rollouts to every task and trajectory state, even though some rollouts provide much more useful learning signals than others. Recent work has started to treat rollout generation as an adaptive decision, but two important limitations remain. First, intervention strategies are often based on fixed heuristics and therefore cannot adjust as the policy changes during training. Second, these methods usually decide only how many rollouts to generate, without explicitly controlling where and how to intervene. To address these limitations, we propose Recoverability-Aware Intervention Learning (RAIL), a training-time framework that learns how to generate rollouts based on the improvement produced by each intervention. RAIL models intervention selection as an online contextual-bandit problem and trains a recoverability controller using intervention traces collected through a shadow-to-live procedure. This allows the controller to keep learning while the underlying policy evolves. We evaluate RAIL in terms of effectiveness, adaptivity, expressiveness, and efficiency. Across multiple settings, RAIL consistently improves performance under limited rollout budgets. These results show that recoverability-aware intervention provides a principled way to generate more informative and less redundant rollouts, leading to stronger learning signals during post-training.

Zheyuan Zhang, Manqing Mao, Hong Wang +8

University of Notre Dame · Amazon, Inc

RL for Language Models RL Post-Training Policy Learning

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

Probe with Participation Trophies: Random-Reward RL as a Probe of LLM Capability

Do LLMs Learn from Rewards in Context? : Rethinking the role of reward in In-Context RL

Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learning