Outcome-predictive signals are often treated as promising optimization targets in reinforcement learning, but predictive validity and optimization utility are different claims. We expose this distinction in GuanDan, a multi-round imperfect-information game in which the implication of a round outcome depends on the public match state. Held-out analysis shows that state--gain interactions provide incremental predictive value beyond a constant signed-gain effect, establishing that match progress is predictively meaningful. We then use changes in a frozen match-win predictor, ΔΦ, as a controlled round-level supervision intervention in a matched DanZero-style Monte Carlo self-play protocol. The intervention can succeed in an individual lineage: a selected progress-augmented checkpoint substantially outperforms its matched Baseline on a locked evaluation. Yet the method-level result reverses this impression: none of five preregistered replication seeds is positive, the pooled ten-seed interval includes zero, and across-iteration effects repeatedly change sign without a common late-stage advantage. These results identify a predictive--optimization gap: under the studied learner, a signal can describe outcome-relevant progress while failing to provide a reliably beneficial optimization direction.
Figures & tables
Fig. 1: One online collect–annotate–train iteration. Each iteration first generates new complete matches with the current policy πi . After a match ends, the frozen predictor Φ annotates each round boundary with ΔΦ=Φ(xafter)−Φ(xbefore) , and the resulting round-level supervision target is broadcast to every executed action within that round. These round-level labels then supervise one epoch of policy training with a fresh optimizer state, yielding πi+1 for the next collection round. The predictor Φ remains frozen across all policy-training iterations and is not updated with the policy.
Split
Matches
Boundaries
Train
1,411
34,818
Val. (Platt)
199
4,886
Test
390
9,568
Total
2,000
49,272
TABLE I: Complete-match dataset partitions. Boundary rows include both team-relative views.
Model
Log loss
Brier
AUROC
M0
0.54458
0.18431
0.79633
M1
0.53983
0.18370
0.79572
TABLE II: Held-out incremental predictive-value test. M0 uses a constant gain effect and M1 adds state–gain interactions. Lower is better for log loss and Brier score; higher is better for AUROC.
Fig. 2: Mean ΔΦ vs. signed rank gain by gap stage (top) and near-A contrast (bottom; overlapping slice, not a fourth exclusive stage). Error bars: match-clustered 95% bootstrap CIs. Stage dependence is strongest at larger ∣ gain ∣ .
Metric
Test value
Log loss
0.59497
Brier score
0.20462
AUROC
0.75064
ECE
0.05064
Clipping rate
0.000209
Fallback rate
0
TABLE III: Frozen Φ performance on the untouched test split.
Fig. 3: Reliability diagram for frozen Φ on the test split ( n=4,784 ); 95% Wilson intervals per bin, with bin counts below. Predictions track the diagonal except in sparse upper bins (AUROC =0.751 , ECE =0.051 ). Platt scaling was fit on validation only.
Arm
β
ca
Pos.
Agg. Δ
Final
Baseline
0
2.556
—
0
no
P10
2.002
2.747
3/5
+0.034
yes
P20
4.005
2.948
2/5
+0.004
no
TABLE IV: Five-seed progress-scale selection before the final holdout. “Agg. Δ ” is the Progress-minus-Baseline MeanDelta on the confirmation protocol and is therefore 0 for Baseline by definition. Parent-relative aggregates are reported in the text ( 0.120 for Baseline; 0.154 for Progress-10). Final marks admission to the locked holdout.
Cohort
Mean Ds
95% CI
Positive
Seeds 1–5
+0.034
[−0.080,0.147]
3/5
Seeds 6–10
−0.056
[−0.083,−0.030]
0/5
Seeds 1–10
−0.030
[−0.084,0.025]
3/10
TABLE V: Method-level Progress-10 effect across training-seed cohorts. The original interval is a seed bootstrap; the preregistered replication and pooled intervals use Student- t inference over training seeds. Positive counts are descriptive.
Seed
Original
Parent-rel.
Direct
Cross-fit
1
+0.145
+0.128
+0.128
+0.14
2
+0.135
+0.182
+0.074
+0.02
3
−0.125
−0.102
−0.160
−0.13
4
+0.155
−0.044
−0.004
−0.05
5
−0.140
−0.038
−0.068
−0.05
TABLE VI: Seed-level Progress-10 effects under the original confirmation analysis, independent 500-pair replication, and cross-fitted checkpoint-selection audit. “Parent-rel.” is the difference between Progress-versus-Parent and Baseline-versus-Parent MeanDelta; “Direct” evaluates Progress-10 against its matched Baseline. Cross-fit values are held-out estimates of the original three-candidate selection rule and do not replace the frozen scientific checkpoints.
Comparison
W/S/L
MD
95% CI
vs. Parent
369/516/115
0.254
[0.214, 0.294]
vs. Baseline
293/557/150
0.143
[0.103, 0.183]
TABLE VII: Locked paired team-swap evaluation of the selected Progress-10 checkpoint on 1,000 pairs. W/S/L are double-win, split, and double-loss counts; MD is MeanDelta.
Fig. 4: Training-horizon diagnostics across ten matched self-play seeds. Panels A and B show the paired MeanDelta of Progress-10 and Baseline, respectively, against the frozen Parent; Panel C reports direct Progress-10-versus-Baseline evaluation and is therefore the relevant treatment-effect trajectory. Each matchup uses 200 paired team-swap seeds from the diagnostic common-random-number pool. Thin gray lines denote individual training seeds, thick black lines denote across-seed means, gray regions show simultaneous 95% confidence bands over the eight inspected iterations, and the dashed line marks the fixed endpoint at iteration 7. Both policies improve in absolute strength relative to Parent, whereas their direct comparison fluctuates around zero and shows no common late-stage Progress advantage. This post-hoc diagnostic is not used for checkpoint selection.