Reinforcement learning for long-horizon agents relies on purely retrospective training signals: credit is assigned only after observing environmental consequences, leaving the agent's belief at action time invisible to the gradient. We introduce Prospective Hindsight (PH), a self-calibrating training principle that augments any retrospective base method with a signal derived from the gap between the agent's prospective prediction (before feedback) and the retrospective evaluation (after feedback). This per-rollout surprise identifies samples where the agent's self-model is most inaccurate and amplifies their gradient contribution through a stop-gradient surprise-weighted advantage. Since the prospective predictor shares parameters with the policy, the two co-evolve, progressively shifting focus to the agent's remaining blind spots. We connect this principle to a privileged-information gap and show that minimizing the surprise residual provides a descent pathway on the agent's miscalibration rate; calibration thus emerges as a byproduct of optimization rather than from an added objective. On single-turn verifiable tasks and a multi-turn personal-agent task (under GRPO, on-policy distillation, and their combination), PH improves both task performance and calibration, with consistent gains across model scales. Notably, the dominant miscalibration mode shifts structurally between regimes, overconfident failures in single-turn, underconfident successes in multi-turn, yet the same training principle addresses both successfully.
Figures & tables
Figure 1: Prospective Hindsight (PH) converts hindsight feedback into a self-calibrating training signal. PH compares an agent’s action-time prediction with the realized verifier/PRM outcome, partitions rollouts into calibrated and miscalibrated cases, and upweights surprising prediction–reality mismatches during policy updating within GRPO, on-policy distillation and their combination.
Cell
zt
yt
Epistemic interpretation
CS (confident success)
+1
1
calibrated competence
OF (overconfident failure)
+1
0
miscalibrated; agent trusts a wrong action
US (underconfident success)
−1
1
miscalibrated; agent doubts a correct action
AF (aware failure)
−1
0
calibrated incompetence
Table 1: Calibration cells of rollouts, indexed by the agent’s prospective belief zt∈{−1,+1} and the realized outcome yt∈{0,1} . Rollouts with zt=0 are deferred to App. C.4 .
Dataset
Method
Val. Acc.
OFR
Surprise
Avg. ↑
Max ↑
Early
Late ↓
Δ↓
Early
Late ↓
Δ↓
Science Q&A
SDPO
63.0
73.4
75.3
34.2
-41.1
70.9
35.2
-35.7
+Random
65.4
74.3
74.1
33.3
-41.7
69.0
35.3
-33.7
+Failure-only
65.1
74.7
73.9
25.0
-48.9
69.0
24.4
-44.6
+PH ( α=0.5 )
65.7
75.5
76.8
20.9
-56.0
71.7
21.8
-49.9
+PH ( α=1.0 )
64.0
74.2
73.2
23.9
-49.3
68.6
21.1
-47.5
Table 2: Single-turn results. Validation accuracy is the average / maximum logged mean@16 (%); OFR =OF/(CS+OF) (failure rate among predicted successes) and Surprise =OF+US (total prediction–reality mismatch percentage); Early / Late are the first / last 5 training steps and Δ=Late−Early .
Figure 2: Self-evaluation dynamics under PH on Science Q&A ( α=1 ). Left: per-step composition of the four PH cells (CS / OF / US / AF); overconfident failures shrink while confident successes grow. Right: the mean PH weight decays as the verifier pass rate rises, indicating that PH acts as an adaptive training signal that anneals as the prediction–reality gap closes.
Capability
Calibration
Method
Setting
Avg. ↑
Best ↑
Δbest↑
OFR ↓
Surprise ↓
Prosp. Acc. ↑
RL
GRPO
68.6
77.7
+8.5
5.3
16.5
83.5
+PH ( α=0.5 )
72.6
77.7
+8.7
4.9
15.0
85.0
+PH ( α=1.0 )
75.8
82.9
+14.8
3.1
12.2
87.8
+PH ( α=2.0 )
68.6
77.3
+9.4
2.5
16.0
84.0
OPD
SDPO
54.0
65.6
+2.5
15.9
37.2
62.8
Table 3: Multi-turn personal-agent results (GPT-4.1 simulator). PH plugged into RL, OPD, and the Combined recipe; performance is the teacher-side personalization-evaluator score, calibration is computed over teacher-side rollouts within mixed training, Δbest=Best−initial .
Figure 3: Capability and calibration dynamics on Science Q&A across α . Left: OFR vs. Validation Accuracy; Middle: prospective accuracy over training. Right: Effect of α on overconfidence rate.
Figure 4: Capability and calibration gains from PH transfer across model scale using SDPO baseline and SDPO+PH ( α=1.0 ) on OLMo-3-7B-Instruct and gpt-oss-20B.
Figure 5: PH cell composition differs structurally between single-turn and multi-turn settings.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Models, GPU layout
Backbone (student / PRM / teacher)
OLMo-3-7B-Instruct (shared)
Total GPUs
8 (1 node)
Actor
4 GPUs, TP=4, sequence-parallel
Rollout (sglang)
2 GPUs, TP=1 each
PRM judge (sglang)
1 GPU
Appendix
Table 4: Hyperparameters for multi-turn personal agent: Combined (Binary RL + OPD) and Combined + PH on OpenClaw-RL.
Parameter
Value
Models, data, and inference
Backbone models
OLMo-3-7B-Instruct; gpt-oss-20B (ablation)
Thinking mode
disabled ( enable_thinking=false )
Max prompt length
2048 tokens
Max response length
8192 tokens
Inference engine
vLLM
Appendix
Table 5: Hyperparameters for single-turn SDPO and SDPO + PH on Science Q&A and Tool Use.
α
Early OFR
Late OFR
OFR Drop
Rel. Drop
Mean Weight
Verifier Pass Rate
0.1
0.741
0.253
0.487
65.8%
1.07→1.02
24.8%→71.4%
0.25
0.739
0.250
0.489
66.2%
1.17→1.06
25.6%→71.2%
0.5
0.768
0.209
0.560
72.9%
1.35→1.09
22.9%→73.2%
1.0
0.732
0.239
0.423
57.8%
1.67→1.29
26.0%→67.5%
2.0
0.744
0.249
0.495
66.5%
1.68→1.22
24.9%→71.7%
Appendix
Table 6: PH weight strength on Science Q&A: numerical summary across the full α -sweep. OFR =OF/(CS+OF) ; Early/Late are first/last 5 training steps; Mean Weight and Verifier Pass Rate are reported as early → late. Bold : best per column.
Figure 6: PH cell composition across α on Chemistry. Per-step decomposition of training rollouts into CS / OF / US / AF for α∈{0.1,0.25,0.5,2.0} (the α=1.0 panel is in Figure 2 ). The compress-OF / grow-CS pattern is qualitatively unchanged across two orders of magnitude of α .
Figure 7: OFR and PH-weight dynamics across α on Science Q&A. Left: OFR over training for each α ; all five trajectories descend monotonically. Right: the mean PH weight starts above 1 and anneals toward 1 as ξt→0 (App. C.3 ). Faint lines: 5-step bins; solid lines: EMAs.
Figure 8: Multi-turn α -sweep (Combined, GPT-4.1 simulator). Left: teacher-side evaluation score across training rounds. Middle: teacher-side surprise rate across conversation turns. Right: teacher-side OFR across conversation turns. Curves: α∈{0.1,0.5,1.0,2.0} . Later turns start more overconfident than early turns, and PH compresses this region most.
Figure 9: Capability and calibration across user simulators. OLMo-3-7B-Instruct trained on the multi-turn personal-agent task with 8k response-length budget and mixed Student/Teacher rounds, under two simulators (GPT-4.1 and GPT-5.4). Left: teacher-side personalization-evaluator score (higher is better). Right: teacher-side OFR (lower is better).
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.