Reinforcement learning for long-horizon agents relies on purely retrospective training signals: credit is assigned only after observing environmental consequences, leaving the agent's belief at action time invisible to the gradient. We introduce Prospective Hindsight (PH), a self-calibrating training principle that augments any retrospective base method with a signal derived from the gap between the agent's prospective prediction (before feedback) and the retrospective evaluation (after feedback). This per-rollout surprise identifies samples where the agent's self-model is most inaccurate and amplifies their gradient contribution through a stop-gradient surprise-weighted advantage. Since the prospective predictor shares parameters with the policy, the two co-evolve, progressively shifting focus to the agent's remaining blind spots. We connect this principle to a privileged-information gap and show that minimizing the surprise residual provides a descent pathway on the agent's miscalibration rate; calibration thus emerges as a byproduct of optimization rather than from an added objective. On single-turn verifiable tasks and a multi-turn personal-agent task (under GRPO, on-policy distillation, and their combination), PH improves both task performance and calibration, with consistent gains across model scales. Notably, the dominant miscalibration mode shifts structurally between regimes, overconfident failures in single-turn, underconfident successes in multi-turn, yet the same training principle addresses both successfully.
Figures & tables
Figure 1: Prospective Hindsight (PH) converts hindsight feedback into a self-calibrating training signal. PH compares an agent’s action-time prediction with the realized verifier/PRM outcome, partitions rollouts into calibrated and miscalibrated cases, and upweights surprising prediction–reality mismatches during policy updating within GRPO, on-policy distillation and their combination.
Cell
zt
yt
Epistemic interpretation
CS (confident success)
+1
1
calibrated competence
OF (overconfident failure)
+1
0
miscalibrated; agent trusts a wrong action
US (underconfident success)
−1
1
miscalibrated; agent doubts a correct action
AF (aware failure)
−1
0
calibrated incompetence
Table 1: Calibration cells of rollouts, indexed by the agent’s prospective belief zt∈{−1,+1} and the realized outcome yt∈{0,1} . Rollouts with zt=0 are deferred to App. C.4 .
Dataset
Method
Val. Acc.
OFR
Surprise
Avg. ↑
Max ↑
Early
Late ↓
Δ↓
Early
Late ↓
Δ↓
Science Q&A
SDPO
63.0
73.4
75.3
34.2
-41.1
70.9
35.2
-35.7
+Random
65.4
74.3
74.1
33.3
-41.7
69.0
35.3
-33.7
+Failure-only
65.1
74.7
73.9
25.0
-48.9
69.0
24.4
-44.6
+PH ( α=0.5 )
65.7
75.5
76.8
20.9
-56.0
71.7
21.8
-49.9
+PH ( α=1.0 )
64.0
74.2
73.2
23.9
-49.3
68.6
21.1
-47.5
Table 2: Single-turn results. Validation accuracy is the average / maximum logged mean@16 (%); OFR =OF/(CS+OF) (failure rate among predicted successes) and Surprise =OF+US (total prediction–reality mismatch percentage); Early / Late are the first / last 5 training steps and Δ=Late−Early .
Figure 2: Self-evaluation dynamics under PH on Science Q&A ( α=1 ). Left: per-step composition of the four PH cells (CS / OF / US / AF); overconfident failures shrink while confident successes grow. Right: the mean PH weight decays as the verifier pass rate rises, indicating that PH acts as an adaptive training signal that anneals as the prediction–reality gap closes.
Capability
Calibration
Method
Setting
Avg. ↑
Best ↑
Δbest↑
OFR ↓
Surprise ↓
Prosp. Acc. ↑
RL
GRPO
68.6
77.7
+8.5
5.3
16.5
83.5
+PH ( α=0.5 )
72.6
77.7
+8.7
4.9
15.0
85.0
+PH ( α=1.0 )
75.8
82.9
+14.8
3.1
12.2
87.8
+PH ( α=2.0 )
68.6
77.3
+9.4
2.5
16.0
84.0
OPD
SDPO
54.0
65.6
+2.5
15.9
37.2
62.8
Table 3: Multi-turn personal-agent results (GPT-4.1 simulator). PH plugged into RL, OPD, and the Combined recipe; performance is the teacher-side personalization-evaluator score, calibration is computed over teacher-side rollouts within mixed training, Δbest=Best−initial .
Figure 3: Capability and calibration dynamics on Science Q&A across α . Left: OFR vs. Validation Accuracy; Middle: prospective accuracy over training. Right: Effect of α on overconfidence rate.
Figure 4: Capability and calibration gains from PH transfer across model scale using SDPO baseline and SDPO+PH ( α=1.0 ) on OLMo-3-7B-Instruct and gpt-oss-20B.
Figure 5: PH cell composition differs structurally between single-turn and multi-turn settings.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Models, GPU layout
Backbone (student / PRM / teacher)
OLMo-3-7B-Instruct (shared)
Total GPUs
8 (1 node)
Actor
4 GPUs, TP=4, sequence-parallel
Rollout (sglang)
2 GPUs, TP=1 each
PRM judge (sglang)
1 GPU
Appendix
Table 4: Hyperparameters for multi-turn personal agent: Combined (Binary RL + OPD) and Combined + PH on OpenClaw-RL.
Parameter
Value
Models, data, and inference
Backbone models
OLMo-3-7B-Instruct; gpt-oss-20B (ablation)
Thinking mode
disabled ( enable_thinking=false )
Max prompt length
2048 tokens
Max response length
8192 tokens
Inference engine
vLLM
Appendix
Table 5: Hyperparameters for single-turn SDPO and SDPO + PH on Science Q&A and Tool Use.
α
Early OFR
Late OFR
OFR Drop
Rel. Drop
Mean Weight
Verifier Pass Rate
0.1
0.741
0.253
0.487
65.8%
1.07→1.02
24.8%→71.4%
0.25
0.739
0.250
0.489
66.2%
1.17→1.06
25.6%→71.2%
0.5
0.768
0.209
0.560
72.9%
1.35→1.09
22.9%→73.2%
1.0
0.732
0.239
0.423
57.8%
1.67→1.29
26.0%→67.5%
2.0
0.744
0.249
0.495
66.5%
1.68→1.22
24.9%→71.7%
Appendix
Table 6: PH weight strength on Science Q&A: numerical summary across the full α -sweep. OFR =OF/(CS+OF) ; Early/Late are first/last 5 training steps; Mean Weight and Verifier Pass Rate are reported as early → late. Bold : best per column.
Figure 6: PH cell composition across α on Chemistry. Per-step decomposition of training rollouts into CS / OF / US / AF for α∈{0.1,0.25,0.5,2.0} (the α=1.0 panel is in Figure 2 ). The compress-OF / grow-CS pattern is qualitatively unchanged across two orders of magnitude of α .
Figure 7: OFR and PH-weight dynamics across α on Science Q&A. Left: OFR over training for each α ; all five trajectories descend monotonically. Right: the mean PH weight starts above 1 and anneals toward 1 as ξt→0 (App. C.3 ). Faint lines: 5-step bins; solid lines: EMAs.
Figure 8: Multi-turn α -sweep (Combined, GPT-4.1 simulator). Left: teacher-side evaluation score across training rounds. Middle: teacher-side surprise rate across conversation turns. Right: teacher-side OFR across conversation turns. Curves: α∈{0.1,0.5,1.0,2.0} . Later turns start more overconfident than early turns, and PH compresses this region most.
Figure 9: Capability and calibration across user simulators. OLMo-3-7B-Instruct trained on the multi-turn personal-agent task with 8k response-length budget and mixed Student/Teacher rounds, under two simulators (GPT-4.1 and GPT-5.4). Left: teacher-side personalization-evaluator score (higher is better). Right: teacher-side OFR (lower is better).
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Reinforcement learning with verifiable rewards (RLVR) turns agent experience into learning signals primarily through scalar outcome rewards after interaction. For group-relative objectives, however, this signal vanishes when all rollouts receive the same reward, even though their trajectories may reveal useful information about what the task requires and how the agent fails. We ask a complementary question: can hindsight teach an agent what it could have anticipated before acting? We introduce prospective learning, which uses post-hoc experience to supervise foresight predictions from the pre-interaction view, and instantiate it with Self-Retrospection Distillation (SRD). Intuitively, a completed trajectory reveals knowledge that would have been useful and pitfalls that should be avoided; SRD distills this privileged hindsight into trajectory-blind foresight of the same policy. Foresight serves only as a training target and need not be explicitly generated at inference time. Across 10 tool-integrated reasoning and long-horizon agentic tasks, SRD complements RLVR and self-distillation baselines with gains of up to 24.2 pp. Its advantage is especially pronounced when reward contrast is scarce: when 37--98% of rollout groups are reward-uniform across model scales, yet SRD can still exploit learning signal from sampled trajectories. In the 2B setting, where 98% of groups are all-failure, the RLVR training ends up at 0.0% success, while adding SRD reaches 60.6% under the same rollout budget. Our results suggest that post-hoc agent experience is useful not only for evaluating or improving behavior, but also for shaping predictive representations before available interaction. Training code can be found at https://github.com/SalesforceAIResearch/SRD.
Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi +9
Salesforce AI Research · UC San Diego · Texas A&M University +1
Long-horizon LLM-based agents receive rich environmental observations during interaction, yet outcome rewards provide limited explicit supervision about how individual actions advance task completion. We investigate whether agents can turn this interaction evidence into useful training signals through retrospective progress assessment. A WebShop pilot shows that direct progress prompting reduces task success, whereas hindsight-annotated demonstrations improve it. We introduce RePro, Retrospective Progress-Aware Training, with a forward-then-reflect rollout: the agent estimates progress while acting, then reassesses each step using the completed trajectory and outcome. After warmup with externally generated demonstrations, policy optimization combines self-generated progress differences, online-retrospective alignment, and format rewards with environment feedback, requiring neither a separate process reward model nor ongoing teacher annotation. Experiments on WebShop, ALFWorld, and Sokoban show that RePro enhances the Qwen family's performance, with up to 11.57% success rate gains.
Xinbei Ma, Congmin Zheng, Jiyang Qiu +11
Shanghai Jiao Tong University · OPPO Research Institute
Training long-horizon LLM agents with reinforcement learning is challenging because sparse outcome rewards reveal whether a task succeeds, but not which intermediate actions caused the outcome or how they should be corrected. Recent methods alleviate this issue by generating rewards or textual hints from turn-level action-output signals, or by using feedback-conditioned self-distillation. However, generating feedback at every turn is inefficient when many intermediate turns are already successful or neutral, and applying feedback at a fixed or misaligned turn often fails to supervise the actions that contributed to the failure. To bridge this gap, we propose HINT-SD, a targeted self-distillation framework that uses full-trajectory hindsight to select failure-relevant actions and applies feedback-conditioned distillation only on targeted action spans. Experiments on BFCL v3 and AppWorld show that our method improves over the dense per-turn feedback baseline by up to 18.80 percent while achieving 2.26× lower time per training step, suggesting that selecting where to distill is a key factor for both effective and efficient long-horizon agent training.