Long-horizon interaction requires agents to make multiple decisions in a changing environment. Outcome-based GRPO lacks fine-grained guidance for intermediate decisions. Recent methods distill completed trajectories into reusable skills and use hindsight-conditioned OPD to provide dense token-level credit. However, skill-induced probability shifts do not necessarily represent reward-aligned credit; especially after an erroneous action changes subsequent interaction states, skill-conditioned supervision may become mismatched with the current state. To address this mismatch, we propose \methodname{}, which routes two complementary hindsight skills according to trajectory outcomes and error boundaries: Positive skills capture successful workflows to reinforce successful trajectories and pre-error decisions, while Negative skills capture failure-specific corrections to target decisive errors. The post-error suffix is optimized exclusively by GRPO, retaining outcome-based learning without extending potentially mismatched hindsight supervision. Extensive experiments demonstrate competitive task performance and support effective fine-grained credit allocation.
Figures & tables
Figure 1: OPD signals versus GiGPO-based step advantages on Qwen2.5-1.5B-Instruct failed trajectories. Each point represents one interaction turn.
Figure 2: Overview of ETRO’s trajectory-aware routing of Positive and Negative hindsight credit.
Method
ALFWorld
Search QA
WebShop
Succ.
NQ
Triv
Pop
Hotp
2Wk
MuS
Bam
Avg.
Score
Succ.
Qwen2.5-1.5B-Instruct
GRPO
65.7
16.6
34.1
15.9
16.1
24.8
3.4
58.1
24.1
72.0
62.0
GRPO+OPSD
71.1
16.9
34.3
16.4
16.3
24.8
3.4
56.9
24.1
84.9
73.4
Skill-SD
68.0
16.7
35.4
14.8
16.7
25.2
3.9
11.2
17.7
80.6
68.8
RLSD
67.2
13.9
29.4
12.4
14.2
21.1
3.0
9.5
14.8
80.5
63.3
Table 1: Results on ALFWorld, Search QA, and WebShop across Qwen2.5-Instruct model scales (%). Higher values indicate better performance.
Table 4
Figure 3: Qwen2.5-1.5B-Instruct training dynamics on WebShop for pure GRPO, full-trajectory Positive and Negative OPD, and ETRO. Training reward is reported in [0,1] . Validation uses a 15-turn interaction limit.
Method
α
Success (%)
Score
ETRO
0
80.8
87.9
Suffix Positive
0.25
79.0
85.1
Suffix Positive
0.5
78.4
85.2
Suffix Positive
1
77.6
84.6
Suffix Negative
0.25
76.6
82.7
Suffix Negative
0.5
74.4
80.6
Table 4: Post-error OPD weight sweep on WebShop using Qwen2.5-1.5B-Instruct, with λ=0.001 .
Metric
Rate (%)
Boundary acceptance
92.0
Exact-turn agreement
84.0
Workflow validity
92.0
Correction validity
90.0
Table 5: Writer quality on WebShop, evaluated against human references by GPT-5.6 sol (%).
Figure 4: Comparison of Negative-skill conditioning regions on WebShop using Qwen2.5-1.5B-Instruct.