Learning after Execution with Action-Specific Credit

Yufeng Tian; Jiahao Tang; Minghao Wu

cs.AIOct 10, 2026

Learning after Execution with Action-Specific Credit

Authors: Yufeng Tian, Jiahao Tang, Minghao Wu

Organizations: The Chinese University of Hong Kong, Shenzhen, China · Central South University, Changsha, China

Abstract

Execution is often used as terminal feedback: it scores a generated program, but the resulting state is not used for another learning action. Learning after execution instead uses the execution result as an intermediate state that supports an additional learning interaction for the same policy. This creates a credit-assignment problem because the two actions share the same final objective but play different roles in the trajectory. Accordingly, action-specific credit assigns the initial action credit from the final trajectory outcome and the corrective action credit for improvement over the intermediate state. Image-to-TikZ serves as an executable visual testbed for this setting. On 945 held-out examples, learning after execution strengthens the shared policy under single-call inference and also produces a post-execution corrective capability. Single-call render success reaches 92.70%, compared with 88.25% for continued single-stage GRPO after the same number of additional training iterations. This corrective capability is not reproduced by continued single-stage training and depends on the intermediate state. Alternative credit definitions also induce different group-relative preferences over the same trajectories. Together, these results suggest that executable tasks contain useful learning structure beyond a terminal reward. Code is available at https://github.com/tianyufengcn/Learning-after-Execution.

Figures & tables

Figure 1: Overview of learning after execution with action-specific credit. Execution first provides a score for the initial action; learning after execution additionally uses the resulting render or failure state for a corrective action by the same policy. The initial action receives final-outcome credit, while the corrective action receives improvement-relative credit. The right panel illustrates the Image-to-TikZ instantiation with valid and failed executions.

Method	Output	Render Success $\uparrow$	RSim $\uparrow$	DINO $\uparrow$	DreamSim $\uparrow$	CLIPImg $\uparrow$
Base model	Direct	71.64%	0.4033	0.6025	0.5545	0.6435
SFT	Direct	64.66%	0.3554	0.5427	0.4960	0.5781
Direct-GRPO	Direct	81.90%	0.4709	0.7046	0.6468	0.7410
Direct-GRPO (+100)	Direct	88.25%	0.5135	0.7629	0.7019	0.7996
Ours (initializer)	Stage 0	83.07%	0.4766	0.7129	0.6544	0.7489
Ours (initializer)	Stage 1	83.17%	0.4781	0.7142	0.6556	0.7501

Table 1: Main results on Final-945. Visual metrics use the fixed-zero convention. Direct denotes single-call generation. Stage 0 is the initial single-call output of our policy, while Stage 1 is produced after execution-conditioned correction. Selected chooses between the two outputs using execution success and RSim. Direct-GRPO (+100) continues single-stage GRPO for 100 additional training iterations from the same Direct-GRPO checkpoint. Best results are shown in bold and second-best results are underlined.

Figure 2: Learning dynamics of our method on Dev-150. The figure tracks Stage 0 and Stage 1 render rates, paired visual changes $\Delta M$ , the distribution of correction outcomes, and execution transitions across training checkpoints.

	Stage 1 credit
Stage 0 credit	$\lambda(S_{1}-S_{0})$	$\lambda S_{1}$
$\lambda S_{1}$	A (Ours)	C
$\lambda S_{0}$	B	D

Table 2: Factorized ablation of action-specific credit assignment. Rows vary Stage 0 credit and columns vary Stage 1 credit.

Figure 3: Effect of action-specific credit assignment. The figure compares Stage 0 quality and correction gain under alternative credit definitions, measures how the resulting group-relative preferences differ, and tracks the training signal of all four ablation variants.

Figure 4: Correction behavior as a function of Stage 0 quality. We report the mean Stage 1 change in RSim and the frequency with which Stage 1 achieves higher RSim than Stage 0 across Stage 0 quality ranges, together with qualitative examples of beneficial and harmful corrections.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Additional qualitative examples of our method from Final-945.

Figure 6: Qualitative progression across the main learning settings on Final-945. Base model, Direct-SFT, and Direct-GRPO show single-call predictions. Ours Stage 0 is also a single-call prediction from the trained policy, while Stage 1 is produced after observing the Stage 0 execution state. The examples include visible corrective changes and cases where an already strong Stage 0 result is largely preserved.

Figure 7: Successful visual revisions from Final-945 at checkpoint 100. Stage 0 already renders successfully, and Stage 1 revises the executed state to better match the target. The examples illustrate both large structural corrections and more localized changes to geometry, visual elements, or layout.

Figure 8: Successful execution repair from Final-945. Stage 0 does not produce a valid render and therefore receives zero execution score. Stage 1 conditions on the failed program and its execution status and produces a valid TikZ program. The illustrated repairs include cases in which Stage 0 has already produced a complete document that nevertheless fails compilation.

Figure 9: Examples of over-revision at checkpoint 100. Stage 0 already provides a useful reconstruction, while the Stage 1 modification reduces RSim or introduces visible artifacts. The execution-aware selection rule retains Stage 0 in these cases.

Figure 10: Examples of preservation and unresolved cases on Final-945. Stage 1 may preserve an already strong Stage 0 result, or leave an imperfect reconstruction largely unchanged.

S_{0,i}=\begin{cases}\mathrm{RSim}(I_{0,i},I^{\star}),&\text{if execution succeeds},\\ 0,&\text{otherwise}\end{cases}

Appendix

Algorithm 1 One training iteration of our method

Figure 11: Direct generation prompt.

Figure 12: Visual revision prompt.

Figure 13: Execution repair prompt.

	Direct-SFT	Direct-GRPO	Ours
Training duration	3 epochs	1 epoch / 125 iterations	1 epoch / 125 iterations
Learning rate	$1\times 10^{-4}$	$5\times 10^{-6}$	$5\times 10^{-6}$
Weight decay	–	$10^{-2}$	$10^{-2}$
LoRA	$r=256,\ \alpha=512$	continued	continued
Group size $K$	–	8	8
GRPO loss	–	Dr.GRPO	Dr.GRPO

Appendix

Table 3: Optimization and runtime configuration for Direct-SFT, Direct-GRPO, and our method. The final column reports the configuration used by our method in the main experiments.

Method	Training Examples	Initialization	Training Signal
Direct-SFT	3,851	Base	Reference TikZ code
Direct-GRPO	1,000	Direct-SFT	Executed outcome $S$
Ours	1,000	Direct-GRPO	$S_{1}$ / $S_{1}-S_{0}$

Appendix

Table 4: Training data used by the main learning settings. The Direct-GRPO subset is contained in the supervised-training pool, and the same 1K subset is reused by our method.

Split	arXiv	GitHub	TeX.SE	Total
Dev-150	82	43	25	150
Final-945	458	309	178	945
All	540	352	203	1,095

Appendix

Table 5: Composition of the held-out evaluation set. All samples are drawn from source records dated in 2026.

Metric	Role in Training	Main Evaluation Convention
Render success	Execution signal	Success / failure
RSim	RL reward + evaluation	Failure $\rightarrow 0$
DINO	Evaluation only	Failure $\rightarrow 0$
DreamSim	Evaluation only	Failure $\rightarrow 0$
CLIPImg	Evaluation only	Failure $\rightarrow 0$

Appendix

Table 6: Evaluation metrics and their roles. All main visual results use the fixed-zero convention.

Checkpoint	Stage 0 Render	Stage 1 Render	Selected Render	RSim	DINO	DreamSim	CLIPImg
Initializer	82.00%	82.67%	82.67%	0.4736	0.7088	0.6497	0.7468
Step 25	87.33%	88.00%	89.33%	0.5161	0.7667	0.7076	0.8116
Step 50	89.33%	88.00%	90.67%	0.5223	0.7805	0.7160	0.8196
Step 75	91.33%	90.00%	92.00%	0.5454	0.7975	0.7386	0.8416
Step 100	96.67%	94.67%	97.33%	0.5646	0.8326	0.7677	0.8774
Step 125	95.33%	92.67%	95.33%	0.5549	0.8117	0.7538	0.8666

Appendix

Table 7: Checkpoint evaluation for our method on Dev-150. Visual metrics are computed on the execution-aware selected output using the fixed-zero convention. Checkpoint 100 gives the highest selected render rate and all four selected visual metrics.

Checkpoint	Stage 0 Render	Stage 1 Render	SS	SF	FS	FF
Initializer	82.00%	82.67%	123	0	1	26
Step 25	87.33%	88.00%	129	2	3	16
Step 50	89.33%	88.00%	130	4	2	14
Step 75	91.33%	90.00%	134	3	1	12
Step 100	96.67%	94.67%	141	4	1	4
Step 125	95.33%	92.67%	139	4	0	7

Appendix

Table 8: Complete execution dynamics of our method on Dev-150. Each row contains 150 examples.

\mathrm{SS}

\mathrm{SF}

\mathrm{FS}

, and

\mathrm{FF}

denote the execution outcomes of Stage 0 and Stage 1, respectively.

Checkpoint	Paired $n$	$\Delta$ RSim	$\Delta$ DINO	$\Delta$ DreamSim	$\Delta$ CLIPImg	Improve	Neutral	Degrade
Initializer	123	$-0.0031$	$-0.0027$	$-0.0039$	$-0.0017$	12.20%	65.04%	22.76%
Step 25	129	$+0.0013$	$-0.0014$	$+0.0043$	$+0.0042$	28.68%	48.06%	23.26%
Step 50	130	$+0.0098$	$+0.0053$	$+0.0077$	$+0.0016$	43.85%	26.92%	29.23%
Step 75	134	$+0.0121$	$+0.0058$	$+0.0112$	$+0.0063$	42.54%	30.60%	26.87%
Step 100	141	$-0.0012$	$-0.0072$	$-0.0047$	$-0.0000$	43.26%	19.86%	36.88%
Step 125	139	$+0.0065$	$+0.0057$	$+0.0025$	$-0.0013$	46.76%	19.42%	33.81%

Appendix

Table 9: Visual correction dynamics on the paired-success subset of Dev-150. The four

\Delta

columns report the mean Stage 1 minus Stage 0 change. The last three columns show the composition of correction outcomes under the

\pm 0.005

RSim neutral region used in the main text.

Additional Iterations	Method	Render	RSim	DINO	DreamSim	CLIPImg
$+100$	Continued Direct-GRPO	88.25%	0.5135	0.7629	0.7019	0.7996
$+100$	Ours (Stage 0)	92.70%	0.5382	0.8014	0.7330	0.8372
$+125$	Continued Direct-GRPO	87.20%	0.5119	0.7557	0.6984	0.7952
$+125$	Ours (Stage 0)	93.54%	0.5356	0.8011	0.7346	0.8431

Appendix

Table 10: Continued Direct-GRPO single-call results on Final-945. Both methods start from the same Direct-GRPO checkpoint and use the same 1K training images. The

+100

row corresponds to the Dev-selected checkpoint of our method;

+125

reports the training endpoint.

Additional Iterations	Method	Paired $n$	$\Delta$ RSim	$\Delta$ DINO	$\Delta$ DreamSim	$\Delta$ CLIPImg
$+100$	Continued Direct-GRPO	828	$+0.0002$	$+0.0019$	$+0.0020$	$-0.0007$
$+100$	Ours	863	$+0.0073$	$+0.0034$	$+0.0074$	$+0.0031$
$+125$	Continued Direct-GRPO	815	$+0.0017$	$+0.0013$	$+0.0009$	$+0.0008$
$+125$	Ours	869	$+0.0079$	$+0.0046$	$+0.0068$	$+0.0035$

Appendix

Table 11: Continued Direct-GRPO paired-success correction on Final-945.

\Delta M

denotes Stage 1 minus Stage 0 on examples where both outputs render successfully.

Metric	SFT $\rightarrow$ Direct-GRPO	Direct-GRPO $\rightarrow$ Ours Stage 1	Stage 1 $\rightarrow$ Selected
$\Delta$ Render Success	$+17.25$ pp $[+13.86,+20.63]$	$+10.90$ pp $[+8.25,+13.54]$	$+1.38$ pp $[+0.74,+2.12]$
$\Delta$ RSim	$+0.1154$ $[+0.0962,+0.1345]$	$+0.0738$ $[+0.0580,+0.0892]$	$+0.0203$ $[+0.0162,+0.0249]$
$\Delta$ DINO	$+0.1619$ $[+0.1334,+0.1905]$	$+0.1000$ $[+0.0765,+0.1238]$	$+0.0175$ $[+0.0113,+0.0243]$
$\Delta$ DreamSim	$+0.1508$ $[+0.1244,+0.1773]$	$+0.0932$ $[+0.0721,+0.1149]$	$+0.0192$ $[+0.0137,+0.0254]$
$\Delta$ CLIPImg	$+0.1629$ $[+0.1333,+0.1939]$	$+0.0999$ $[+0.0755,+0.1245]$	$+0.0174$ $[+0.0111,+0.0246]$

Appendix

Table 12: Paired bootstrap uncertainty on the fixed Final-945 evaluation set (

N=945

, 10,000 paired resamples). Each entry reports the observed difference with its

95\%

bootstrap interval. Positive values favor the second method. Render differences are in percentage points; visual metrics use fixed-zero scores. Intervals quantify uncertainty over the fixed evaluation samples rather than variation across training seeds.

Variant	Stage 0 credit	Stage 1 credit
A (Ours)	$S_{1}$	$S_{1}-S_{0}$
B	$S_{0}$	$S_{1}-S_{0}$
C	$S_{1}$	$S_{1}$
D	$S_{0}$	$S_{1}$

Appendix

Table 13: Factorized credit-assignment variants. Variant A is our method. The table reports the raw credit before group centering and scaling.

Checkpoint	Variant	Stage 0 Render	Stage 1 Render	$\Delta$ RSim	$\Delta$ DINO	$\Delta$ DreamSim	$\Delta$ CLIPImg
Step 25	A	87.33%	88.00%	$+0.0013$	$-0.0014$	$+0.0043$	$+0.0042$
	B	84.67%	86.00%	$+0.0012$	$-0.0015$	$+0.0005$	$-0.0007$
Step 50	A	89.33%	88.00%	$+0.0098$	$+0.0053$	$+0.0077$	$+0.0016$
	B	90.00%	89.33%	$+0.0002$	$+0.0023$	$+0.0038$	$+0.0001$
Step 75	A	91.33%	90.00%	$+0.0121$	$+0.0058$	$+0.0112$	$+0.0063$
	B	88.00%	88.67%	$-0.0008$	$-0.0023$	$-0.0037$	$-0.0014$

Appendix

Table 14: Comparison of downstream Stage 0 credit (A) and local Stage 0 credit (B) on Dev-150.

\Delta M

denotes the mean Stage 1 minus Stage 0 change on the paired-success subset. A and B use the same improvement-relative Stage 1 credit and differ only in whether Stage 0 receives

S_{1}

S_{0}

Advantage Comparison	Pearson	Spearman	Sign Flip	Group Top-1 Change	Pairwise Ordering Change
$A_{0}(S_{1})$ vs. $A_{0}(S_{0})$	0.7408	0.7083	21.4%	46.1%	–
$A_{1}(S_{1}-S_{0})$ vs. $A_{1}(S_{1})$	0.3848	0.3146	37.2%	74.5%	41.76%

Appendix

Table 15: Comparison of group-relative advantages induced by alternative credit definitions on the same 8,000 trajectories from 1,000 target-image groups. Sign flips are computed over all trajectories, with exact-zero advantages counted as neither positive nor negative. Top-1 change denotes the fraction of groups whose highest-advantage trajectory changes under the alternative definition.

Variant	Stable Until	First Abnormal Step	Observed Behavior
A	125	–	Completed normally
B	$\sim 81$	82	Late length/termination pathology
C	24	25–26	Early zero-signal collapse
D	25	26	Early zero-signal collapse

Appendix

Table 16: Optimization behavior observed for the four factorized credit variants.

Variant	Step	Stage 1 Tokens	Hit Max	End Doc	Stage 0 Compile	Stage 1 Compile	FF	$A_{0}$ Nonzero	$A_{1}$ Nonzero
C	25	2723	31%	45%	10	11	52	48	48
C	26	5639	69%	0%	0	0	64	0	0
C	27	7425	91%	0%	0	0	64	0	0
D	25	1313	11%	80%	38	39	23	64	57
D	26	7041	86%	0%	0	0	64	0	0
D	29–31	8192	100%	0%	0	0	64	0	0

Appendix

Table 17: Training diagnostics around the C/D collapse. “Hit Max” is the fraction of Stage 1 generations reaching the token limit, “End Doc” is the fraction terminating normally at \end{document} , and

A_{0}/A_{1}

nonzero counts the trajectories receiving nonzero group-relative advantages. Each training iteration contains 64 complete trajectories.

Stage 0 RSim	$n$	$\Delta$ RSim	$95\%$ CI	$P(S_{1}>S_{0})$	Improve	Neutral	Degrade
$[0.0,0.4)$	17	$+0.0260$	$[+0.0103,+0.0442]$	76.5%	58.8%	29.4%	11.8%
$[0.4,0.5)$	29	$+0.0029$	$[-0.0103,+0.0168]$	55.2%	41.4%	20.7%	37.9%
$[0.5,0.6)$	39	$+0.0021$	$[-0.0087,+0.0128]$	53.8%	48.7%	7.7%	43.6%
$[0.6,0.7)$	26	$-0.0026$	$[-0.0287,+0.0194]$	53.8%	53.8%	15.4%	30.8%
$[0.7,1.0]$	30	$-0.0235$	$[-0.0593,+0.0026]$	23.3%	20.0%	33.3%	46.7%

Appendix

Table 18: Complete state-dependent correction statistics for our method at checkpoint 100 on Dev-150. The analysis is restricted to paired-success examples.

\Delta

values denote Stage 1 minus Stage 0. The

95\%

intervals are bootstrap intervals over examples within each bucket. Improve, neutral, and degrade use the

\pm 0.005

RSim threshold from the main learning-dynamics analysis, whereas

P(S_{1}>S_{0})

uses the strict

\Delta\mathrm{RSim}>0

criterion.

Stage 0 RSim	$\Delta$ RSim	$\Delta$ DINO	$\Delta$ DreamSim	$\Delta$ CLIPImg
$[0.0,0.4)$	$+0.0260$	$+0.0195$	$+0.0352$	$+0.0191$
$[0.4,0.5)$	$+0.0029$	$+0.0002$	$-0.0001$	$+0.0015$
$[0.5,0.6)$	$+0.0021$	$-0.0166$	$-0.0093$	$-0.0033$
$[0.6,0.7)$	$-0.0026$	$-0.0222$	$-0.0189$	$+0.0030$
$[0.7,1.0]$	$-0.0235$	$-0.0044$	$-0.0133$	$-0.0109$

Appendix

Table 19: Cross-metric Stage 1 changes at checkpoint 100. RSim is the optimization reward; DINO, DreamSim, and CLIPImg are evaluation-only metrics.

Checkpoint	Weak $\Delta$ RSim	Weak $P(S_{1}>S_{0})$	Strong $\Delta$ RSim	Strong $P(S_{1}>S_{0})$
Step 75	$+0.0412$	58.3%	$+0.0045$	28.0%
Step 100	$+0.0260$	76.5%	$-0.0235$	23.3%
Step 125	$+0.0337$	81.3%	$-0.0158$	28.0%

Appendix

Table 20: State-dependent correction across later checkpoints of our method. “Weak” denotes Stage 0 RSim below

0.4

and “strong” denotes Stage 0 RSim in

[0.7,1.0]

. All statistics use paired-success examples and the same fixed bucket boundaries.

Group composition	Groups	Fraction
Revision only	738	73.8%
Mixed revision / repair	259	25.9%
Repair only	3	0.3%
All groups	1,000	100%

Appendix

Table 21: Routing composition of the 1,000 training groups. Each group contains eight independently sampled complete trajectories. A mixed group contains at least one revision trajectory and at least one repair trajectory.

Split	Policy	SS	SF	FS	FF	Stage 0 failures	Repair success
Dev-150	Initializer	123	0	1	26	27	3.7%
Dev-150	Step 100	141	4	1	4	5	20.0%
Final-945	Initializer	780	5	6	154	160	3.8%
Final-945	Step 100	863	13	14	55	69	20.3%

Appendix

Table 22: Execution transitions for the initializer and the Dev-selected checkpoint of our method. “Repair success” is

P(S_{1}\text{ succeeds}\mid S_{0}\text{ fails})

Split	Output	Render	RSim	DINO	DreamSim	CLIPImg
Dev-150	Stage 0	96.67%	0.5479	0.8225	0.7554	0.8668
	Stage 1	94.67%	0.5358	0.7994	0.7348	0.8485
	Selected	97.33%	0.5646	0.8326	0.7677	0.8774
Final-945	Stage 0	92.70%	0.5382	0.8014	0.7330	0.8372
	Stage 1	92.80%	0.5446	0.8046	0.7401	0.8409
	Selected	94.18%	0.5649	0.8221	0.7593	0.8583

Appendix

Table 23: Stage 0, raw Stage 1, and execution-aware selection for our method at checkpoint 100. Visual metrics use the fixed-zero convention.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,704 papers

Success Leaves Detours: Learning Executable Walkthroughs for Long-Horizon Agents
Aug 20, 2026Kaijie Chen, Chenyu Fang, Liang Yan +3Long-Horizon LLM Agents Memory-Augmented Agents
Tongji University · Shanghai AI Laboratory · Fudan University
Execution-Grounded Credit Assignment for GRPO in Code Generation
Mar 17, 2026Abhijit Kumar, Natalya Kumar, Shikhar GuptaRL for Code Generation Credit Assignment
Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents
Oct 2, 2026Yu Li, Guangfeng Cai, Long-Fei Li +5Credit Assignment Step-Level Credit Assignment in RL
School of Computer Science and Engineering, Southeast University, Nanjing, China · Huawei Noah’s Ark Lab, China

Aug 20, 2026J/K move · Enter open · S save

Success Leaves Detours: Learning Executable Walkthroughs for Long-Horizon Agents

Kaijie Chen, Chenyu Fang, Liang Yan, Bo Li+2

Tongji University · Shanghai AI Laboratory · Fudan University

Long-Horizon LLM Agents Memory-Augmented Agents LLM Agent Trajectory Synthesis

Open paper AskPDF arXiv

Test-time self-evolving agents improve by reusing past experience, yet sparse-reward trajectories contain failures, loops, and detours, while summaries often omit the state conditions and action dependencies needed for execution. We study executable Walkthrough induction from sparse-reward trajectories: extracting compact, state-conditioned, and verifiable procedures. Our key observation is that delayed credit identifies actions associated with progress but cannot determine whether they produce facts required by later actions. We propose Trace, a credit-guided, dependency-grounded framework that compiles noisy trajectories into executable Walkthrough Memory. It detects progress anchors from rewards and persistent state changes, propagates credit to identify valuable transitions, and estimates action prerequisites from cross-episode success and failure evidence. Backward dependency slicing then traces required facts to their producers, extracting dependency-consistent action chains while removing irrelevant loops and detours. The resulting Walkthroughs encode entry conditions, ordered state--action--effect steps, and completion and failure predicates, supporting reuse, intermediate-state resumption, and programmatic verification. Experiments on J-TTL, WebShop, and ScienceWorld with three open-source LLMs show that Trace consistently outperforms eight test-time learning and memory baselines. Compared with the strongest baseline, it improves average AUC and Final-

3

30.0%

and

40.5%

, respectively, while using fewer inference tokens. These results show that long-horizon interaction benefits more from state-conditioned executable procedures than from complete trajectories or abstract summaries.

Abstract

Reinforcement Learning

Explore similar work

Success Leaves Detours: Learning Executable Walkthroughs for Long-Horizon Agents

Execution-Grounded Credit Assignment for GRPO in Code Generation

Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents