Improving a coding model is a sequence of decisions: find the limiting behavior, create useful practice, and determine which attempts deserve reinforcement. We investigate a practical way to make those decisions increasingly repeatable by grounding feedback in execution. Starting from GLM-4.7-Flash, the work combines whole-file supervised fine-tuning, targeted C++ repair curricula, group-relative reinforcement learning, and verifiers that turn program behavior into structured rewards. Compiler errors and failing tests provide specific contexts for repair practice, while executable checks score the resulting programs. We evaluate the trained models on 26 C++ coding tasks, measuring both initial success and completion after test feedback. The strongest reported post-training configuration achieves 45.2% success in the first test round and 61.5% within two rounds. These results come from an assisted development suite with evolving prompt contracts; the strongest configuration uses four selected evaluation trials and includes six tasks also used in training. The study describes an execution-guided workflow for improving coding models, including how repair tasks are constructed, how program behavior determines rewards, and how the resulting models are evaluated.
Figures & tables
Figure 1: Execution-guided post-training overview. Blue groups task construction and validation, green shows model training, and ochre shows checkpoint evaluation. Within the training loop, execution-derived rewards guide GRPO adapter updates. The dashed handoff marks the researcher’s selection of the next intervention. This schematic summarizes components across experiments with different training lineages and reward configurations; the evaluation qualifications in Section 5.2 apply.
Concrete input or observation
Required behavior
Lesson
+1 (223) 456-7890
Return 2234567890; render (223) 456-7890.
Preserve valid normalization.
2231567890
Throw std::domain_error .
Validate the exchange prefix.
Member named number and method number()
Produce an interface that compiles.
Repair the declaration collision.
Table 1: Phone Number inputs, requirements, and targeted repair mechanisms.
Group
Checks
Basic valid-input behavior ( PH-E01 )
Plain ten-digit input, area-code and number accessors, and a leading country code of 1.
Formatting and rejection rules ( PH-E02 )
Permitted formatting; invalid length, country code, and characters; invalid area and exchange prefixes.
Formatting and boundary cases ( PH-E03 )
Whitespace, the valid number 9999999999, exact string formatting, and empty, overlong, or malformed inputs.
Combined interface and input checks ( PH-E04 )
Consistent public outputs, multiple valid formats, and combined rejection cases.
Table 2: Coverage of the four verification groups. Descriptive labels summarize the checks; identifiers match the released implementation. Each group contains three Boolean kernels.
Metric
Calculation across four trials
Interpretation
First-round mean
First-round successes / 4
Tasks solved immediately, out of 26.
Two-round mean
Tasks solved by round two / 4
Cumulative completion, out of 26.
Conditional recovery
New round-two successes / first-round failures
Use of the available feedback opportunity.
Table 3: Evaluation metrics and their denominators.
Figure 2: Mean tasks solved out of 26, from the release results. Four trials per row. *Generalized C++ is the selected best-four cohort. Bars show observed outcomes under the contracts in Section 5.2 ; the complete set of configurations appears below.
Configuration
First round mean / 26
Two rounds mean / 26
Conditional recovery
GLM-4.7-Flash base
0.50 (1.9%)
4.50 (17.3%)
16/102 (15.7%)
Luna reference
6.25 (24.0%)
16.75 (64.4%)
42/79 (53.2%)
SFT v5
6.00 (23.1%)
10.25 (39.4%)
17/80 (21.2%)
Synth v1, epoch 50
9.50 (36.5%)
12.00 (46.2%)
10/66 (15.2%)
Execution-midband RL v1
8.25 (31.7%)
13.00 (50.0%)
19/71 (26.8%)
Execution-midband RL v2
10.50 (40.4%)
14.50 (55.8%)
16/62 (25.8%)
Table 4: Release-reported means and recovery counts. Each row contains four trials and 104 task-trial trajectories. The Generalized C++ row is a development-suite evaluation using test-derived API/build prompt information with the selection and training overlap described in Section 5.2 .
Task
Synth v1 first-round successes
Phone Number first-round successes
Phone Number
1 / 4
4 / 4
Parallel Letter Frequency
0 / 4
3 / 4
Gigasecond
3 / 4
0 / 4
Sublist
3 / 4
1 / 4
Table 5: Selected task-level outcomes from the released per-task table.
Research and engineering decisions
Repeated operations handled by the system
Choose the target behavior and intervention.
Construct the configured task variants and prompts.
Define requirements, references, and controls.
Compile, execute, score, and retain check outcomes.
Select training settings and checkpoints to study.
Sample responses, update adapters, and run configured evaluations.
Interpret results and choose the next experiment.
Produce task-level records for comparison and diagnosis.
Table 6: Research decisions and repeated operations handled by the configured system.
Sequence limit 12,288; response limit 8,192; temperature 0.7; thinking enabled.
Appendix
Table 7: Recorded phone-number validation training configuration.
Reward path
Definition and role
Whole-file execution reward
Default scoring: prohibited edits or runtime violations -1; other parse failures -0.8; compilation failure or candidate timeout -0.5; full pass +1; otherwise 0.6 × the fraction of passed tests. Recoverable formatting errors incur a penalty of 0.1; scores are clipped to [-1,1]. A strict binary mode is also available.
Phone-number validation reward
Mean of 12 signed Boolean check outcomes, in [-1,1]. The recorded outcomes replace the default execution reward. Execution-environment errors are flagged separately.
Current shared C++ verification reward
Validated pass +1 (0.975 with a recoverable formatting error). Failed candidates receive -1 plus weighted passed-check credit and available semantic-test credit, clipped to [-1,0]. Invalid execution records are flagged as infrastructure errors.
Current task-specific verification adjustment
A positive shared verification score is retained when required task-specific checks pass. Otherwise use 0.5min(Rs,0)+0.5(ft−1) . Invalid task-specific evaluations are flagged separately.
Appendix
Table 8: Experiment-specific reward definitions and roles.
Configuration
First round SD; range; 95% CI
Two rounds SD; range; 95% CI
Recovery 95% CI
Base
0.58; 0-1; 0-1.25
1.29; 3-6; 2.25-6.75
7.8-24.8%
Luna
1.89; 5-9; 3.25-9.5
1.50; 15-18; 12.75-20.5
36.8-70%
SFT v5
1.63; 4-8; 3-9.25
1.71; 8-12; 7-13.75
11.9-31.6%
Synth v1
0.58; 9-10; 5.75-13.5
0.82; 11-13; 8.25-15.75
7.4-24.4%
Midband RL v1
2.06; 6-11; 4.75-12
0.82; 12-14; 9-17
14.5-41.4%
Midband RL v2
1.29; 9-12; 7-14
2.08; 12-17; 10.5-18.5
12.9-41.9%
Appendix
Table 9: Trial variability and task-bootstrap confidence intervals for the reported configurations.
A promising research direction in enabling LLMs to generate consistently correct code involves addressing their inability to properly estimate program execution, particularly for code they generate. In this work, we demonstrate that Code LLMs can be trained to simulate program execution in a step-by-step manner and that this capability can be leveraged to improve competitive programming performance. Our approach combines supervised fine-tuning on natural language execution traces, textual explanations grounded in true execution, with reinforcement learning using verifiable rewards. We introduce two complementary objectives: output prediction given code and inputs, and solving competitive programming tasks with either ground-truth or self-predicted execution feedback. These objectives enable models to perform self-verification over multiple candidate solutions, and iterative self-fixing by simulating test execution. Across multiple competitive programming benchmarks, our method yields consistent improvements over standard reasoning approaches. We further present ablations and analysis to elucidate the role of execution simulation and its limitations.
Gallil Maimon, Ori Yoran, Felix Kreuk +4
FAIR team, Meta · Hebrew University of Jerusalem · Inria
Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue. Agent-generated tests can encode incomplete or incorrect behavioral targets; when the same trajectory writes both the patch and the test, their errors can agree and create false confidence. We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it. The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests. Both roles use Qwen-3.5-35B-A3B as the backbone and are trained separately. In Learn to Test, the Test agent learns to produce behaviorally valid tests that distinguish correct from incorrect patches. In Test to Improve, the Repair agent learns both direct task resolution and feedback-guided revision. On SWE-bench Verified, test quality determines whether feedback helps: holding the base Repair agent fixed, tests from the base Test agent reduce resolved rate from a no-test baseline of 61.2% to 57.3%, whereas tests from GPT-5.6-sol raise it to 65.3%. Role-specific post-training raises the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%; composing the two post-trained Qwen agents reaches 72.6%, an 11.4-point gain over the original no-test baseline without stronger-model or Oracle feedback at evaluation time. Code is publicly available at https://github.com/MSR-Orchard/execcritic.
Leitian Tao, Baolin Peng, Haorui Wang +7
University of Wisconsin–Madison · Microsoft Research · Georgia Tech
Reinforcement learning (RL) from unit tests is the common recipe for post-training code models, but a unit test only asks whether the final program passes. We analyze what this unit-test-only reward teaches a model during debugging and find that, while it raises pass rates, it also teaches the model to edit more and to hallucinate bugs in code that is already correct. We introduce PreciseCoder, a training algorithm that augments the unit-test reward with a signal for semantically verified, localized edits, which correlates with smaller edits and higher pass rates. On 4B, 9B, and 27B models, PreciseCoder reduces the average number of edit lines from 7-15to 3 at a pass rate on par with unit-test-only training. Trained only on single-line bugs, PreciseCoder can generalize to repository-level program repair, where it resolves more issues with smaller patches. Its precision also transfers from weaker to stronger models: given the reasoning trace of a 9B PreciseCoder model, frontier models such as GPT-5.6 Sol and Qwen3.8-Max gain 4.6-12.4 points in edit precision with no additional training. Linguistic analysis of the thinking traces shows that our reward encourages the model to reason about individual lines, name the buggy ones, and form consistent bug hypotheses.
Yunkai Zhan, Miaosen Chai, Shangshang Wang +5
University of Southern California · University of Chicago · Google +1