Improving a coding model is a sequence of decisions: find the limiting behavior, create useful practice, and determine which attempts deserve reinforcement. We investigate a practical way to make those decisions increasingly repeatable by grounding feedback in execution. Starting from GLM-4.7-Flash, the work combines whole-file supervised fine-tuning, targeted C++ repair curricula, group-relative reinforcement learning, and verifiers that turn program behavior into structured rewards. Compiler errors and failing tests provide specific contexts for repair practice, while executable checks score the resulting programs. We evaluate the trained models on 26 C++ coding tasks, measuring both initial success and completion after test feedback. The strongest reported post-training configuration achieves 45.2% success in the first test round and 61.5% within two rounds. These results come from an assisted development suite with evolving prompt contracts; the strongest configuration uses four selected evaluation trials and includes six tasks also used in training. The study describes an execution-guided workflow for improving coding models, including how repair tasks are constructed, how program behavior determines rewards, and how the resulting models are evaluated.
Figures & tables
Figure 1: Execution-guided post-training overview. Blue groups task construction and validation, green shows model training, and ochre shows checkpoint evaluation. Within the training loop, execution-derived rewards guide GRPO adapter updates. The dashed handoff marks the researcher’s selection of the next intervention. This schematic summarizes components across experiments with different training lineages and reward configurations; the evaluation qualifications in Section 5.2 apply.
Concrete input or observation
Required behavior
Lesson
+1 (223) 456-7890
Return 2234567890; render (223) 456-7890.
Preserve valid normalization.
2231567890
Throw std::domain_error .
Validate the exchange prefix.
Member named number and method number()
Produce an interface that compiles.
Repair the declaration collision.
Table 1: Phone Number inputs, requirements, and targeted repair mechanisms.
Group
Checks
Basic valid-input behavior ( PH-E01 )
Plain ten-digit input, area-code and number accessors, and a leading country code of 1.
Formatting and rejection rules ( PH-E02 )
Permitted formatting; invalid length, country code, and characters; invalid area and exchange prefixes.
Formatting and boundary cases ( PH-E03 )
Whitespace, the valid number 9999999999, exact string formatting, and empty, overlong, or malformed inputs.
Combined interface and input checks ( PH-E04 )
Consistent public outputs, multiple valid formats, and combined rejection cases.
Table 2: Coverage of the four verification groups. Descriptive labels summarize the checks; identifiers match the released implementation. Each group contains three Boolean kernels.
Metric
Calculation across four trials
Interpretation
First-round mean
First-round successes / 4
Tasks solved immediately, out of 26.
Two-round mean
Tasks solved by round two / 4
Cumulative completion, out of 26.
Conditional recovery
New round-two successes / first-round failures
Use of the available feedback opportunity.
Table 3: Evaluation metrics and their denominators.
Figure 2: Mean tasks solved out of 26, from the release results. Four trials per row. *Generalized C++ is the selected best-four cohort. Bars show observed outcomes under the contracts in Section 5.2 ; the complete set of configurations appears below.
Configuration
First round mean / 26
Two rounds mean / 26
Conditional recovery
GLM-4.7-Flash base
0.50 (1.9%)
4.50 (17.3%)
16/102 (15.7%)
Luna reference
6.25 (24.0%)
16.75 (64.4%)
42/79 (53.2%)
SFT v5
6.00 (23.1%)
10.25 (39.4%)
17/80 (21.2%)
Synth v1, epoch 50
9.50 (36.5%)
12.00 (46.2%)
10/66 (15.2%)
Execution-midband RL v1
8.25 (31.7%)
13.00 (50.0%)
19/71 (26.8%)
Execution-midband RL v2
10.50 (40.4%)
14.50 (55.8%)
16/62 (25.8%)
Table 4: Release-reported means and recovery counts. Each row contains four trials and 104 task-trial trajectories. The Generalized C++ row is a development-suite evaluation using test-derived API/build prompt information with the selection and training overlap described in Section 5.2 .
Task
Synth v1 first-round successes
Phone Number first-round successes
Phone Number
1 / 4
4 / 4
Parallel Letter Frequency
0 / 4
3 / 4
Gigasecond
3 / 4
0 / 4
Sublist
3 / 4
1 / 4
Table 5: Selected task-level outcomes from the released per-task table.
Research and engineering decisions
Repeated operations handled by the system
Choose the target behavior and intervention.
Construct the configured task variants and prompts.
Define requirements, references, and controls.
Compile, execute, score, and retain check outcomes.
Select training settings and checkpoints to study.
Sample responses, update adapters, and run configured evaluations.
Interpret results and choose the next experiment.
Produce task-level records for comparison and diagnosis.
Table 6: Research decisions and repeated operations handled by the configured system.
Sequence limit 12,288; response limit 8,192; temperature 0.7; thinking enabled.
Appendix
Table 7: Recorded phone-number validation training configuration.
Reward path
Definition and role
Whole-file execution reward
Default scoring: prohibited edits or runtime violations -1; other parse failures -0.8; compilation failure or candidate timeout -0.5; full pass +1; otherwise 0.6 × the fraction of passed tests. Recoverable formatting errors incur a penalty of 0.1; scores are clipped to [-1,1]. A strict binary mode is also available.
Phone-number validation reward
Mean of 12 signed Boolean check outcomes, in [-1,1]. The recorded outcomes replace the default execution reward. Execution-environment errors are flagged separately.
Current shared C++ verification reward
Validated pass +1 (0.975 with a recoverable formatting error). Failed candidates receive -1 plus weighted passed-check credit and available semantic-test credit, clipped to [-1,0]. Invalid execution records are flagged as infrastructure errors.
Current task-specific verification adjustment
A positive shared verification score is retained when required task-specific checks pass. Otherwise use 0.5min(Rs,0)+0.5(ft−1) . Invalid task-specific evaluations are flagged separately.
Appendix
Table 8: Experiment-specific reward definitions and roles.
Configuration
First round SD; range; 95% CI
Two rounds SD; range; 95% CI
Recovery 95% CI
Base
0.58; 0-1; 0-1.25
1.29; 3-6; 2.25-6.75
7.8-24.8%
Luna
1.89; 5-9; 3.25-9.5
1.50; 15-18; 12.75-20.5
36.8-70%
SFT v5
1.63; 4-8; 3-9.25
1.71; 8-12; 7-13.75
11.9-31.6%
Synth v1
0.58; 9-10; 5.75-13.5
0.82; 11-13; 8.25-15.75
7.4-24.4%
Midband RL v1
2.06; 6-11; 4.75-12
0.82; 12-14; 9-17
14.5-41.4%
Midband RL v2
1.29; 9-12; 7-14
2.08; 12-17; 10.5-18.5
12.9-41.9%
Appendix
Table 9: Trial variability and task-bootstrap confidence intervals for the reported configurations.