Improving Large Language Models for Code through Runtime Program-State Reasoning

Hongwei Li; Spandan Garg; Yufan Huang

cs.AISep 28, 2026

Improving Large Language Models for Code through Runtime Program-State Reasoning

Authors: Hongwei Li, Spandan Garg, Yufan Huang

Organizations: University of California, Santa Barbara · Microsoft

Abstract

Large language models receive limited explicit training in reasoning about runtime program states. We study whether training models to reason about runtime program states improves downstream software-engineering capabilities. We introduce two complementary program-state reasoning tasks. Buggy input-output reasoning requires a model to generate a concrete input that exposes a behavioral difference between a buggy program and a hidden correct implementation and to predict the resulting execution behavior. Precondition-postcondition reasoning requires an agent to symbolically characterize a bug-triggering precondition, predict the expected postcondition, explain their causal connection, and instantiate this reasoning as an executable regression test. By incorporating these two tasks into a staged post-training pipeline, we develop Comet-9B, a 9B language model based on Qwen3.5-9B Base. We evaluate the resulting checkpoints on repository-level patch generation, regression-test generation, and security PoC generation. Adding both program-state reasoning tasks to supervised fine-tuning (SFT) on issue resolution improves success rates by 7.25 percentage points on SWE-bench Pro and 9.70 points on SWT-Bench Verified. Sequential reinforcement learning on the two tasks yields further gains of 7.25, 26.79, and 4.67 percentage points on SWE-bench Pro, SWT-Bench Verified, and CyberGym, respectively. Despite having only 9B parameters, Comet-9B achieves a score comparable to the reported GPT-5.2 result on SWE-bench Pro and matches the reported success rate of a GPT-4o-based agent on SWT-Bench Verified.

Figures & tables

Figure 1: Two complementary program-state reasoning tasks, with inputs, model outputs, and verification separated by dashed lines. (a) Without execution tools, the model constructs a bug-triggering input and predicts whether the buggy program terminates normally ( ok ), terminates abnormally due to a runtime error ( crash ), or exceeds the execution time limit ( timeout ). For ok , it also predicts the standard output. (b) The agent investigates a repository and produces a reasoning file and a regression test. The test undergoes execution checks, while both artifacts inform the reasoning judge. Dashed boxes indicate hidden references used only for verification.

Public leaderboard baselines
Model / training stage	Agent	Python	Non-Python	All
DeepSeek V3.2	SWE-Agent	–	–	15.56
GPT-OSS-120B	SWE-Agent	$19.25^{\dagger}$	$14.47^{\dagger}$	16.20
Qwen3-235B-A22B	SWE-Agent	–	–	21.41
Kimi K2 Instruct	SWE-Agent	$40.38^{\dagger}$	$20.47^{\dagger}$	27.67
GPT-5.2	SWE-Agent	–	–	29.94

Table 1: SWE-bench Pro results (%). Published leaderboard baselines ( Scale AI, 2026 ) use SWE-Agent with 250 turns; all models evaluated by us use mini-SWE-agent with 400 steps. Bold marks the best result among models evaluated by us. Dashes denote unavailable subset results;

\dagger

marks subset scores computed from incomplete released outcomes ( Section E.2 ).

Model / training stage	Agent	Success (%)
Published leaderboard baselines
Claude 3.5 Sonnet	OpenHands	27.7
GPT-4o	AssertFlip	45.5
Amazon Q	Amazon Q Developer Agent	51.0
GPT-5-mini	OpenHands	62.4
Qwen baselines

Table 2: SWT-Bench Verified success rates (%) in unit-test mode. All models evaluated by us use mini-SWE-agent with 250 steps. Published baselines are from SWT-Bench Team (2026) ; their configurations differ ( Section E.2 ). Bold marks the best result among models evaluated by us.

	SFT variants			RL variants
	Pure SWE SFT	SWE + Buggy-I/O SFT	Mixed SFT	Buggy-I/O RL	Pre–Post RL	Comet-9B
Success (%)	7.00	5.33	6.00	6.00	11.33	10.67

Table 3: CyberGym success rates (%) on 300 Level-1 tasks, without security-specific post-training. Bold marks the best result.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Run 1	Run 2	Run 3	Run 4	Run 5	Mean $\pm$ std.
Qwen3.5-9B Base	70.0	67.8	71.1	72.2	66.7	69.6 $\pm$ 2.1
Qwen3.5-9B Instruct	73.3	75.6	68.9	75.6	73.3	73.3 $\pm$ 2.4
Mathematics RL (iter. 74)	80.0	74.4	74.4	76.7	73.3	75.8 $\pm$ 2.4
Code RL (iter. 45)	76.7	71.1	74.4	76.7	75.6	74.9 $\pm$ 2.1

Appendix

Table 4: AIME accuracy across five independent runs. All values are percentages, and the final column reports the mean and population standard deviation.

Model	Run 1	Run 2	Run 3	Run 4	Run 5	Mean $\pm$ std.
Qwen3.5-9B Base	37.1	31.7	37.7	32.3	32.3	34.3 $\pm$ 2.6
Qwen3.5-9B Instruct	50.9	57.5	52.1	51.5	54.5	53.3 $\pm$ 2.4
Mathematics RL (iter. 74)	44.3	48.5	55.1	51.5	51.5	50.2 $\pm$ 3.6
Code RL (iter. 45)	56.3	51.5	54.5	53.9	52.7	53.8 $\pm$ 1.6

Appendix

Table 5: LiveCodeBench v5 pass@1 across five independent runs. All values are percentages, and the final column reports the mean and population standard deviation.

Benchmark	Tasks	Max. steps	Required output
SWE-bench Pro	731	400	Source-code patch
SWT-Bench Verified	433	250	Regression-test patch
CyberGym	300	400	Security PoC

Appendix

Table 6: Evaluation tasks and budgets for all models evaluated by us. Each task receives one trajectory.

	SWE-bench Pro			CyberGym
Model / training stage	Python	Non-Python	All
Qwen3.5-9B Base	6/266	3/465	9/731	11/300
Qwen3.5-9B Instruct	75/266	103/465	178/731	27/300
Pure SWE SFT	54/266	63/465	117/731	21/300
SWE + Buggy-I/O SFT	82/266	95/465	177/731	16/300
Mixed SFT	80/266	90/465	170/731	18/300

Appendix

Table 7: Successful tasks / evaluated tasks for the internal model comparisons.

Explore similar work

Sep 23, 2026cs.SE

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

Large language models (LLMs) are increasingly used in coding tasks, but their ability to reason about code execution remains unclear. Existing repository-level QA benchmarks mainly evaluate static code understanding and often rely on LLM-based evaluation, while execution-reasoning benchmarks are mostly limited to snippets or functions. We introduce SWE-Flux, a repository-level benchmark for dynamic execution reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written manually or judged by LLMs. The benchmark covers singletest and multi-test questions over control flow, loops, program state, dataflow, exceptions, and program invariants. Evaluating five LLMs shows that this task remains challenging. The best model achieves only 37% accuracy. Models perform better on localized behavior such as invariants, intra-procedural control flow, exceptions, and simple loops, but struggle with dataflow, inter-procedural execution, precise state reasoning, and suite-level aggregation. Finally, we show that the oracle-harvesting pipeline can generate fresh benchmark variants using input perturbation. It successfully harvests valid variants for almost 90% of the selected instances, and the resulting variants are substantially more challenging for the evaluated models.

Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband +3

York University Toronto, Canada · The University of Texas at Dallas Texas, USA

Large Language Model Benchmarks LLM Reasoning Strategies Runtime Agent Behavior Modeling

Ask PDF

May 5, 2026cs.SE

ProgramBench: Can Language Models Rebuild Programs From Scratch?

Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95% of tests on only 3% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.

John Yang, Kilian Lieret, Jeffrey Ma +9

1Meta FAIR · 4Harvard University · 2Meta TBD +1

Agentic Benchmarks Codebases

Ask PDF

May 12, 2026cs.SE

StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning

Existing code reasoning methods primarily supervise final code outputs, ignoring intermediate states, often leading to reward hacking where correct answers are obtained through inconsistent reasoning. We propose StepCodeReasoner, a framework that introduces explicit intermediate execution-state supervision. By automatically inserting structured print-based execution-trace anchors into code, the model is trained to predict runtime states at each step, transforming code reasoning into a verifiable, stepwise execution modeling problem. Building on this execution-aware method, we introduce Bi-Level GRPO, a reinforcement learning algorithm for structured credit assignment at two levels: inter-trajectory, comparing alternative execution paths, and intra-trajectory, rewarding intermediate accuracy based on its impact on downstream correctness. Extensive experiments demonstrate that StepCodeReasoner achieves SOTA performance in code reasoning. In particular, our 7B model achieves 91.1% on CRUXEval and 86.5% on LiveCodeBench, outperforming the CodeReasoner-7B baseline (86.0% and 77.7%) and GPT-4o (85.6% and 75.1%). Furthermore, on the execution-trace benchmark REval, our model scores 82.9%, outperforming baseline CodeReasoner-7B (72.3%), its 14B counterpart (81.1%), and GPT-4o (77.3%). Additionally, our approach also improves code generation performance, demonstrating that explicit execution modeling enhances both code reasoning and code generation.

Hao Wang, Rui Li, Lei Sha +1

Beihang University, Beijing, China · Peking University, Beijing, China · Zhongguancun Laboratory, Beijing, China +1

Code Generation Search-Augmented Reasoning Reasoning Skills Process-Level Supervision Traces

Ask PDF