Improving Large Language Models for Code through Runtime Program-State Reasoning

Hongwei Li; Spandan Garg; Yufan Huang

cs.AISep 28, 2026

Improving Large Language Models for Code through Runtime Program-State Reasoning

Authors: Hongwei Li, Spandan Garg, Yufan Huang

Organizations: University of California, Santa Barbara · Microsoft

Abstract

Large language models receive limited explicit training in reasoning about runtime program states. We study whether training models to reason about runtime program states improves downstream software-engineering capabilities. We introduce two complementary program-state reasoning tasks. Buggy input-output reasoning requires a model to generate a concrete input that exposes a behavioral difference between a buggy program and a hidden correct implementation and to predict the resulting execution behavior. Precondition-postcondition reasoning requires an agent to symbolically characterize a bug-triggering precondition, predict the expected postcondition, explain their causal connection, and instantiate this reasoning as an executable regression test. By incorporating these two tasks into a staged post-training pipeline, we develop Comet-9B, a 9B language model based on Qwen3.5-9B Base. We evaluate the resulting checkpoints on repository-level patch generation, regression-test generation, and security PoC generation. Adding both program-state reasoning tasks to supervised fine-tuning (SFT) on issue resolution improves success rates by 7.25 percentage points on SWE-bench Pro and 9.70 points on SWT-Bench Verified. Sequential reinforcement learning on the two tasks yields further gains of 7.25, 26.79, and 4.67 percentage points on SWE-bench Pro, SWT-Bench Verified, and CyberGym, respectively. Despite having only 9B parameters, Comet-9B achieves a score comparable to the reported GPT-5.2 result on SWE-bench Pro and matches the reported success rate of a GPT-4o-based agent on SWT-Bench Verified.

Figures & tables

Figure 1: Two complementary program-state reasoning tasks, with inputs, model outputs, and verification separated by dashed lines. (a) Without execution tools, the model constructs a bug-triggering input and predicts whether the buggy program terminates normally ( ok ), terminates abnormally due to a runtime error ( crash ), or exceeds the execution time limit ( timeout ). For ok , it also predicts the standard output. (b) The agent investigates a repository and produces a reasoning file and a regression test. The test undergoes execution checks, while both artifacts inform the reasoning judge. Dashed boxes indicate hidden references used only for verification.

Public leaderboard baselines
Model / training stage	Agent	Python	Non-Python	All
DeepSeek V3.2	SWE-Agent	–	–	15.56
GPT-OSS-120B	SWE-Agent	$19.25^{\dagger}$	$14.47^{\dagger}$	16.20
Qwen3-235B-A22B	SWE-Agent	–	–	21.41
Kimi K2 Instruct	SWE-Agent	$40.38^{\dagger}$	$20.47^{\dagger}$	27.67
GPT-5.2	SWE-Agent	–	–	29.94

Table 1: SWE-bench Pro results (%). Published leaderboard baselines ( Scale AI, 2026 ) use SWE-Agent with 250 turns; all models evaluated by us use mini-SWE-agent with 400 steps. Bold marks the best result among models evaluated by us. Dashes denote unavailable subset results;

\dagger

marks subset scores computed from incomplete released outcomes ( Section E.2 ).

Model / training stage	Agent	Success (%)
Published leaderboard baselines
Claude 3.5 Sonnet	OpenHands	27.7
GPT-4o	AssertFlip	45.5
Amazon Q	Amazon Q Developer Agent	51.0
GPT-5-mini	OpenHands	62.4
Qwen baselines

Table 2: SWT-Bench Verified success rates (%) in unit-test mode. All models evaluated by us use mini-SWE-agent with 250 steps. Published baselines are from SWT-Bench Team (2026) ; their configurations differ ( Section E.2 ). Bold marks the best result among models evaluated by us.

	SFT variants			RL variants
	Pure SWE SFT	SWE + Buggy-I/O SFT	Mixed SFT	Buggy-I/O RL	Pre–Post RL	Comet-9B
Success (%)	7.00	5.33	6.00	6.00	11.33	10.67

Table 3: CyberGym success rates (%) on 300 Level-1 tasks, without security-specific post-training. Bold marks the best result.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Run 1	Run 2	Run 3	Run 4	Run 5	Mean $\pm$ std.
Qwen3.5-9B Base	70.0	67.8	71.1	72.2	66.7	69.6 $\pm$ 2.1
Qwen3.5-9B Instruct	73.3	75.6	68.9	75.6	73.3	73.3 $\pm$ 2.4
Mathematics RL (iter. 74)	80.0	74.4	74.4	76.7	73.3	75.8 $\pm$ 2.4
Code RL (iter. 45)	76.7	71.1	74.4	76.7	75.6	74.9 $\pm$ 2.1

Appendix

Table 4: AIME accuracy across five independent runs. All values are percentages, and the final column reports the mean and population standard deviation.

Model	Run 1	Run 2	Run 3	Run 4	Run 5	Mean $\pm$ std.
Qwen3.5-9B Base	37.1	31.7	37.7	32.3	32.3	34.3 $\pm$ 2.6
Qwen3.5-9B Instruct	50.9	57.5	52.1	51.5	54.5	53.3 $\pm$ 2.4
Mathematics RL (iter. 74)	44.3	48.5	55.1	51.5	51.5	50.2 $\pm$ 3.6
Code RL (iter. 45)	56.3	51.5	54.5	53.9	52.7	53.8 $\pm$ 1.6

Appendix

Table 5: LiveCodeBench v5 pass@1 across five independent runs. All values are percentages, and the final column reports the mean and population standard deviation.

Benchmark	Tasks	Max. steps	Required output
SWE-bench Pro	731	400	Source-code patch
SWT-Bench Verified	433	250	Regression-test patch
CyberGym	300	400	Security PoC

Appendix

Table 6: Evaluation tasks and budgets for all models evaluated by us. Each task receives one trajectory.

	SWE-bench Pro			CyberGym
Model / training stage	Python	Non-Python	All
Qwen3.5-9B Base	6/266	3/465	9/731	11/300
Qwen3.5-9B Instruct	75/266	103/465	178/731	27/300
Pure SWE SFT	54/266	63/465	117/731	21/300
SWE + Buggy-I/O SFT	82/266	95/465	177/731	16/300
Mixed SFT	80/266	90/465	170/731	18/300

Appendix

Table 7: Successful tasks / evaluated tasks for the internal model comparisons.

Explore similar work

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
Sep 23, 2026Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband +3Large Language Model Benchmarks LLM Reasoning Strategies
York University Toronto, Canada · The University of Texas at Dallas Texas, USA
ProgramBench: Can Language Models Rebuild Programs From Scratch?
May 5, 2026John Yang, Kilian Lieret, Jeffrey Ma +9Agentic Benchmarks Codebases
1Meta FAIR · 4Harvard University · 2Meta TBD +1
StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning
May 12, 2026Hao Wang, Rui Li, Lei Sha +1Code Generation Search-Augmented Reasoning
Beihang University, Beijing, China · Peking University, Beijing, China · Zhongguancun Laboratory, Beijing, China +1

Sep 23, 2026, cs.SEJ/K move · Enter open · S save

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala+2

York University Toronto, Canada · The University of Texas at Dallas Texas, USA

Large Language Model Benchmarks LLM Reasoning Strategies Runtime Agent Behavior Modeling

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used in coding tasks, but their ability to reason about code execution remains unclear. Existing repository-level QA benchmarks mainly evaluate static code understanding and often rely on LLM-based evaluation, while execution-reasoning benchmarks are mostly limited to snippets or functions. We introduce SWE-Flux, a repository-level benchmark for dynamic execution reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written manually or judged by LLMs. The benchmark covers singletest and multi-test questions over control flow, loops, program state, dataflow, exceptions, and program invariants. Evaluating five LLMs shows that this task remains challenging. The best model achieves only 37% accuracy. Models perform better on localized behavior such as invariants, intra-procedural control flow, exceptions, and simple loops, but struggle with dataflow, inter-procedural execution, precise state reasoning, and suite-level aggregation. Finally, we show that the oracle-harvesting pipeline can generate fresh benchmark variants using input perturbation. It successfully harvests valid variants for almost 90% of the selected instances, and the resulting variants are substantially more challenging for the evaluated models.

Abstract

Explore similar work

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

ProgramBench: Can Language Models Rebuild Programs From Scratch?

StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning