How Robustly do LLMs Understand Execution Semantics?

Claudio Spiess; Prem Devanbu; Earl T. Barr

cs.SEFeb 24, 2026

How Robustly do LLMs Understand Execution Semantics?

Authors: Claudio Spiess, Prem Devanbu, Earl T. Barr

Organizations: UC Davis, Davis, California, USA · University College London, London, UK

Abstract

LLMs demonstrate remarkable reasoning capabilities, yet whether they utilize internal world models or rely on sophisticated pattern matching remains open. We study LLMs through the lens of robustness of their code understanding using a standard program-output prediction task. Our results reveal a stark divergence in model behavior: while open-source reasoning models (DeepSeek-R1 family) maintain stable, albeit somewhat lower accuracies (38% to 67%) under code transformations & input perturbations, the frontier model GPT-5.2 exhibits significant brittleness. Despite achieving a near-perfect score of 99% on the original, unperturbed CRUXEval benchmark, perturbed inputs trigger accuracy declines between 20% and 24%. In addition, we find that many models perform much worse at predicting behavior on perturbed inputs that raise exceptions, and that prediction performance depends on the kind of exception. We study remedies to address this deficiency in exception prediction, and evaluate the effect of these remedies on the ability to predict non-exception behaviors. Our findings both point to limitations in the way all models understand code, and establish the value of using perturbation to evaluate code models.

Figures & tables

Model	$C_{o}$ (↑)	$C_{\tilde{o}}$ (↑)	$C_{o\,\cup\,\tilde{o}}$ (↑)	PSR (↑)	$\mathcal{R}_{\Delta}$ (↑)
Qwen2.5 Math 1.5B	$12.481644640234949$	$12.21732745961821$	$12.24135629421973$	$1.908957415565345$	$-0.26431718061673964$
DS R1 1.5B	$38.45029239766082$	$37.5$	$37.58639021796917$	$4.385964912280701$	$-0.950292397660818$
Qwen2.5 Math 7B	$34.941520467836256$	$33.2748538011696$	$33.42636895268474$	$11.11111111111111$	$-1.6666666666666607$
Qwen2.5 Instr 7B	$40.93567251461988$	$41.812865497076025$	$41.73312068048911$	$16.812865497076025$	$+0.8771929824561431$
Nemotron 7B	$46.198830409356724$	$43.260233918128655$	$43.52737905369484$	$4.385964912280701$	$-2.9385964912280684$
DS R1 7B	$65.05847953216374$	$62.61695906432748$	$62.83891547049441$	$27.339181286549707$	$-2.4415204678362556$

Table 1. Accuracy & robustness metrics for original vs. perturbed CruxEval inputs . Arrow ↑ indicates higher values are preferred. Each model was evaluated on 684 programs, for which the original benchmark input and 10 unique perturbed variants were available.

Model	Strict Match %	Any Match %
GPT-5 Nano	$71.9468$	$73.156$
GPT-5.2	$14.7659$	$14.7659$
Gemini 3 Pro	$48.5612$	$50.7194$

Table 2. Performance of the frontier models on accuracy of prediction the precise exception that actually occurs (“Strict Match”) or that some exception occurs (“Any Match”)

Model	CruxEval	$\textsc{CruxEval}_{p}$	$\textsc{CruxEval}_{v}$
Qwen2.5 Math 1.5B	$12.25$	$9.26966$ *	$11.125$
DS R1 1.5B	$40$	$40.4494$	$35.5$ *
Qwen2.5 Math 7B	$37$	$35.5337$	$34.375$
Qwen2.5 Instr 7B	$42.625$	$42.9775$	$40.5$
Nemotron 7B	$47$	$37.2191$ ***	$44.75$
DS R1 7B	$62.625$	$60.2528$	$61.5$

Table 3. Average model accuracy on canonical CruxEval , code-perturbed

\textsc{CruxEval}_{p}

, and

\textsc{CruxEval}_{v}

. Asterisks *, **, and *** represent McNemar test

p<0.05

0.01

, and

0.001

respectively, for original vs. code-perturbed. As each model is a different experiment, we do not make claims across all models based the p -values and thus do not apply corrections.

Figure 1. Number of decisions encountered on execution path vs. model accuracy on output prediction. The yellow line shows the varying number of sample programs with increasing numbers of conditions on the execution paths

Number of Decisions	Samples	Accuracy (%)	Unique Programs	Unique Inputs
(0, 1]	$58,141$	$58.5353$	$462$	$4,154$
(1, 3]	$17,177$	$55.4753$	$208$	$1,227$
(3, 4]	$5,487$	$47.4941$	$104$	$392$
(4, 5]	$4,353$	$45.1643$	$116$	$311$
(5, 7]	$6,230$	$48.0578$	$134$	$445$
(7, 10]	$5,106$	$48.4528$	$127$	$365$

Table 4. Binned number of decisions encountered during execution vs. mean model accuracy. Columns report number of samples (model, program, input triples), unique programs and unique inputs per bin.

Model	Prompt	Strict Match %	Any Match %
GPT-5 Nano	Original	$71.9468$	$73.156$
GPT-5 Nano	Exceptions	$88.4892$	$93.6451$
GPT-5 Nano	Type	$73.5012$	$74.7002$
GPT-5 Nano	Exc. + Type	$89.9281$	$95.9233$
GPT-5.2	Original	$14.7659$	$14.7659$
GPT-5.2	Exceptions	$80.5755$	$88.6091$

Table 6. Results for frontier models on

\textsc{CruxEval}_{\text{exc}}

. Strict Match requires the generation to contain either the ground-truth exception type, exception message, or a heuristic substring. Any Match requires the generation to contain any exception type or heuristic substring e.g. , model output contains “infinite” , but ground-truth exception is TypeError .

Panel of Figure 2 — Figure 2. The prompts we used: the “Direct Output Prompt” is from the original CruxEval paper Gu et al., 2024 . The lower prompt adds instructions for both type-tracking and exception-tracking . We also experimented with each separately

Model	Prompt	$C_{o}$ (↑)	$C_{\tilde{o}}$ (↑)	$C_{o\,\cup\,\tilde{o}}$ (↑)	PSR (↑)
GPT-5 Nano	Original	$96.7836$	$96.6082$	$96.6241$	$82.7485$
GPT-5.2	Original	$99.1228$	$79.6053$	$81.3796$	$56.1404$
Gemini 3 Pro	Original	$99.1045$	$99.3582$	$99.3351$	$97.1642$
GPT-5 Nano	Exc.+Type	$97.076$	$96.8713$	$96.89$	$83.6257$
GPT-5.2	Exc.+Type	$76.1696$	$75.9795$	$75.9968$	$50.2924$
Gemini 3 Pro	Exc.+Type	$99.4152$	$99.3275$	$99.3355$	$97.2222$

Table 7. Evaluating the combined Type & Exception Tracking prompt on the (original and perturbed) inputs that run without exceptions

Code Language Models

Code language models are AI models that generate, explain, edit, and reason about source code across programming tasks.

See all 114 papers

From Brewing to Resolution: Tracing the Internal Lifecycle of Code Reasoning in LLMs
Jun 16, 2026Siyue Chen, Yifu Guo, Yuquan Lu +9LLM Interpretability Code Language Models
1South China University of Technology · 2Sun Yat-sen University · 4Shanghai Jiao Tong University +5
Predicting Program Exit Code with LLMs and Programming Language Semantics
Sep 1, 2026Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa +2LLM Evaluation Code Language Models
The University of Texas at Austin · Cisco Research
Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse
Jul 5, 2026Raj Jaiswal, Anany Singh Divy, Savar Bhasin +3LLM Self-Correction Automated Program Repair
IIIT Delhi · Microsoft Research India · IIT Kanpur

Jun 16, 2026J/K move · Enter open · S save

From Brewing to Resolution: Tracing the Internal Lifecycle of Code Reasoning in LLMs

Siyue Chen, Yifu Guo, Yuquan Lu, Zishan Xu+8

1South China University of Technology · 2Sun Yat-sen University · 4Shanghai Jiao Tong University+5

LLM Interpretability Code Language Models LLM Reasoning Reasoning Trace Analysis

Open paper AskPDF arXiv

Standard accuracy metrics cannot explain why LLMs handle variable tracking but fail on semantically equivalent loops. We study an internal lifecycle of code reasoning in which models first brew the answer, making it linearly recoverable many layers before it becomes self-decodable, and then diverge into one of four resolution outcomes: Resolved, Overprocessed, Misresolved, or Unresolved. Understanding this lifecycle matters because similar task accuracies can mask fundamentally different failure modes that surface-level evaluation cannot detect. We introduce a dual diagnostic framework pairing layer-wise linear probing with Context-Stripped Decoding (CSD) and apply it to six code-reasoning task families across 16 models spanning Qwen, Llama, and DeepSeek architectures. All four outcomes carry substantial mass in every task family: overall Resolved is only 41.5%, with multiple tasks below 30%. Controlled sweeps over structure, depth, and operators expose task-specific failure bottlenecks: Function Call Resolved plunges from 61.1% to 2.5% as call depth increases from one to three. Across architectures and scales, the brewing scaffold remains stable, with normalized brewing duration 24-42% across all 16 models, while resolution success varies with capability. This indicates that the scaffold is a stable empirical regularity across the tested decoder-only Transformer families, whereas resolution success covaries with capability, scale, and training. Code: https://github.com/euyis1019/llm-brewing

Abstract

Code Language Models

Explore similar work

From Brewing to Resolution: Tracing the Internal Lifecycle of Code Reasoning in LLMs

Predicting Program Exit Code with LLMs and Programming Language Semantics

Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse