SCRBench: Beyond Equivalence Checking for Semantic Change Reasoning in Code
Authors: Omid Saghatchian, Ali Mahzoun, Khashayar Etemadi, Saeed Reza Kheradpisheh
Organizations: Department of Computer and Data Sciences Faculty of Mathematical Sciences Shahid Beheshti University Tehran, Iran · Independent Researcher · Department of Computer Science ETH Zurich Zurich, Switzerland
Understanding how code changes affect program semantics is a core aspect of code reasoning in language models. It is commonly evaluated by asking whether two programs are semantically equivalent, but a yes/no answer does not reveal whether a model knows which code change caused a semantic difference, or which sequence of code changes produced the modified program. We study two tasks that ask these questions directly, diagnosing semantic change and reconstructing sequences of code changes, alongside detecting semantic equivalence. We build SCRBench, 67,658 Python program pairs generated by applying sequences of code changes that preserve or alter program semantics, and evaluate five language models on a stratified subset of 4,500 pairs. Detecting semantic equivalence exposes weaker models, whose accuracy declines as changes accumulate, but the strongest models remain near the ceiling. Diagnosing semantic change reveals weaknesses that equivalence accuracy hides: a model with 94.71% equivalence accuracy diagnoses only 54.44% of semantic changes. Reconstructing sequences of code changes is much harder: the best model reconstructs the exact sequence for only 39.27% of pairs, and no model exceeds 1.11% when five changes are applied. High equivalence accuracy therefore overestimates how well models reason about the semantic effects of code changes.
Figures & tables
ID
Abbr.
Type
Effect
1
AOM
Altering
Swaps +↔− and ∗↔/ .
2
BLM
Altering
Negates the first True (else the first False ).
3
LOM
Altering
Swaps and ↔ or .
4
COM
Altering
Flips each comparison’s leading operator ( <↔> , ≤↔≥ , =↔= ).
5
RV
Preserving
Renames the most frequent variable to VAR_0 .
6
DCI_V1
Preserving
Inserts an unused assignment.
Table 1: The nine transformations in the catalog T .
Model
Prompt
Overall
k=1
k=2
k=3
k=4
k=5
Qwen3-235B
Base
88.02%
98.33%
91.78%
88.11%
82.56%
79.33%
Qwen3-235B
Base+CoT
99.60%
100.00%
99.56%
99.56%
99.56%
99.33%
Qwen3-235B
Base+one-shot
85.73%
97.44%
89.67%
87.56%
78.89%
75.11%
Qwen3-235B
Base+CoT+one-shot
99.69%
99.89%
99.22%
99.56%
99.89%
99.89%
GPT-OSS-120B
Base+CoT+one-shot
98.16%
99.22%
98.56%
98.67%
97.22%
97.11%
Mistral-Small-24B
Base+CoT+one-shot
94.71%
98.89%
95.56%
93.56%
93.67%
91.89%
Table 2: T1 accuracy across prompt and model variants.
Model
Prompt
Overall
k=1
k=2
k=3
k=4
k=5
Qwen3-235B
Base
43.51%
93.78%
52.44%
29.11%
24.00%
18.22%
Qwen3-235B
Base+CoT
93.51%
100.00%
92.67%
92.22%
91.33%
91.33%
Qwen3-235B
Base+one-shot
37.29%
96.89%
45.56%
20.44%
15.11%
8.44%
Qwen3-235B
Base+CoT+one-shot
93.47%
100.00%
93.11%
90.67%
91.56%
92.00%
GPT-OSS-120B
Base+CoT+one-shot
97.07%
100.00%
95.33%
96.22%
95.56%
98.22%
Mistral-Small-24B
Base+CoT+one-shot
54.44%
94.00%
58.00%
44.00%
41.33%
34.89%
Table 3: T2 accuracy across prompt and model variants.
Model
AOM
BLM
LOM
COM
Qwen3-235B
93.98%
95.93%
99.77%
86.85%
GPT-OSS-120B
93.57%
99.10%
100.00%
97.65%
Mistral-Small-24B
56.77%
22.17%
62.10%
68.86%
Llama-3.1-8B
28.73%
6.79%
6.85%
37.56%
GPT-4.1-nano
65.25%
43.67%
51.83%
74.65%
Table 4: T2 accuracy by altering transformation using the designated prompt Base+CoT+one-shot.
Model
Prompt
Overall
k=1
k=2
k=3
k=4
k=5
Qwen3-235B
Base
25.42%
78.89%
38.22%
7.78%
2.11%
0.11%
Qwen3-235B
Base+CoT
27.93%
99.33%
32.11%
7.78%
0.33%
0.11%
Qwen3-235B
Base+one-shot
22.47%
75.44%
27.22%
8.56%
1.11%
0.00%
Qwen3-235B
Base+CoT+one-shot
26.82%
99.00%
28.56%
5.89%
0.67%
0.00%
GPT-OSS-120B
Base+CoT+one-shot
39.27%
99.22%
62.56%
26.22%
7.22%
1.11%
Mistral-Small-24B
Base+CoT+one-shot
25.11%
85.11%
35.22%
4.44%
0.78%
0.00%
Table 5: T3 accuracy across prompt and model variants.
Figure 1: Overall accuracy on all tasks for all models, using Base+CoT+one-shot.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
k
Equivalent pairs
Non-equivalent pairs
Total
1
1,315
464
1,779
2
2,958
2,818
5,776
3
4,098
9,889
13,987
4
2,760
20,367
23,127
5
1,379
21,610
22,989
Total
12,510
55,148
67,658
Appendix
Table 6: Number of labeled pairs by transformation sequence length k .
Code language models are now trusted collaborators in production workflows for debugging, refactoring, and iterative repair, and every benchmark that evaluates them assumes the instructions they act on are correct. We study what happens when that assumption breaks. We evaluate code language models across four experiments designed to assess whether models resist or obey incorrect instructions in single-pass and iterative repair settings, using the RunBugRun dataset of algorithmic Python problems with deterministic test cases. Our findings reveal a striking behavioral pattern: models correctly identify an incorrect instruction as wrong, then follow it anyway. This compliance unknowingly introduces errors beyond the original bug, and the corrupted code state cannot be recovered through subsequent self-guided iterative repair, which fails to converge across passes. We term this Blind Obedience, characterize the Ghost (Unknown) Errors it introduces, quantify the proportion of cases where semantic corruption proves irrecoverable, and show that extended reasoning cannot reverse it. These findings surface behavioral properties invisible to pass-rate evaluation, with direct consequences for code language models deployed in production settings.
Raj Jaiswal, Anany Singh Divy, Savar Bhasin +3
IIIT Delhi · Microsoft Research India · IIT Kanpur
We study how well LLMs can determine whether two programs are functionally equivalent. This is an important problem because benchmarking code equivalence helps assess LLM capability in tasks such as code rewriting and translation. To this end, we introduce CETBench - Code Equivalence with Transformations Benchmark - built from a repository of programs that may solve the same or different tasks. Each dataset instance is created by sampling a program pair and applying a random sequence of predefined code transformations, yielding either equivalent or non-equivalent pairs. Our analysis shows that even simple transformations cause a significant performance drop in state-of-the-art LLMs on code-equivalence checking. These challenges are further amplified in the cross-lingual setting when comparing programs written in different languages. To remedy this, we present a simple fine-tuning-based approach to boost LLM performance on the transformed pairs of programs. Our approach for dataset generation is generic, supporting cross-lingual equivalence checking, the generation of program pairs with varying difficulty levels, and the application of diverse transformations. In our experiments, we perform ablations over the difficulty level of original programs, as well as the kind of transformations used in generating pairs for equivalence checking. Our analysis presents deep insights into the working of LLMs for the task of code-equivalence, and points to the fact that they may still be far from what could be termed as a semantic understanding of the underlying code.
Neeva Oza, Ishaan Govil, Parul Gupta +3
Indian Institute of Technology Delhi · IBM Research AI
Large language models receive limited explicit training in reasoning about runtime program states. We study whether training models to reason about runtime program states improves downstream software-engineering capabilities. We introduce two complementary program-state reasoning tasks. Buggy input-output reasoning requires a model to generate a concrete input that exposes a behavioral difference between a buggy program and a hidden correct implementation and to predict the resulting execution behavior. Precondition-postcondition reasoning requires an agent to symbolically characterize a bug-triggering precondition, predict the expected postcondition, explain their causal connection, and instantiate this reasoning as an executable regression test. By incorporating these two tasks into a staged post-training pipeline, we develop Comet-9B, a 9B language model based on Qwen3.5-9B Base. We evaluate the resulting checkpoints on repository-level patch generation, regression-test generation, and security PoC generation. Adding both program-state reasoning tasks to supervised fine-tuning (SFT) on issue resolution improves success rates by 7.25 percentage points on SWE-bench Pro and 9.70 points on SWT-Bench Verified. Sequential reinforcement learning on the two tasks yields further gains of 7.25, 26.79, and 4.67 percentage points on SWE-bench Pro, SWT-Bench Verified, and CyberGym, respectively. Despite having only 9B parameters, Comet-9B achieves a score comparable to the reported GPT-5.2 result on SWE-bench Pro and matches the reported success rate of a GPT-4o-based agent on SWT-Bench Verified.
Hongwei Li, Spandan Garg, Yufan Huang
University of California, Santa Barbara · Microsoft