SCRBench: Beyond Equivalence Checking for Semantic Change Reasoning in Code
Authors: Omid Saghatchian, Ali Mahzoun, Khashayar Etemadi, Saeed Reza Kheradpisheh
Organizations: Department of Computer and Data Sciences Faculty of Mathematical Sciences Shahid Beheshti University Tehran, Iran · Independent Researcher · Department of Computer Science ETH Zurich Zurich, Switzerland
Understanding how code changes affect program semantics is a core aspect of code reasoning in language models. It is commonly evaluated by asking whether two programs are semantically equivalent, but a yes/no answer does not reveal whether a model knows which code change caused a semantic difference, or which sequence of code changes produced the modified program. We study two tasks that ask these questions directly, diagnosing semantic change and reconstructing sequences of code changes, alongside detecting semantic equivalence. We build SCRBench, 67,658 Python program pairs generated by applying sequences of code changes that preserve or alter program semantics, and evaluate five language models on a stratified subset of 4,500 pairs. Detecting semantic equivalence exposes weaker models, whose accuracy declines as changes accumulate, but the strongest models remain near the ceiling. Diagnosing semantic change reveals weaknesses that equivalence accuracy hides: a model with 94.71% equivalence accuracy diagnoses only 54.44% of semantic changes. Reconstructing sequences of code changes is much harder: the best model reconstructs the exact sequence for only 39.27% of pairs, and no model exceeds 1.11% when five changes are applied. High equivalence accuracy therefore overestimates how well models reason about the semantic effects of code changes.
Figures & tables
ID
Abbr.
Type
Effect
1
AOM
Altering
Swaps +↔− and ∗↔/ .
2
BLM
Altering
Negates the first True (else the first False ).
3
LOM
Altering
Swaps and ↔ or .
4
COM
Altering
Flips each comparison’s leading operator ( <↔> , ≤↔≥ , =↔= ).
5
RV
Preserving
Renames the most frequent variable to VAR_0 .
6
DCI_V1
Preserving
Inserts an unused assignment.
Table 1: The nine transformations in the catalog T .
Model
Prompt
Overall
k=1
k=2
k=3
k=4
k=5
Qwen3-235B
Base
88.02%
98.33%
91.78%
88.11%
82.56%
79.33%
Qwen3-235B
Base+CoT
99.60%
100.00%
99.56%
99.56%
99.56%
99.33%
Qwen3-235B
Base+one-shot
85.73%
97.44%
89.67%
87.56%
78.89%
75.11%
Qwen3-235B
Base+CoT+one-shot
99.69%
99.89%
99.22%
99.56%
99.89%
99.89%
GPT-OSS-120B
Base+CoT+one-shot
98.16%
99.22%
98.56%
98.67%
97.22%
97.11%
Mistral-Small-24B
Base+CoT+one-shot
94.71%
98.89%
95.56%
93.56%
93.67%
91.89%
Table 2: T1 accuracy across prompt and model variants.
Model
Prompt
Overall
k=1
k=2
k=3
k=4
k=5
Qwen3-235B
Base
43.51%
93.78%
52.44%
29.11%
24.00%
18.22%
Qwen3-235B
Base+CoT
93.51%
100.00%
92.67%
92.22%
91.33%
91.33%
Qwen3-235B
Base+one-shot
37.29%
96.89%
45.56%
20.44%
15.11%
8.44%
Qwen3-235B
Base+CoT+one-shot
93.47%
100.00%
93.11%
90.67%
91.56%
92.00%
GPT-OSS-120B
Base+CoT+one-shot
97.07%
100.00%
95.33%
96.22%
95.56%
98.22%
Mistral-Small-24B
Base+CoT+one-shot
54.44%
94.00%
58.00%
44.00%
41.33%
34.89%
Table 3: T2 accuracy across prompt and model variants.
Model
AOM
BLM
LOM
COM
Qwen3-235B
93.98%
95.93%
99.77%
86.85%
GPT-OSS-120B
93.57%
99.10%
100.00%
97.65%
Mistral-Small-24B
56.77%
22.17%
62.10%
68.86%
Llama-3.1-8B
28.73%
6.79%
6.85%
37.56%
GPT-4.1-nano
65.25%
43.67%
51.83%
74.65%
Table 4: T2 accuracy by altering transformation using the designated prompt Base+CoT+one-shot.
Model
Prompt
Overall
k=1
k=2
k=3
k=4
k=5
Qwen3-235B
Base
25.42%
78.89%
38.22%
7.78%
2.11%
0.11%
Qwen3-235B
Base+CoT
27.93%
99.33%
32.11%
7.78%
0.33%
0.11%
Qwen3-235B
Base+one-shot
22.47%
75.44%
27.22%
8.56%
1.11%
0.00%
Qwen3-235B
Base+CoT+one-shot
26.82%
99.00%
28.56%
5.89%
0.67%
0.00%
GPT-OSS-120B
Base+CoT+one-shot
39.27%
99.22%
62.56%
26.22%
7.22%
1.11%
Mistral-Small-24B
Base+CoT+one-shot
25.11%
85.11%
35.22%
4.44%
0.78%
0.00%
Table 5: T3 accuracy across prompt and model variants.
Figure 1: Overall accuracy on all tasks for all models, using Base+CoT+one-shot.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
k
Equivalent pairs
Non-equivalent pairs
Total
1
1,315
464
1,779
2
2,958
2,818
5,776
3
4,098
9,889
13,987
4
2,760
20,367
23,127
5
1,379
21,610
22,989
Total
12,510
55,148
67,658
Appendix
Table 6: Number of labeled pairs by transformation sequence length k .