LLMs are Brittle to Simple Code Transformations: Introducing CETBench - A Benchmark for Code-Equivalence Checking

Neeva Oza; Ishaan Govil; Parul Gupta; Dinesh Khandelwal; Dinesh Garg; Parag Singla

cs.SEJun 4, 2025

LLMs are Brittle to Simple Code Transformations: Introducing CETBench - A Benchmark for Code-Equivalence Checking

Authors: Neeva Oza, Ishaan Govil, Parul Gupta, Dinesh Khandelwal, Dinesh Garg, Parag Singla

Organizations: Indian Institute of Technology Delhi · IBM Research AI

Abstract

We study how well LLMs can determine whether two programs are functionally equivalent. This is an important problem because benchmarking code equivalence helps assess LLM capability in tasks such as code rewriting and translation. To this end, we introduce CETBench - Code Equivalence with Transformations Benchmark - built from a repository of programs that may solve the same or different tasks. Each dataset instance is created by sampling a program pair and applying a random sequence of predefined code transformations, yielding either equivalent or non-equivalent pairs. Our analysis shows that even simple transformations cause a significant performance drop in state-of-the-art LLMs on code-equivalence checking. These challenges are further amplified in the cross-lingual setting when comparing programs written in different languages. To remedy this, we present a simple fine-tuning-based approach to boost LLM performance on the transformed pairs of programs. Our approach for dataset generation is generic, supporting cross-lingual equivalence checking, the generation of program pairs with varying difficulty levels, and the application of diverse transformations. In our experiments, we perform ablations over the difficulty level of original programs, as well as the kind of transformations used in generating pairs for equivalence checking. Our analysis presents deep insights into the working of LLMs for the task of code-equivalence, and points to the fact that they may still be far from what could be termed as a semantic understanding of the underlying code.

Figures & tables

Perturbation Type	Semantic-Preserving Variant	Semantic-Non-Preserving Variant
If-Else Swapping	Negates the if condition and swaps the if / else blocks.	Negates the if condition but does not swap the blocks.
For-While Swapping	Correctly converts a for loop to an equivalent while loop.	Converts to a while loop with an intentional error (e.g., faulty iterator update).
If Condition Flipping	Rewrites the if condition to a logically equivalent form (e.g., $a<b$ to not $(a\geq b)$ ).	Inverts the relational operator in the condition (e.g., $a<b$ to $a\geq b$ ).
Variable Renaming	Consistently renames a variable across its entire scope.	Renames a variable inconsistently, breaking its semantic link in one or more places.
Boolean Inversion	Inverts a boolean variable’s declaration and all its subsequent uses.	Inverts the variable’s declaration but leaves its subsequent uses unchanged.
Statement Reordering	Swaps two statements that are data-independent.	Swaps two data-dependent statements, altering the execution logic.

Table 1: Summary of Perturbation Types. Each type has a semantic-preserving (SP) variant that maintains program logic and a semantic-non-preserving (NP) variant that intentionally breaks it.

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
Qwen	100.0 99.0	80.7 78.7	44.0 42.9	100.0 100.0	80.6	80.6	81.2	81.2	72.5	66.8
Qwen	30.5 98.0	39.7 76.9	61.4 47.6	99.5 100.0	69.2	69.2	78.7	78.7	69.2	78.7
Mistral	100.0 100.0	90.3 90.3	25.2 24.7	86.5 89.5	76.3	76.3	82.6	65.1	71.7	58.4
Mistral	0.2 98.0	16.5 86.5	33.0 27.8	90.0 89.0	56.6	68.3	64.4	49.9	44.8	54.9
DSC	100.0 100.0	100.0 99.9	0.1 0.3	2.5 2.0	50.6	50.6	51.1	32.5	63.1	1.9

Table 2: Results comparing performance of various models before and after perturbations. First 10 columns show accuracy values. W-Avg is average accuracy after re-weighing examples to have equal weight on positively and negatively labelled examples. Macro F1, Micro F1, and F1 scores of each individual labels are also reported.

Table 3: Comparing the performance of our Benchmark to existing Benchmarks

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
ASTNN	100.0 100.0	100.0 100.0	0.0 0.0	0.0 0.0	50.0	50.0	50.0	31.6	63.2	0.0
	47.5 59.5	43.3 56.7	43.8 43.1	43.0 39.0	47.0	50.7	63.6	44.1	33.0	55.3

Table 4: Results comparing performance of ASTNN baseline.

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
Qwen	100.0 98.5	56.8 57.7	54.5 57.5	99.0 100.0	79.6	79.6	87.3	69.0	67.6	70.4
Qwen	24.1 85.7	43.3 58.9	63.5 59.6	99.5 99.5	66.7	68.6	77.2	58.0	77.2	43.9
Mistral	100.0 98.9	56.8 57.7	63.7 64.5	96.0 93.0	78.8	78.8	86.9	68.4	65.5	71.5
Mistral	1.57 94.9	39.6 59.6	59.0 60.2	93.0 89.5	62.1	67.2	71.5	55.9	42.8	68.9
DSC	98.9 98.7	87.2 87.2	23.8 24.0	89.0 86.0	74.3	74.3	81.1	62.4	69.4	55.5

Table 5: Results comparing various models using COT prompt

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
GPT-4o	100.0 100.0	81.0 78.0	67.0 67.0	100.0 100.0	86.6	86.6	92.2	89.0	87.0	90.1
	75.5 94.0	65.5 80.5	90.5 71.0	100.0 100.0	84.7	85.6	91.2	83.1	74.1	92.0

Table 6: Results on GPT-4o on combinations of all other perturbations except variable-renaming

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
Qwen	98.0 97.0	81.0 81.5	46.3 45.1	99.5 99.0	80.9	80.9	87.6	71.5	73.1	69.8
Qwen	82.7 96.5	82.2 90.1	85.3 39.4	99.5 97.5	84.2	87.3	90.1	75.8	66.9	84.8
Mistral	100.0 99.5	63.6 63.7	61.5 59.3	99.0 98.5	80.6	80.6	88.0	65.7	87.7	64.7
Mistral	77.0 87.5	76.7 67.6	87.5 55.6	100.0 98.0	81.2	80.0	88.9	71.2	58.5	83.9
DSC	100.0 100.0	84.1 83.9	42.6 41.0	99.5 100.0	81.3	81.3	87.5	71.2	73.6	68.7

Table 7: Effect of finetuning

Model	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp
Qwen	87.5 87.8	33.4 33.9	100.0 99.5	73.7	77.2	81.2	68.3	72.1	64.4
Qwen	46.7 82.8	65.7 41.3	100.0 100.0	72.8	75.0	82.0	63.0	52.8	73.2
Mistral	46.7 82.8	63.7 64.5	96.0 93.0	78.8	78.8	86.9	68.4	65.5	71.5
Mistral	39.6 59.6	59.0 60.2	93.0 89.5	62.1	67.8	71.5	55.9	64.0	68.9
DSC	99.7 99.3	0.9 0.8	3.5 2.5	34.5	50.7	35.8	33.1	63.3	2.9

Table 8: Results comparing various models for pairs with one Java code and one Python code

Model	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp
Qwen	81.6 84.4	37.9 40.9	100.0 100.0	73.9	76.0	82.3	67.7	68.3	67.2
Qwen	39.6 82.1	60.9 42.1	100.0 100.0	70.8	75.3	80.3	60.0	48.3	71.7
Mistral	93.6 93.4	12.6 12.9	72.0 73.0	59.6	68.1	67.8	54.7	64.7	44.8
Mistral	13.3 91.7	22.2 15.3	78.0 73.5	49.0	66.1	59.1	42.2	39.8	44.6
DSC	99.3 99.2	1.4 0.6	8.0 19.0	37.9	53.3	41.8	34.5	60.2	8.9

Table 9: Results comparing various models for pairs with one Java code and one Python code (Both Perturbed)

Table 10: Testing finetuned model across difficulty levels and cross-lingual setting. B : Before, A : After Finetuning

Appendix figures & tables51 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 1 — Figure 1: Example of perturbation on programs.

Original Category	Sample Count	Perturbed Category	Sample Count
id	400	$\langle\textit{id},k,\text{np}\rangle$	200
id	400	$\langle\textit{id},k,\text{sp}\rangle$	200
fe	2000	$\langle\textit{fe},k,\text{np}\rangle$	1000
fe	2000	$\langle\textit{fe},k,\text{sp}\rangle$	1000
ne	2000	$\langle\textit{ne},k,\text{np}\rangle$	1000
ne	2000	$\langle\textit{ne},k,\text{sp}\rangle$	1000

Appendix

Table 11: Sample Counts of original and perturbed program-pairs

Panel of Figure 2 — Figure 2: Change in Average Accuracy with varying Difficulty Level for different Models

Panel of Figure 3 — Figure 3: Change in Average Accuracy with applying individual Perturbation Types for different Models

Panel of Figure 4 — Figure 4: Change in Weighted Accuracy with varying Difficulty Level for different Models

Panel of Figure 5 — Figure 5: Change in Weighted Accuracy with applying individual Perturbation Types for different Models

Panel of Figure 6 — Figure 6: Change in F1 score with varying Difficulty Level for different Models

Panel of Figure 7 — Figure 7: Change in F1 score with applying individual Perturbation Types for different Models

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
Qwen	100.0 99.0	82.5 82.0	39.7 38.5	100.0 100.0	80.2	80.2	86.6	69.4	72.0	66.8
Qwen	28.3 97.4	35.4 79.7	57.0 43.7	100.0 100.0	67.7	74.6	77.2	59.5	49.9	69.1
Mistral	100.0 99.0	89.5 89.7	26.0 26.2	89.0 90.0	76.2	76.2	82.5	64.7	71.1	58.3
Mistral	2.1 97.7	17.8 85.6	57.7 35.0	93.5 91.2	56.6	68.3	64.4	49.9	44.9	54.9
DSC	100.0 98.8	95.8 97.4	5.8 6.3	20.0 16.0	55.0	55.0	60.0	42.3	67.7	16.9

Appendix

Table 12: Results comparing performance of various models using self-consistency decoding method.

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
900	100.0 99.0	80.7 78.7	44.0 42.9	100.0 100.0	80.6	80.6	81.2	81.2	72.5	66.8
900	30.5 98.0	39.7 76.9	61.4 47.6	99.5 100.0	69.2	69.2	78.7	78.7	69.2	78.7
1200	100.0 100.0	81.4 80.9	24.2 22.2	100.0 100.0	76.1	76.1	81.8	62.5	68.1	57.0
1200	26.0 95.0	32.5 77.3	42.1 28.4	100.0 100.0	62.6	70.5	72.3	53.2	45.0	61.5
1600	100.0 100.0	57.4 60.3	53.7 53.7	100.0 100.0	78.1	78.1	86.0	66.1	64.1	68.2

Appendix

Table 13: Effect of varying difficulty level on Qwen model.

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
900	100.0 100.0	90.3 90.3	25.2 24.7	86.5 89.5	76.3	76.3	82.6	65.1	71.7	58.4
900	0.2 98.0	16.5 86.5	33.0 27.8	90.0 89.0	56.6	68.3	64.4	49.9	44.8	54.9
1200	100.0 100.0	95.5 96.5	10.7 8.6	61.0 63.5	66.9	66.9	73.1	53.5	68.1	38.8
1200	1.5 99.0	4.5 95.1	16.2 12.9	74.0 67.5	46.3	63.2	53.1	39.4	42.2	36.5
1600	98.9 98.4	96.1 96.0	10.8 11.1	58.5 54.0	65.5	65.5	72.3	52.6	67.5	37.7

Appendix

Table 14: Effect of varying difficulty level on Mistral model.

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
900	100.0 100.0	100.0 99.9	0.1 0.3	2.5 2.0	50.6	50.6	51.1	32.5	63.1	1.9
900	0.0 99.5	0.0 99.9	1.2 0.3	2.5 2.5	26.3	50.6	27.8	20.2	37.6	2.8
1200	100.0 99.5	99.8 99.7	0.1 0.4	9.0 10.0	52.3	52.3	54.4	34.8	63.9	5.6
1200	1.0 100.0	1.2 99.9	1.0 0.6	12.5 10.5	28.3	52.2	31.1	22.1	38.2	5.9
1600	98.9 98.4	99.6 99.7	0.7 0.3	3.0 5.0	50.7	50.7	52.0	33.1	63.2	3.0

Appendix

Table 15: Effect of varying difficulty level on Deepseek-Coder model.

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
900	99.5 100.0	72.5 73.0	67.7 66.7	100.0 100.0	84.9	84.9	91.1	76.8	75.5	78.1
900	81.0 97.0	77.4 71.8	91.8 72.2	100.0 100.0	86.4	85.7	92.3	77.3	66.66	88.1
1200	100.0 99.5	40.2 40.3	65.2 63.9	100.0 100.0	76.6	76.6	81.1	59.1	44.9	73.3
1200	78.5 92.5	86.2 38.4	87.7 68.6	100.0 100.0	81.4	76.1	88.4	66.0	47.7	84.4
1600	100.0 99.5	17.4 18.9	87.9 88.9	100.0 100.0	76.6	76.6	81.1	59.1	44.9	73.3

Appendix

Table 16: Effect of varying difficulty level on GPT-4o model.

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
boolinv	100.0 99.5	77.3 76.3	51.3 56.9	99.5 100.0	82.6	82.6	89.2	73.4	73.6	73.1
boolinv	30.5 81.0	28.9 75.2	58.0 59.7	100.0 100.0	66.7	70.5	76.9	59.3	48.3	70.4
ifelse	99.5 100.0	80.2 78.0	48.1 44.2	100.0 100.0	80.4	80.4	87.1	69.9	71.7	68.1
ifelse	23.2 92.5	24.9 78.4	46.1 47.6	100.0 100.0	64.1	71.2	73.6	55.8	46.8	64.9
ifflip	99.5 100.0	80.5 79.8	42.8 41.8	100.0 100.0	80.5	80.5	87.1	70.0	72.0	68.0

Appendix

Table 17: Effect of different perturbations on Qwen model.

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
boolinv	99.5 99.0	81.9 82.5	36.7 38.7	84.5 88.0	76.4	76.4	84.3	66.6	70.4	62.7
boolinv	1.5 96.0	20.1 80.0	37.6 37.3	88.5 89.0	56.2	66.8	65.0	50.2	43.8	56.6
ifelse	99.4 100.0	88.1 88.3	25.6 27.7	90.0 89.5	76.0	76.0	82.6	65.0	71.7	58.3
ifelse	0.5 94.0	14.9 86.9	27.6 31.1	93.5 94.5	55.9	67.4	63.7	48.7	44.3	53.1
ifflip	99.5 100.0	87.9 89.0	27.7 25.1	88.5 91.5	76.1	76.1	82.6	64.6	70.8	58.3

Appendix

Table 18: Effect of different perturbations on Mistral model.

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
boolinv	100.0 99.5	99.7 99.3	1.9 1.3	5.0 9.0	52.0	52.0	54.0	34.9	63.7	6.1
boolinv	0.0 99.5	0.2 99.6	1.1 0.8	5.5 4.5	26.4	50.8	27.8	20.4	37.7	3.0
ifelse	99.5 100.0	99.7 99.8	0.6 0.5	3.0 2.5	50.7	50.7	51.5	32.8	63.2	2.3
ifelse	1.0 100.0	0.2 99.7	0.6 0.3	3.5 1.5	25.8	50.5	26.7	19.6	37.5	1.6
ifflip	99.5 100.0	99.8 99.9	0.6 0.4	2.5 4.0	50.8	50.8	51.8	32.9	63.3	2.5

Appendix

Table 19: Effect of different perturbations on Deepseek-Coder model.

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
boolinv	100.0 100.0	72.4 71.8	80.4 77.2	100.0 100.0	87.7	87.7	93.0	80.9	78.9	82.9
boolinv	79.0 61.0	52.5 71.9	86.2 78.5	100.0 100.0	78.6	74.6	87.1	70.7	57.8	83.6
ifelse	100.0 100.0	75.9 75.6	66.1 69.6	100.0 100.0	85.9	85.9	91.7	78.2	77.2	79.3
ifelse	60.1 95.0	49.3 73.8	79.5 72.8	100.0 100.0	78.8	80.7	87.0	69.8	58.1	81.5
ifflip	100.0 100.0	73.8 72.6	70.1 70.1	99.5 100.0	85.8	85.8	91.7	78.0	76.6	79.5

Appendix

Table 20: Effect of different perturbations on GPT-4o model.

Model	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
Qwen	81.6 82.6	37.4 38.0	100.0 100.0	73.3	74.8	81.6	67.6	70.1	65.0
Qwen	52.5 74.9	70.9 47.8	100.0 100.0	74.3	72.2	83.7	51.6	75.6	63.6
Mistral	96.3 96.7	11.1 13.0	66.5 70.0	58.9	68.1	66.9	55.8	68.8	42.9
Mistral	0.8 95.6	17.9 14.5	76.5 77.0	48.2	64.6	57.0	41.9	43.2	40.6
DSC	99.9 99.6	0.9 0.6	3.0 4.0	34.7	50.9	36.0	33.2	63.4	3.0

Appendix

Table 21: Results comparing various models for C++ and Python pairs of code

Model	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp
Qwen	79.3 83.1	44.1 50.2	100.0 100.0	76.1	77.4	84.6	71.1	71.8	70.3
Qwen	35.2 82.2	58.7 50.5	100.0 100.0	71.1	75.5	80.6	61.3	50.2	72.4
Mistral	94.1 96.1	10.9 11.8	82.0 88.0	63.7	71.6	69.9	58.5	69.0	48.0
Mistral	9.7 94.3	15.9 14.4	79.8 81.3	49.2	67.3	57.6	42.8	41.8	43.8
DSC	100.0 99.9	0.1 0.0	8.2 12.1	36.7	52.5	39.4	34.5	63.2	5.8

Appendix

Table 22: Results comparing various models for pairs with one C++ code and one Python code (Both Perturbed)

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
Qwen	100.0 99.0	78.8 78.4	44.0 41.5	100.0 100.0	80.2	80.2	86.9	69.6	71.4	67.7
Qwen	13.5 99.0	38.1 74.4	60.0 47.8	100.0 100.0	66.6	73.3	75.4	59.4	48.8	69.9
Mistral	100.0 99.0	68.4 70.4	50.7 47.9	90.0 93.5	77.5	77.5	85.8	67.4	67.8	67.0
Mistral	15.5 86.5	46.5 58.8	65.6 60.7	95.5 98.5	66.0	68.1	76.1	58.6	44.4	72.7
DSC	100.0 98.5	99.8 99.6	0.4 0.3	0.5 0.5	49.9	50.0	50.3	31.9	63.1	0.8

Appendix

Table 23: Results of performance after disclosing information about perturbation to the prompts.

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
GPT-5-nano	100.0 100.0	58.3 64.7	91.1 90.5	100.0 100.0	88.1	88.1	92.8	81.0	77.4	84.6
GPT-5-nano	90.0 98.5	85.9 62.3	97.0 90.5	100.0 100.0	90.5	87.1	94.6	81.4	71.1	91.8
GPT-5.1	100.0 100.0	61.2 63.9	93.8 93.7	100.0 100.0	89.1	89.1	93.4	82.5	79.1	86.0
	93.0 99.5	91.8 62.5	98.8 93.8	100.0 100.0	92.4	88.6	95.6	84.2	75.1	93.3

Appendix

Table 24: Results comparing GPT-5 models using COT prompt

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
GPT-5-nano	99.0 100.0	75.0 74.5	94.0 93.5	100.0 100.0	92.0	92.0	95.5	93.2	91.6	94.8
GPT-5-nano	89.5 98.5	85.0 69.5	97.5 95.0	100.0 100.0	91.9	89.2	95.5	90.0	83.8	95.9
GPT-5.1	100.0 99.5	62.5 54.0	92.0 89.5	100.0 100.0	87.2	87.2	92.2	89.0	86.0	91.9
	90.5 91.5	87.0 53.0	99.5 92.5	100.0 100.0	89.2	83.6	93.6	85.9	77.1	94.7

Appendix

Table 25: Results of GPT-5 models on combinations of all other perturbations except variable-renaming

Model	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp
GPT-5-nano	73.8 74.2	89.5 90.7	100.0 100.0	88.0	84.5	93.3	83.8	81.2	86.4
GPT-5-nano	79.4 72.2	96.1 91.1	100.0 100.0	89.8	82.8	94.3	81.1	71.0	91.1
GPT-5.1	51.1 49.7	91.4 92.6	100.0 100.0	80.8	73.2	87.6	72.0	64.1	79.6
	93.2 49.0	98.7 92.4	100.0 100.0	89.0	72.9	92.9	75.4	60.0	90.7

Appendix

Table 26: Results comparing GPT-5 models for pairs with one Java code and one Python code

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
Qwen	100.0 100.0	80.5 82.2	45.7 44.8	100.0 99.0	81.5	81.5	87.9	71.6	73.4	69.8
Qwen	76.0 96.5	68.8 80.6	57.2 45.6	99.5 99.5	78.0	78.0	86.3	70.4	70.5	70.4
Mistral	100.0 98.5	56.2 57.1	56.2 55.0	94.0 94.0	76.4	76.4	85.1	65.1	62.9	67.4
Mistral	76.5 81.5	51.4 51.6	60.1 61.3	94.0 94.5	71.4	71.4	82.2	63.2	58.8	67.6
DSC	100.0 99.0	99.9 99.9	1.4 2.7	96.5 93.0	74.0	74.0	75.2	57.4	70.4	44.4

Appendix

Table 27: Results comparing performance of models trained only using unperturbed dataset.

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
Qwen	96.5 97.5	77.1 77.2	47.7 46.6	100.0 99.5	80.3	80.3	87.4	70.6	71.6	69.5
Qwen	87.0 93.0	80.1 84.2	85.8 41.5	100.0 98.0	83.7	85.3	90.0	74.4	64.5	84.3
Mistral	44.5 42.0	66.9 68.7	44.0 42.8	86.5 87.0	60.3	60.3	73.6	59.4	59.2	59.7
Mistral	76.0 70.5	60.0 73.2	65.1 35.4	93.5 80.0	69.2	70.1	80.7	60.3	48.2	72.3
DSC	89.5 82.6	85.8 85.5	35.2 34.8	97.5 98.0	76.1	76.1	83.9	67.6	71.1	64.1

Appendix

Table 28: Results comparing performance of models trained only using perturbed dataset.

Model	id	fe	di
Qwen	100.0	83.0	99.8
Mistral	100.0	87.8	96.0
DSC	100.0	100.0	2.0

Appendix

Table 29: Results on CLCDSA – A (Easy level problems) dataset

Model	id	fe	di
Qwen	100.0	83.4	99.0
Mistral	100.0	92.3	69.0
DSC	99.0	99.2	8.0

Appendix

Table 30: Results on CLCDSA – D (Difficult level problems) dataset

Model	id	fe	di
Qwen	100.0	81.2	100.0
Mistral	100.0	90.8	83.8
DSC	100.0	100.0	7.5

Appendix

Table 31: Results on CodeNet dataset

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
Qwen	33.5 79.0	56.6 37.8	68.7 68.1	85.8 91.5	65.1	62.9	77.0	54.0	34.4	73.7
Mistral	5.5 90.5	14.7 85.4	28.7 24.9	76.0 76.0	50.2	63.0	60.0	44.8	42.0	47.6
DSC	25.5 44.0	14.3 36.9	18.9 17.5	33.5 43.5	29.3	33.0	44.1	25.0	19.1	30.9
GPT-4o	66.0 96.5	50.3 74.8	72.2 63.1	100.0 100.0	77.9	80.4	86.4	67.6	56.1	79.1

Appendix

Table 32: Results on perturbed set after disclosing information about perturbation to the COT prompt

Model	id	fe	ne	di	Avg	W-Avg	Mac-F1	Mic-F1	PosF1	NegF1
	np sp	np sp	np sp	np sp
Qwen	100.0 98.5	100.0 100.0	0.0 0.1	0.0 0.5	49.9	49.9	50.0	31.7	63.1	0.2
Qwen	8.5 98.5	6.7 100.0	8.0 0.1	6.0 0.5	28.5	52.1	31.8	23.8	38.6	9.1
Mistral	99.5 99.0	99.4 99.3	0.3 1.0	0.0 0.5	49.9	49.9	50.3	32.0	63.0	1.1
Mistral	0.0 99.5	0.2 99.3	0.5 1.1	1.0 1.0	25.3	50.0	25.9	19.4	37.4	1.3
DSC	99.5 98.0	99.7 99.9	0.1 0.4	0.0 0.5	49.8	49.8	50.1	31.7	62.9	0.5

Appendix

Table 33: Few shot prompt

Model	id	fe	ne	di
Unperturbed	0	4.77e-3	4.92e-2	0
NP Perturbed	4.02e-2	9.32e-2	5.66e-2	3.54e-3
SP Perturbed	6.86e-3	5.02e-2	6.78e-2	0

Appendix

Table 34: Variance on changing original pairs

Model	id	fe	ne	di
NP Perturbed	8.92e-4	8.33e-4	4.35e-4	9.17e-5
SP Perturbed	3.22e-4	1.22e-4	1.21e-3	4.92e-4

Appendix

Table 35: Variance on changing perturbed pairs

Perturbation type	error % in sp	error % in np
If-else swapping	3.86	11.68
For-while swapping	4.64	5.529
If condition flipping	0.12	4.37
Variable renaming	5.35	9.86
Boolean variable inversion	2.10	0.55
Statement reordering	1.93	8.99

Appendix

Table 36: Error bounds in dataset generation

Panel of Figure 8 — Figure 8: Sample pairs (Program -1 ).

Panel of Figure 9 — Figure 9: Sample pairs (Program -2 ).

Figure 10: Sample generation of Mistral on original pair

Figure 11: Sample generation of Mistral on SP perturbed pair

Figure 12: Sample generation of Mistral on NP perturbed pair

Panel of Figure 13 — Figure 13: Error Analysis of GPT-4o’s CoT: < id, k, np > pair - second if statement has the counters of c and d swapped

Figure 14: Error Analysis of GPT-4o’s CoT: < id, k, np > pair - The response misses the perturbation.

Panel of Figure 15 — Figure 15: Error Analysis of GPT-4o’s CoT: < fe, k, np > pair - the break statements in while loops in Program 2 make it exit after processing just 2 rows.

Figure 16: Error Analysis of GPT-4o’s CoT: < fe, k, np > pair - The response incorrectly says that the loop continues untill the end of ‘sushiRow’.

Panel of Figure 17 — Figure 17: Error Analysis of GPT-4o’s CoT: < id, k, sp > pair - Semantically equivalent if statements

Figure 18: Error Analysis of GPT-4o’s CoT: < id, k, sp > pair - The response misses the semantic equivalence of lines with different syntax.

Panel of Figure 19 — Figure 19: Error Analysis of GPT-4o’s CoT: < fe, k, sp > pair - Two different programs performing the same operation.

Figure 20: Error Analysis of GPT-4o’s CoT: < fe, k, sp > pair.

Qwen’s Generation	id	fe	ne	di
	np sp	np sp	np sp	np sp
Original: Yes, Perturbed: Yes	73.4 82.1	36.5 36.6	15.4 16.8	0.0 0.0
Original: Yes, Perturbed: No	26.0 14.3	31.1 25.7	29.2 24.4	0.5 1.5
Original: No, Perturbed: Yes	0.4 3.6	18.8 23.1	20.4 23.0	0.5 0.5
Original: No, Perturbed: No	0.0 0.0	13.6 14.6	35.0 35.8	98.9 98.0

Appendix

Table 37: Analysis of percentage of answers changed or unchanged by Qwen on the program pair before and after perturbation.

Mistral’s Generation	id	fe	ne	di
	np sp	np sp	np sp	np sp
Original: Yes, Perturbed: Yes	98.5 94.5	38.7 42.4	19.2 18.7	0.1 0.0
Original: Yes, Perturbed: No	1.5 4.5	18.6 16.3	16.3 15.6	3.0 2.6
Original: No, Perturbed: Yes	0.5 1.5	19.6 19.1	20.5 20.7	6.0 9.9
Original: No, Perturbed: No	0.0 0.5	22.9 22.2	43.9 45.0	90.0 87.4

Appendix

Table 38: Analysis of percentage of answers changed or unchanged by Mistral on the program pair before and after perturbation.

DSC’s Generation	id	fe	ne	di
	np sp	np sp	np sp	np sp
Original: Yes, Perturbed: Yes	83.0 92.0	73.3 78.5	59.9 63.5	2.5 3.5
Original: Yes, Perturbed: No

Appendix

Table 61

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Evaluating Language Models on Cross-Language Code Functional Equivalence
Aug 25, 2026Hui Sun, Anderson Uchôa, Rohit Gheyi +1Code Language Models LLM Evaluation
Federal University of Campina Grande, Campina Grande, Paraíba, Brazil
Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
Jun 18, 2026Maria Ivanova, Pavel Zadorozhny, Rodion Levichev +5Code Generation Code Generation Evaluation
Yandex School of Data Analysis, Applied AI Institute · GigaCode
Beyond BLEU: A Semantic Evaluation Method for Code Translation
May 6, 2026Julius Näumann, Sven Keidel, Amir Molzam Sharifloo +1Code Translation Code Generation Evaluation
∗TU Darmstadt · †Hessian Center for Artificial Intelligence, Darmstadt, Germany · ‡National Research Center for Applied Cybersecurity ATHENE

Aug 25, 2026J/K move · Enter open · S save

Evaluating Language Models on Cross-Language Code Functional Equivalence

Hui Sun, Anderson Uchôa, Rohit Gheyi, Wesley K. G. Assunção

Federal University of Campina Grande, Campina Grande, Paraíba, Brazil

Code Language Models LLM Evaluation

Open paper AskPDF arXiv

Background: Large Language Models (LLMs) have demonstrated strong performance across a variety of code-understanding tasks, leading many to believe that they can reason about program semantics. However, existing evaluations primarily focus on single-language settings or rely on synthetically generated code, raising concerns about whether current results reflect true semantic understanding. Aims: We investigate whether LLMs can accurately judge functional equivalence across different programming languages in human-written code, a setting that requires deeper reasoning beyond superficial similarity. Method: We introduce PolyHuman, a dataset of human-written programs in CPP, Java, and Python. Using this dataset, we evaluate intra- and inter-language equivalence detection across open-weight and proprietary LLMs, selecting GPT-o4-mini as a representative model to assess stability. We then manually analyze 81 cases of systematic disagreement in which models incorrectly judge functional equivalence, examining the code logic and the generated Chain-of-Thought reasoning. Finally, we categorize these failures and compare them across GPT-o4-mini, Claude-Opus-4.7, and Gemini-3-Flash to determine whether they reflect model-specific issues or broader limitations of state-of-the-art LLMs. Results: We identify a difficulty-dependent breakdown in equivalence judgment (harder problems make the model increasingly prone to misclassifying non-equivalent code as equivalent), a model-specific sensitivity to programming language for the best-performing model (particularly a more conservative behavior on Python), and a partial reliance on similarity-based cues. GPT-o4-mini also shows substantial run-to-run instability under identical settings, indicating inconsistent rather than absent capability. Conclusions: Current LLMs do not reliably capture functional equivalence within or across languages.

Abstract

LLM Evaluation

Explore similar work

Evaluating Language Models on Cross-Language Code Functional Equivalence

Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

Beyond BLEU: A Semantic Evaluation Method for Code Translation