Evaluating Language Models on Cross-Language Code Functional Equivalence

Hui Sun; Anderson Uchôa; Rohit Gheyi; Wesley K. G. Assunção

cs.SEAug 25, 2026

Evaluating Language Models on Cross-Language Code Functional Equivalence

Authors: Hui Sun, Anderson Uchôa, Rohit Gheyi, Wesley K. G. Assunção

Organizations: Federal University of Campina Grande, Campina Grande, Paraíba, Brazil

Abstract

Background: Large Language Models (LLMs) have demonstrated strong performance across a variety of code-understanding tasks, leading many to believe that they can reason about program semantics. However, existing evaluations primarily focus on single-language settings or rely on synthetically generated code, raising concerns about whether current results reflect true semantic understanding. Aims: We investigate whether LLMs can accurately judge functional equivalence across different programming languages in human-written code, a setting that requires deeper reasoning beyond superficial similarity. Method: We introduce PolyHuman, a dataset of human-written programs in CPP, Java, and Python. Using this dataset, we evaluate intra- and inter-language equivalence detection across open-weight and proprietary LLMs, selecting GPT-o4-mini as a representative model to assess stability. We then manually analyze 81 cases of systematic disagreement in which models incorrectly judge functional equivalence, examining the code logic and the generated Chain-of-Thought reasoning. Finally, we categorize these failures and compare them across GPT-o4-mini, Claude-Opus-4.7, and Gemini-3-Flash to determine whether they reflect model-specific issues or broader limitations of state-of-the-art LLMs. Results: We identify a difficulty-dependent breakdown in equivalence judgment (harder problems make the model increasingly prone to misclassifying non-equivalent code as equivalent), a model-specific sensitivity to programming language for the best-performing model (particularly a more conservative behavior on Python), and a partial reliance on similarity-based cues. GPT-o4-mini also shows substantial run-to-run instability under identical settings, indicating inconsistent rather than absent capability. Conclusions: Current LLMs do not reliably capture functional equivalence within or across languages.

Figures & tables

Category	Equivalence	Examples
Intra-language	Equivalent	JAVA_Pass1 vs. JAVA_Pass2; CPP_Pass1 vs. CPP_Pass2; PYTHON3_Pass1 vs. PYTHON3_Pass2
Intra-language	Non-equivalent	JAVA_Pass1 vs. JAVA_Fail; CPP_Pass1 vs. CPP_Fail; PYTHON3_Pass1 vs. PYTHON3_Fail
Inter-language	Equivalent	JAVA_Pass1 vs. PYTHON3_Pass1; JAVA_Pass1 vs. CPP_Pass1; CPP_Pass1 vs. PYTHON3_Pass1
Inter-language	Non-equivalent	JAVA_Fail vs. CPP_Pass1; JAVA_Fail vs. PYTHON3_Pass1; CPP_Fail vs. JAVA_Pass1; CPP_Fail vs. PYTHON3_Pass1; PYTHON3_Fail vs. JAVA_Pass1; PYTHON3_Fail vs. CPP_Pass1

Table 1: Categories of code pairs used to construct the s.

Category	Metric	EquiBench		SeqCoBench	PolyHuman
Category	Metric	OJ-A	OJ-V	Sampled Subset	Each (15 total)
Language		Python	Python	Python	CPP	Java	Python
Pairs	Equal	200	200	250	500	500	500
Pairs	Not Equal	200	200	250	500	500	500
Lines of Code	Min	3	2	2	1	1	1
	Max	3,403	4,087	50	1,233	1,423	1,042

Table 2: Dataset statistics comparison across EquiBench, SeqCoBench, and PolyHuman.

Model	Baselines			PolyHuman (Intra-language)
Model	EquiBench		SeqCoBench	Pass vs Pass			Pass vs Fail
	OJ-V	OJ-A	SeqCoBench	CPP	Java	Python	CPP	Java	Python
GPT-o4-mini	0.91	0.82	0.97	0.83	0.84	0.80	0.83	0.89	0.93
GPT-OSS-20B	0.89	0.74	0.95	0.59	0.62	0.66	0.91	0.94	0.93
Llama-3.1-8B-Instruct	0.54	0.54	0.84	0.98	0.98	0.98	0.05	0.04	0.07
Llama-3.2-3B-Instruct	0.52	0.53	0.66	0.54	0.44	0.78	0.53	0.66	0.28

Table 3: Model accuracy across baseline datasets and PolyHuman intra-language tasks.

Model	Equivalent pairs (Pass–Pass)			Non-equivalent pairs (Pass–Fail)
Model	CPP $\leftrightarrow$ Java	Java $\leftrightarrow$ Python	Python $\leftrightarrow$ CPP	Java(F) $\leftrightarrow$ CPP	Java(F) $\leftrightarrow$ Python	CPP(F) $\leftrightarrow$ Java	CPP(F) $\leftrightarrow$ Python	Python(F) $\leftrightarrow$ Java	Python(F) $\leftrightarrow$ CPP
GPT-o4-mini	0.85	0.82	0.81	0.93	0.90	0.89	0.87	0.92	0.92
GPT-OSS-20B	0.63	0.62	0.64	0.92	0.91	0.88	0.86	0.90	0.92
Llama-3.1-8B-Instruct	1.00	0.99	0.99	0.02	0.03	0.02	0.03	0.02	0.02
Llama-3.2-3B-Instruct	0.51	0.61	0.72	0.56	0.44	0.54	0.35	0.53	0.31
Mistral-7B-Instruct-v0.3	1.00	1.00	1.00	0.01	0.01	0.01	0.01	0.01	0.02

Table 4: Accuracy of functional equivalence detection across inter-language code pairs.

Model	CPP			Java			Python
	AccEq	AccNonEq	Gap	AccEq	AccNonEq	Gap	AccEq	AccNonEq	Gap
GPT-o4-mini	0.830	0.902	0.072	0.835	0.910	0.075	0.815	0.902	0.088
GPT-OSS-20B	0.635	0.895	0.260	0.625	0.902	0.277	0.630	0.898	0.267
Llama-3.1-8B-Instruct	0.995	0.023	-0.972	0.995	0.023	-0.972	0.990	0.025	-0.965
Llama-3.2-3B-Instruct	0.615	0.440	-0.175	0.560	0.518	-0.042	0.665	0.408	-0.258
Mistral-7B-Instruct-v0.3	1.000	0.012	-0.988	1.000	0.010	-0.990	1.000	0.012	-0.988

Table 5: Model-level decision gap across equivalent and non-equivalent inter-language pairs.

Figure 1: GPT-o4-mini’s Yes response rate across problem difficulty levels.

Panel of Figure 2 — Figure 2: GPT-o4-mini’s Yes response rate across code length and similarity bins. Left panels report equivalent pairs (EQ), and right panels report non-equivalent pairs (NEQ).

Predictor	$\beta$	SE	$p$ -value	Odds Ratio
Intercept	1.791	0.110	$<0.001^{***}$	5.993
Difficulty	-0.350	0.067	$<0.001^{***}$	0.705
Difficulty $\times$ NEQ	0.804	0.103	$<0.001^{***}$	2.234
NEQ (vs. EQ)	-3.755	0.115	$<0.001^{***}$	0.023
Java (vs. CPP)	-0.183	0.145	0.208	0.833
Python (vs. CPP)	-0.450	0.134	$0.001^{**}$	0.637

Table 6: Logistic regression of factors associated with the model’s True predictions.

Same language, equal outcome		Same language, different outcome		Different language, equal outcome		Different language, different outcome
Pair	N	Pair	N	Pair	N	Pair	N
Java Pass1 vs. Java Pass2	9	Java Pass1 vs. Java Fail	8	CPP Pass1 vs. Java Pass1	6	Java Fail vs. Python Pass1	9
Python Pass1 vs. Python Pass2	4	CPP Pass1 vs. CPP Fail	6	Python Pass1 vs. CPP Pass1	5	CPP Fail vs. Java Pass1	7
CPP Pass1 vs. CPP Pass2	6	Python Pass1 vs. Python Fail	2	Java Pass1 vs. Python Pass1	5	Java Fail vs. CPP Pass1	6
						CPP Fail vs. Python Pass1	3
						Python Fail vs. Java Pass1	3

Table 7: Pair-type distribution of the 81 manually analyzed instances.

Panel of Figure 3 — Figure 3: Representative examples of knowledge failures.

Panel of Figure 4 — Figure 4: Representative examples of abstraction-level reasoning failure.

Code Language Models

Code language models are AI models that generate, explain, edit, and reason about source code across programming tasks.

See all 114 papers

LLMs are Brittle to Simple Code Transformations: Introducing CETBench - A Benchmark for Code-Equivalence Checking
Jun 4, 2025Neeva Oza, Ishaan Govil, Parul Gupta +3LLM Evaluation Language Model Robustness
Indian Institute of Technology Delhi · IBM Research AI
Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification
May 27, 2026Dylan Bouchard, Mohit Singh Chauhan, Zeya Ahmad +1Language Model Calibration LLM Uncertainty Estimation
CVS Health®, Wellesley, MA, USA
Detecting Functional Memorization in Code Language Models
Jun 11, 2026Matthieu Meeus, Anil Ramakrishna, Matthew Grange +2Memorization in Language Models LLM Auditing
Meta · Imperial College London

Jun 4, 2025J/K move · Enter open · S save

LLMs are Brittle to Simple Code Transformations: Introducing CETBench - A Benchmark for Code-Equivalence Checking

Neeva Oza, Ishaan Govil, Parul Gupta, Dinesh Khandelwal+2

Indian Institute of Technology Delhi · IBM Research AI

LLM Evaluation Language Model Robustness Code Language Models

Open paper AskPDF arXiv

We study how well LLMs can determine whether two programs are functionally equivalent. This is an important problem because benchmarking code equivalence helps assess LLM capability in tasks such as code rewriting and translation. To this end, we introduce CETBench - Code Equivalence with Transformations Benchmark - built from a repository of programs that may solve the same or different tasks. Each dataset instance is created by sampling a program pair and applying a random sequence of predefined code transformations, yielding either equivalent or non-equivalent pairs. Our analysis shows that even simple transformations cause a significant performance drop in state-of-the-art LLMs on code-equivalence checking. These challenges are further amplified in the cross-lingual setting when comparing programs written in different languages. To remedy this, we present a simple fine-tuning-based approach to boost LLM performance on the transformed pairs of programs. Our approach for dataset generation is generic, supporting cross-lingual equivalence checking, the generation of program pairs with varying difficulty levels, and the application of diverse transformations. In our experiments, we perform ablations over the difficulty level of original programs, as well as the kind of transformations used in generating pairs for equivalence checking. Our analysis presents deep insights into the working of LLMs for the task of code-equivalence, and points to the fact that they may still be far from what could be termed as a semantic understanding of the underlying code.

Abstract

Code Language Models

Explore similar work

LLMs are Brittle to Simple Code Transformations: Introducing CETBench - A Benchmark for Code-Equivalence Checking

Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification

Detecting Functional Memorization in Code Language Models