Background: Large Language Models (LLMs) have demonstrated strong performance across a variety of code-understanding tasks, leading many to believe that they can reason about program semantics. However, existing evaluations primarily focus on single-language settings or rely on synthetically generated code, raising concerns about whether current results reflect true semantic understanding. Aims: We investigate whether LLMs can accurately judge functional equivalence across different programming languages in human-written code, a setting that requires deeper reasoning beyond superficial similarity. Method: We introduce PolyHuman, a dataset of human-written programs in CPP, Java, and Python. Using this dataset, we evaluate intra- and inter-language equivalence detection across open-weight and proprietary LLMs, selecting GPT-o4-mini as a representative model to assess stability. We then manually analyze 81 cases of systematic disagreement in which models incorrectly judge functional equivalence, examining the code logic and the generated Chain-of-Thought reasoning. Finally, we categorize these failures and compare them across GPT-o4-mini, Claude-Opus-4.7, and Gemini-3-Flash to determine whether they reflect model-specific issues or broader limitations of state-of-the-art LLMs. Results: We identify a difficulty-dependent breakdown in equivalence judgment (harder problems make the model increasingly prone to misclassifying non-equivalent code as equivalent), a model-specific sensitivity to programming language for the best-performing model (particularly a more conservative behavior on Python), and a partial reliance on similarity-based cues. GPT-o4-mini also shows substantial run-to-run instability under identical settings, indicating inconsistent rather than absent capability. Conclusions: Current LLMs do not reliably capture functional equivalence within or across languages.
Figures & tables
Category
Equivalence
Examples
Intra-language
Equivalent
JAVA_Pass1 vs. JAVA_Pass2; CPP_Pass1 vs. CPP_Pass2; PYTHON3_Pass1 vs. PYTHON3_Pass2
Intra-language
Non-equivalent
JAVA_Pass1 vs. JAVA_Fail; CPP_Pass1 vs. CPP_Fail; PYTHON3_Pass1 vs. PYTHON3_Fail
Inter-language
Equivalent
JAVA_Pass1 vs. PYTHON3_Pass1; JAVA_Pass1 vs. CPP_Pass1; CPP_Pass1 vs. PYTHON3_Pass1
Inter-language
Non-equivalent
JAVA_Fail vs. CPP_Pass1; JAVA_Fail vs. PYTHON3_Pass1; CPP_Fail vs. JAVA_Pass1; CPP_Fail vs. PYTHON3_Pass1; PYTHON3_Fail vs. JAVA_Pass1; PYTHON3_Fail vs. CPP_Pass1
Table 1: Categories of code pairs used to construct the s.
Category
Metric
EquiBench
SeqCoBench
PolyHuman
OJ-A
OJ-V
Sampled Subset
Each (15 total)
Language
Python
Python
Python
CPP
Java
Python
Pairs
Equal
200
200
250
500
500
500
Not Equal
200
200
250
500
500
500
Lines of Code
Min
3
2
2
1
1
1
Max
3,403
4,087
50
1,233
1,423
1,042
Table 2: Dataset statistics comparison across EquiBench, SeqCoBench, and PolyHuman.
Model
Baselines
PolyHuman (Intra-language)
EquiBench
SeqCoBench
Pass vs Pass
Pass vs Fail
OJ-V
OJ-A
CPP
Java
Python
CPP
Java
Python
GPT-o4-mini
0.91
0.82
0.97
0.83
0.84
0.80
0.83
0.89
0.93
GPT-OSS-20B
0.89
0.74
0.95
0.59
0.62
0.66
0.91
0.94
0.93
Llama-3.1-8B-Instruct
0.54
0.54
0.84
0.98
0.98
0.98
0.05
0.04
0.07
Llama-3.2-3B-Instruct
0.52
0.53
0.66
0.54
0.44
0.78
0.53
0.66
0.28
Table 3: Model accuracy across baseline datasets and PolyHuman intra-language tasks.
Model
Equivalent pairs (Pass–Pass)
Non-equivalent pairs (Pass–Fail)
CPP ↔ Java
Java ↔ Python
Python ↔ CPP
Java(F) ↔ CPP
Java(F) ↔ Python
CPP(F) ↔ Java
CPP(F) ↔ Python
Python(F) ↔ Java
Python(F) ↔ CPP
GPT-o4-mini
0.85
0.82
0.81
0.93
0.90
0.89
0.87
0.92
0.92
GPT-OSS-20B
0.63
0.62
0.64
0.92
0.91
0.88
0.86
0.90
0.92
Llama-3.1-8B-Instruct
1.00
0.99
0.99
0.02
0.03
0.02
0.03
0.02
0.02
Llama-3.2-3B-Instruct
0.51
0.61
0.72
0.56
0.44
0.54
0.35
0.53
0.31
Mistral-7B-Instruct-v0.3
1.00
1.00
1.00
0.01
0.01
0.01
0.01
0.01
0.02
Table 4: Accuracy of functional equivalence detection across inter-language code pairs.
Model
CPP
Java
Python
AccEq
AccNonEq
Gap
AccEq
AccNonEq
Gap
AccEq
AccNonEq
Gap
GPT-o4-mini
0.830
0.902
0.072
0.835
0.910
0.075
0.815
0.902
0.088
GPT-OSS-20B
0.635
0.895
0.260
0.625
0.902
0.277
0.630
0.898
0.267
Llama-3.1-8B-Instruct
0.995
0.023
-0.972
0.995
0.023
-0.972
0.990
0.025
-0.965
Llama-3.2-3B-Instruct
0.615
0.440
-0.175
0.560
0.518
-0.042
0.665
0.408
-0.258
Mistral-7B-Instruct-v0.3
1.000
0.012
-0.988
1.000
0.010
-0.990
1.000
0.012
-0.988
Table 5: Model-level decision gap across equivalent and non-equivalent inter-language pairs.
Figure 1: GPT-o4-mini’s Yes response rate across problem difficulty levels.
Figure 2: GPT-o4-mini’s Yes response rate across code length and similarity bins. Left panels report equivalent pairs (EQ), and right panels report non-equivalent pairs (NEQ).
Predictor
β
SE
p -value
Odds Ratio
Intercept
1.791
0.110
<0.001∗∗∗
5.993
Difficulty
-0.350
0.067
<0.001∗∗∗
0.705
Difficulty × NEQ
0.804
0.103
<0.001∗∗∗
2.234
NEQ (vs. EQ)
-3.755
0.115
<0.001∗∗∗
0.023
Java (vs. CPP)
-0.183
0.145
0.208
0.833
Python (vs. CPP)
-0.450
0.134
0.001∗∗
0.637
Table 6: Logistic regression of factors associated with the model’s True predictions.
Same language, equal outcome
Same language, different outcome
Different language, equal outcome
Different language, different outcome
Pair
N
Pair
N
Pair
N
Pair
N
Java Pass1 vs. Java Pass2
9
Java Pass1 vs. Java Fail
8
CPP Pass1 vs. Java Pass1
6
Java Fail vs. Python Pass1
9
Python Pass1 vs. Python Pass2
4
CPP Pass1 vs. CPP Fail
6
Python Pass1 vs. CPP Pass1
5
CPP Fail vs. Java Pass1
7
CPP Pass1 vs. CPP Pass2
6
Python Pass1 vs. Python Fail
2
Java Pass1 vs. Python Pass1
5
Java Fail vs. CPP Pass1
6
CPP Fail vs. Python Pass1
3
Python Fail vs. Java Pass1
3
Table 7: Pair-type distribution of the 81 manually analyzed instances.
Figure 3: Representative examples of knowledge failures.
Figure 4: Representative examples of abstraction-level reasoning failure.
We study how well LLMs can determine whether two programs are functionally equivalent. This is an important problem because benchmarking code equivalence helps assess LLM capability in tasks such as code rewriting and translation. To this end, we introduce CETBench - Code Equivalence with Transformations Benchmark - built from a repository of programs that may solve the same or different tasks. Each dataset instance is created by sampling a program pair and applying a random sequence of predefined code transformations, yielding either equivalent or non-equivalent pairs. Our analysis shows that even simple transformations cause a significant performance drop in state-of-the-art LLMs on code-equivalence checking. These challenges are further amplified in the cross-lingual setting when comparing programs written in different languages. To remedy this, we present a simple fine-tuning-based approach to boost LLM performance on the transformed pairs of programs. Our approach for dataset generation is generic, supporting cross-lingual equivalence checking, the generation of program pairs with varying difficulty levels, and the application of diverse transformations. In our experiments, we perform ablations over the difficulty level of original programs, as well as the kind of transformations used in generating pairs for equivalence checking. Our analysis presents deep insights into the working of LLMs for the task of code-equivalence, and points to the fact that they may still be far from what could be termed as a semantic understanding of the underlying code.
Neeva Oza, Ishaan Govil, Parul Gupta +3
Indian Institute of Technology Delhi · IBM Research AI
Large language models have shown impressive capabilities in code generation, yet they often produce functionally incorrect code. Uncertainty quantification (UQ) methods have emerged as a promising approach for detecting hallucinations in natural language generation, but their effectiveness for code generation tasks remains underexplored. We systematically evaluate how UQ techniques transfer to code generation across three programming languages, five LLMs, and over 1,700 problems. We find that some token-probability-based methods generalize effectively without modification, while sampling-based methods relying on natural language inference (NLI) fail because NLI models cannot distinguish functionally different code, causing most responses to collapse into a single semantic cluster. To address this, we introduce functional equivalence methods, a family of code-specific methods that replace NLI-based semantic equivalence with an LLM-based functional equivalence assessment, including functional entropy, a code-specific analog of semantic entropy. Functional equivalence methods achieve top AUROC in 11 out of 15 model-benchmark combinations and the best calibration across most settings, consistently outperforming both NLI-based counterparts and all other methods evaluated.
Dylan Bouchard, Mohit Singh Chauhan, Zeya Ahmad +1
Large language models (LLMs) are increasingly used to generate code at scale. Meanwhile, prior work has investigated whether training data may be recoverable from model outputs, by auditing the textual overlap between training examples and model generations. Code, however, can be functionally equivalent while textually dissimilar. In this work, we study functional memorization: extraction of functional logic beyond what verbatim metrics detect. We construct a counterfactual setup for Olmo-3-32B, comparing a midtrained model (exposed to target code) against a pretrained reference (not exposed). We prompt both models with Python function signatures and measure both textual and functional similarity (i.e., LLM-as-a-judge, execution-based). Our results show clear evidence of functional memorization, highlighting the need for auditing metrics that go beyond textual overlap.
Matthieu Meeus, Anil Ramakrishna, Matthew Grange +2