Large language models achieve near-ceiling performance on code generation benchmarks, yet most of the programming languages used by popular benchmarks such as SWE-bench and HumanEval (e.g. Python, JavaScript) are squarely in-distribution. They appear at scale in pre-training corpora and are heavily reinforced during post-training. To study LLM performance on unfamiliar programming languages, we introduce EsoLang-Bench, a benchmark using five esoteric programming languages (Brainfuck, Befunge-98, Whitespace, Unlambda, and Shakespeare). All five of our chosen esoteric languages are Turing-complete, so the same algorithmic problems that are solvable in Python or JavaScript are in principle solvable in each of them. Yet, they are unfamiliar to LLMs which makes them a good proxy for evaluating out-of-distribution performance. The unfamiliarity of esoteric languages comprises of: (i) the hard-by-design primitives comprising the language; (ii) substantially less representation in pre-training corpora (340x to over 60,000x fewer public GitHub repositories than Python); (iii) negligible deployment value, which makes targeted inclusion in post-training data economically irrational. We evaluate five frontier models across five prompting strategies and find a dramatic capability gap. The same 80 problems expressed in Python or JavaScript reach 100% accuracy on top frontier models, while the equivalent esoteric versions score only 0-11%. Few-shot learning and self-reflection also fail to close this gap. EsoLang-Bench therefore provides a contamination-resistant testbed for measuring how well frontier models generalise algorithmic problem-solving to programming languages outside their training distribution.
Figures & tables
Figure 1: EsoLang-Bench at a glance. Top: the same Hello World written in Python and in each of the five esoteric languages targeted by the benchmark. Bottom: the dataset (80 problems per language, 4 difficulty tiers, 5 esoteric languages) and the evaluation grid (5 frontier models × 5 prompting strategies).
Figure 2: Per-language GitHub topic-tag counts on a log10 scale. Bar labels show the actual repository counts. Methodology and per-language details in Appendix I .
Brainfuck
Befunge-98
Whitespace
Unlambda
Shakespeare
Model
0-S
3-S
0-S
3-S
0-S
3-S
0-S
3-S
0-S
3-S
GPT-5.4 xhigh
2.5%
2.5%
2.5%
8.8%
0%
0%
0%
0%
2.5%
1.2%
O4-mini
2.5%
3.8%
6.2%
7.5%
0%
0%
0%
0%
1.2%
1.2%
Gemini 3.1 Pro
2.5%
3.8%
5.0%
3.8%
0%
0%
0%
0%
1.2%
1.2%
Qwen-235B
2.5%
1.2%
0%
0%
0%
0%
0%
1.2%
0%
0%
Kimi K2.5
0%
0%
2.5%
1.2%
0%
0%
0%
0%
1.2%
1.2%
Table 1: Zero-shot and few-shot (3 ICL examples) accuracy (%) across all models and languages. Each language contains 80 problems (20 per difficulty tier). Only Easy-tier problems were solved (all Medium/Hard/Extra-Hard = 0%). Best results per language in bold . 0-S = Zero-Shot, 3-S = Three-Shot Few-Shot.
Brainfuck
Befunge-98
Whitespace
Unlambda
Shakespeare
Model
S-S
TSS
Re
S-S
TSS
Re
S-S
TSS
Re
S-S
TSS
Re
S-S
TSS
Re
GPT-5.4 xhigh
6.2
3.8
5.0
11.2
10.0
8.8
0
0
0
1.2
0
0
2.5
2.5
1.2
O4-mini
5.0
2.5
3.8
10.0
6.2
7.5
0
0
0
0
0
0
1.2
1.2
0
Gemini
5.0
3.8
3.8
7.5
6.2
7.5
0
0
0
0
0
0
1.2
0
0
Qwen-235B
2.5
1.2
2.5
0
0
0
0
0
0
1.2
0
0
1.2
0
0
Kimi K2.5
0
0
0
0
0
0
0
0
0
0
0
0
1.2
0
0
Table 2: Scaffolding strategy accuracy (%) across all models and languages. S-S = Self-Scaffolding (direct interpreter feedback, 1 LLM call), TSS = Textual Self-Scaffolding (coder-critic pair, 2 LLM calls), Re = ReAct pipeline. Each language contains 80 problems. Best results in bold .
Figure 3: Best-of-models accuracy on the same 80 EsoLang-Bench problems expressed in mainstream vs. esoteric languages. Mainstream baselines (Python, JavaScript) reach 100% (top frontier models solve every problem), while esoteric peaks are language-specific best-of across all five prompting strategies.
BF
BE
WS
UN
SH
Avg
Model
1
8
1
8
1
8
1
8
1
8
1
8
GPT-5.4 xhigh
2.5
3.8
5.0
8.8
0.0
0.0
0.0
0.0
0.0
0.0
1.5
2.5
O4-mini-high
2.5
2.5
5.0
7.5
0.0
0.0
0.0
1.2
0.0
0.0
1.5
2.2
Gemini 3.1 Pro
2.5
3.8
5.0
5.0
0.0
0.0
0.0
0.0
0.0
0.0
1.5
1.8
Qwen3-235B
1.2
2.5
2.5
3.8
0.0
0.0
0.0
0.0
0.0
0.0
0.8
1.2
Kimi K2.5
1.2
2.5
3.8
5.0
0.0
0.0
0.0
0.0
0.0
0.0
1.0
1.5
Table 3: Zero-shot pass@ k (%, of 80; n=8 draws, T=0.8 ) for k∈{1,8} . BF=Brainfuck, BE=Befunge-98, WS=Whitespace, UN=Unlambda, SH=Shakespeare. WS, UN, and SH stay at ∼ 0% for every model and every k , and all Medium/Hard/Extra-Hard remain unsolved at any k .
Figure 4: Error distribution by language (GPT-5.4 xhigh, zero-shot). BF=Brainfuck, Bef=Befunge-98, WS=Whitespace, Unl=Unlambda, Shk=Shakespeare. Whitespace and Unlambda show near-total compile failure, and Brainfuck and Befunge-98 are dominated by logic errors despite producing parsable code. Per-model error breakdowns are in Appendix M .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: EsoLang-Bench overview (schematic). Left. The benchmark comprises five esoteric programming languages spanning diverse computational paradigms, with 80 problems across four difficulty tiers (400 total evaluations). Right. The evaluation pipeline tests five frontier models across five prompting strategies (zero-shot, few-shot, self-scaffolding, textual self-scaffolding, ReAct) with automated interpreter-based verification. Peak accuracy is 11.2% (GPT-5.4 xhigh, self-scaffolding on Befunge-98), compared to 100% on the same 80 problems expressed in Python.
Problem
Pseudocode
Complexity
Easy tier — single primitive, one arithmetic / I/O / loop step
E04 Sum Two Integers
read a, b; print(a + b)
O(1)
E07 String Length
read line s; c = 0; for ch in s: c += 1; print(c)
O(∣s∣)
E11 Sum Of Digits
read N; N = abs(N); s = 0; while N > 0: s += N % 10; N //= 10; print(s)
O(logN)
Medium tier — composition of two-to-three primitives, single explicit loop with state
M04 Caesar Shift By 3
read s; for ch in s: print(chr((ord(ch)-’a’+3) mod 26 + ’a’))
O(∣s∣)
Appendix
Table 4: Tier calibration anchors. Three representative problems per tier with a language-agnostic pseudocode reference and the asymptotic complexity used to assign the tier. The complexity column is for the underlying algorithm at the pseudocode level, not the esoteric-language realisation (which is substantially longer in raw instruction count, e.g. Brainfuck multiplication is O(N) at pseudocode but requires nested tape loops). N denotes the input size as defined in the problem statement.
Category
Count
Representative Problems
Basic I/O
5
Hello World, Echo Line, Concatenate Lines
Arithmetic
17
Sum, Multiply, Factorial, Modular Exponentiation
String Manipulation
26
Reverse, Palindrome, Caesar Cipher, RLE
Number Theory
8
GCD, Primes, Factorization, LCM
Base Conversion
4
Binary ↔ Decimal, Roman ↔ Integer
Sorting/Arrays
9
Sort, LIS, Inversions, Merge Sorted
Appendix
Table 5: Distribution of problems across algorithmic categories. Counts sum to 80 and each problem belongs to exactly one category.
Python
JavaScript
Model
E
M
H
X
Total
E
M
H
X
Total
GPT-5.4 xhigh
20/20
20/20
20/20
20/20
100%
20/20
20/20
20/20
20/20
100%
Gemini 3.1 Pro
20/20
20/20
20/20
20/20
100%
20/20
20/20
20/20
20/20
100%
Kimi K2.5
20/20
20/20
20/20
19/20
98.8%
20/20
20/20
20/20
19/20
98.8%
Qwen3-235B
20/20
20/20
19/20
20/20
98.8%
20/20
20/20
20/20
19/20
98.8%
Appendix
Table 6: Mainstream-language baseline accuracy (zero-shot = few-shot for all cells). Each tier contains 20 problems; total = 80. The same problems that drive frontier models to 0% beyond Easy in esoteric languages are saturated in both Python and JavaScript.
Model
0-Shot
Few
Self-S
ReAct
Best
GPT-5.4 xhigh
2
2
5
4
5 (6.2%)
O4-mini
2
3
4
3
4 (5.0%)
Gemini 3.1 Pro
2
3
4
3
4 (5.0%)
Qwen3-235B
2
1
2
1
2 (2.5%)
Kimi K2.5
0
0
0
0
0 (0%)
Appendix
Table 7: Brainfuck results by model and strategy (Easy solved / 20). Accuracy = Solved/80.
Model
0-Shot
Few
Self-S
ReAct
Best
GPT-5.4 xhigh
2
7
9
7
9 (11.2%)
O4-mini
5
6
8
6
8 (10.0%)
Gemini 3.1 Pro
4
3
6
5
6 (7.5%)
Qwen3-235B
0
0
0
0
0 (0%)
Kimi K2.5
2
1
2
0
2 (2.5%)
Appendix
Table 8: Befunge-98 results by model and strategy (Easy solved / 20). Accuracy = Solved/80.
Figure 6: Best accuracy achieved per language (across all models and strategies). Befunge-98 is the most tractable (11.2%), while Whitespace remains completely unsolved (0%).
Language
Three demonstration problems
Brainfuck
E01 Print Hello World; E02 Echo Line; E04 Sum Two Integers
Befunge-98
E01 Print Hello World; E04 Sum Two Integers; E06 Even Or Odd
Whitespace
E01 Print Hello World; E04 Sum Two Integers; E10 Sum From 1 To N
Unlambda
E01 Print Hello World; E02 Echo Line; E03 Hello Name
Shakespeare
E01 Print Hello World; E03 Hello Name; E04 Sum Two Integers
Appendix
Table 10: Three Easy-tier problems used as Few-Shot demonstrations per language. The selection is identical across all models and runs.
Prior method
Closest equivalent in our setup
Self-Refine [ Madaan et al., 2023 ]
Strict subset of Textual Self-Scaffolding
Reflexion [ Shinn et al., 2023 ]
Textual Self-Scaffolding (TSS)
Self-debugging [ Chen et al., 2023 ]
Self-Scaffolding
OpenCodeInterpreter [ Zheng et al., 2024 ]
Self-Scaffolding (interpreter-in-the-loop)
ARCS [ Bhattarai et al., 2025 ]
Few-Shot + Self-Scaffolding
Execution-guided line-by-line [ Lavon et al., 2025 ]
Not applicable (see below)
Appendix
Table 11: Mapping of prior self-improvement and execution-guided methods to our prompting strategies. “Strict subset” means the prior method’s information is a subset of what our strategy receives.
Figure 7: Compile error rates by model across languages. All models show near-identical patterns: low compile errors on Brainfuck/Befunge-98, complete failure (100%) on Whitespace, and high failure (88–95%) on Unlambda.
Benchmark
Languages
Scarcity
Contamination control
Verifiable
Size
HumanEval [ Chen et al., 2021 ]
Python
Low
static
exec
164
MBPP [ Austin et al., 2021 ]
Python
Low
static
exec
974
MultiPL-E [ Cassano et al., 2023 ]
18 mainstream
Low
static
exec
∼ 3,000
DS-1000 [ Lai et al., 2023 ]
Python (DS libs)
Low
static
exec
1,000
SWE-bench [ Jimenez et al., 2024 ]
Python (repo)
Low
static
exec
2,294
LiveCodeBench [ Jain et al., 2025 ]
Python
Low
rolling release
exec
> 600
Appendix
Table 12: EsoLang-Bench vs. existing code-generation benchmarks. “Pre-train scarcity” refers to whether target languages are likely under-represented in pre-training corpora.
Large language models (LLMs) achieve high pass rates on code generation benchmarks, yet whether they can transfer this ability to languages absent from pretraining remains poorly understood. We introduce PyLang, a minimal imperative language absent from all pretraining corpora, and evaluate frontier models zero-shot and fine-tuned Qwen3 (4B, 8B, 32B) on 352 problems. We find that fine-tuning quickly teaches syntax but fails to transfer semantic competence: Python outperforms PyLang by up to 19% across all configurations, and no intervention (multi-task learning, preference tuning, code infilling, or latent-space objectives) closes the gap. An LLM judge reveals that frontier models select an identical algorithm to Python 80% of the time, yet cannot translate it into a working PyLang implementation., and CKA analysis confirms that fine-tuned models converge to nearly identical internal representations across languages (CKA > 0.97) while diverging at the output stage. We term this the implementation fidelity gap: models possess language-agnostic algorithmic understanding but cannot express it in an unfamiliar language. Our findings highlight the need for training methods that decouple reasoning from language-specific realization.
This paper introduces Code Bench, a benchmark capable of evaluating Large Language Models (LLMs) concise code generation abilities in 60 programming languages. Based on code golf, a recreational programming competition focused on minimal character or byte solutions, the benchmark provides a distinctive measure of LLMs ability to produce efficient, concise code. Unlike existing benchmarks limited by fixed problem sets and language coverage, CodeGolf Bench leverages the code.golf platform to provide new problems and live human performance baselines. Evaluation of nine LLMs on Python and C++ tasks demonstrates that reasoning models significantly outperform non-reasoning models, achieving best average percentile of 70.97%. This performance gap is particularly pronounced in C++, highlighting reasoning's importance for languages with strict syntax requirements. Non-reasoning models struggle more with efficiency optimization across both languages, with best percentiles significantly lower than reasoning counterparts. CodeGolf Bench offers a dynamic framework for evaluating LLM code generation capabilities against evolving human performance on code golf.
LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB's contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities.
Maria Ivanova, Pavel Zadorozhny, Rodion Levichev +5
Yandex School of Data Analysis, Applied AI Institute · GigaCode