EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages

Aman Sharma; Paras Chopra

cs.AIMar 10, 2026

EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages

Authors: Aman Sharma, Paras Chopra

Organizations: Lossfunk

Abstract

Large language models achieve near-ceiling performance on code generation benchmarks, yet most of the programming languages used by popular benchmarks such as SWE-bench and HumanEval (e.g. Python, JavaScript) are squarely in-distribution. They appear at scale in pre-training corpora and are heavily reinforced during post-training. To study LLM performance on unfamiliar programming languages, we introduce EsoLang-Bench, a benchmark using five esoteric programming languages (Brainfuck, Befunge-98, Whitespace, Unlambda, and Shakespeare). All five of our chosen esoteric languages are Turing-complete, so the same algorithmic problems that are solvable in Python or JavaScript are in principle solvable in each of them. Yet, they are unfamiliar to LLMs which makes them a good proxy for evaluating out-of-distribution performance. The unfamiliarity of esoteric languages comprises of: (i) the hard-by-design primitives comprising the language; (ii) substantially less representation in pre-training corpora (340x to over 60,000x fewer public GitHub repositories than Python); (iii) negligible deployment value, which makes targeted inclusion in post-training data economically irrational. We evaluate five frontier models across five prompting strategies and find a dramatic capability gap. The same 80 problems expressed in Python or JavaScript reach 100% accuracy on top frontier models, while the equivalent esoteric versions score only 0-11%. Few-shot learning and self-reflection also fail to close this gap. EsoLang-Bench therefore provides a contamination-resistant testbed for measuring how well frontier models generalise algorithmic problem-solving to programming languages outside their training distribution.

Figures & tables

Figure 1: EsoLang-Bench at a glance. Top: the same Hello World written in Python and in each of the five esoteric languages targeted by the benchmark. Bottom: the dataset (80 problems per language, 4 difficulty tiers, 5 esoteric languages) and the evaluation grid (5 frontier models

\times

5 prompting strategies).

Panel of Figure 2 — Figure 2: Per-language GitHub topic-tag counts on a $\log_{10}$ scale. Bar labels show the actual repository counts. Methodology and per-language details in Appendix I .

	Brainfuck		Befunge-98		Whitespace		Unlambda		Shakespeare
Model	0-S	3-S	0-S	3-S	0-S	3-S	0-S	3-S	0-S	3-S
GPT-5.4 xhigh	2.5%	2.5%	2.5%	8.8%	0%	0%	0%	0%	2.5%	1.2%
O4-mini	2.5%	3.8%	6.2%	7.5%	0%	0%	0%	0%	1.2%	1.2%
Gemini 3.1 Pro	2.5%	3.8%	5.0%	3.8%	0%	0%	0%	0%	1.2%	1.2%
Qwen-235B	2.5%	1.2%	0%	0%	0%	0%	0%	1.2%	0%	0%
Kimi K2.5	0%	0%	2.5%	1.2%	0%	0%	0%	0%	1.2%	1.2%

Table 1: Zero-shot and few-shot (3 ICL examples) accuracy (%) across all models and languages. Each language contains 80 problems (20 per difficulty tier). Only Easy-tier problems were solved (all Medium/Hard/Extra-Hard = 0%). Best results per language in bold . 0-S = Zero-Shot, 3-S = Three-Shot Few-Shot.

Model	S-S	TSS	Re	S-S	TSS	Re	S-S	TSS	Re	S-S	TSS	Re	S-S	TSS	Re
	Brainfuck			Befunge-98			Whitespace			Unlambda			Shakespeare
GPT-5.4 xhigh	6.2	3.8	5.0	11.2	10.0	8.8	0	0	0	1.2	0	0	2.5	2.5	1.2
O4-mini	5.0	2.5	3.8	10.0	6.2	7.5	0	0	0	0	0	0	1.2	1.2	0
Gemini	5.0	3.8	3.8	7.5	6.2	7.5	0	0	0	0	0	0	1.2	0	0
Qwen-235B	2.5	1.2	2.5	0	0	0	0	0	0	1.2	0	0	1.2	0	0
Kimi K2.5	0	0	0	0	0	0	0	0	0	0	0	0	1.2	0	0

Table 2: Scaffolding strategy accuracy (%) across all models and languages. S-S = Self-Scaffolding (direct interpreter feedback, 1 LLM call), TSS = Textual Self-Scaffolding (coder-critic pair, 2 LLM calls), Re = ReAct pipeline. Each language contains 80 problems. Best results in bold .

Panel of Figure 3 — Figure 3: Best-of-models accuracy on the same 80 EsoLang-Bench problems expressed in mainstream vs. esoteric languages. Mainstream baselines (Python, JavaScript) reach 100% (top frontier models solve every problem), while esoteric peaks are language-specific best-of across all five prompting strategies.

	BF		BE		WS		UN		SH		Avg
Model	1	8	1	8	1	8	1	8	1	8	1	8
GPT-5.4 xhigh	2.5	3.8	5.0	8.8	0.0	0.0	0.0	0.0	0.0	0.0	1.5	2.5
O4-mini-high	2.5	2.5	5.0	7.5	0.0	0.0	0.0	1.2	0.0	0.0	1.5	2.2
Gemini 3.1 Pro	2.5	3.8	5.0	5.0	0.0	0.0	0.0	0.0	0.0	0.0	1.5	1.8
Qwen3-235B	1.2	2.5	2.5	3.8	0.0	0.0	0.0	0.0	0.0	0.0	0.8	1.2
Kimi K2.5	1.2	2.5	3.8	5.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0	1.5

Table 3: Zero-shot pass@

k

(%, of 80;

n=8

draws,

T=0.8

) for

k\in\{1,8\}

. BF=Brainfuck, BE=Befunge-98, WS=Whitespace, UN=Unlambda, SH=Shakespeare. WS, UN, and SH stay at

\sim

0% for every model and every

k

, and all Medium/Hard/Extra-Hard remain unsolved at any

k

Panel of Figure 4 — Figure 4: Error distribution by language (GPT-5.4 xhigh, zero-shot). BF=Brainfuck, Bef=Befunge-98, WS=Whitespace, Unl=Unlambda, Shk=Shakespeare. Whitespace and Unlambda show near-total compile failure, and Brainfuck and Befunge-98 are dominated by logic errors despite producing parsable code. Per-model error breakdowns are in Appendix M .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: EsoLang-Bench overview (schematic). Left. The benchmark comprises five esoteric programming languages spanning diverse computational paradigms, with 80 problems across four difficulty tiers (400 total evaluations). Right. The evaluation pipeline tests five frontier models across five prompting strategies (zero-shot, few-shot, self-scaffolding, textual self-scaffolding, ReAct) with automated interpreter-based verification. Peak accuracy is 11.2% (GPT-5.4 xhigh, self-scaffolding on Befunge-98), compared to 100% on the same 80 problems expressed in Python.

Problem	Pseudocode	Complexity
Easy tier — single primitive, one arithmetic / I/O / loop step
E04 Sum Two Integers	read a, b; print(a + b)	$O(1)$
E07 String Length	read line s; c = 0; for ch in s: c += 1; print(c)	$O(\|s\|)$
E11 Sum Of Digits	read N; N = abs(N); s = 0; while N > 0: s += N % 10; N //= 10; print(s)	$O(\log N)$
Medium tier — composition of two-to-three primitives, single explicit loop with state
M04 Caesar Shift By 3	read s; for ch in s: print(chr((ord(ch)-’a’+3) mod 26 + ’a’))	$O(\|s\|)$

Appendix

Table 4: Tier calibration anchors. Three representative problems per tier with a language-agnostic pseudocode reference and the asymptotic complexity used to assign the tier. The complexity column is for the underlying algorithm at the pseudocode level, not the esoteric-language realisation (which is substantially longer in raw instruction count, e.g. Brainfuck multiplication is

O(N)

at pseudocode but requires nested tape loops).

N

denotes the input size as defined in the problem statement.

Category	Count	Representative Problems
Basic I/O	5	Hello World, Echo Line, Concatenate Lines
Arithmetic	17	Sum, Multiply, Factorial, Modular Exponentiation
String Manipulation	26	Reverse, Palindrome, Caesar Cipher, RLE
Number Theory	8	GCD, Primes, Factorization, LCM
Base Conversion	4	Binary $\leftrightarrow$ Decimal, Roman $\leftrightarrow$ Integer
Sorting/Arrays	9	Sort, LIS, Inversions, Merge Sorted

Appendix

Table 5: Distribution of problems across algorithmic categories. Counts sum to 80 and each problem belongs to exactly one category.

	Python					JavaScript
Model	E	M	H	X	Total	E	M	H	X	Total
GPT-5.4 xhigh	20/20	20/20	20/20	20/20	100%	20/20	20/20	20/20	20/20	100%
Gemini 3.1 Pro	20/20	20/20	20/20	20/20	100%	20/20	20/20	20/20	20/20	100%
Kimi K2.5	20/20	20/20	20/20	19/20	98.8%	20/20	20/20	20/20	19/20	98.8%
Qwen3-235B	20/20	20/20	19/20	20/20	98.8%	20/20	20/20	20/20	19/20	98.8%

Appendix

Table 6: Mainstream-language baseline accuracy (zero-shot

=

few-shot for all cells). Each tier contains 20 problems; total

=

80. The same problems that drive frontier models to 0% beyond Easy in esoteric languages are saturated in both Python and JavaScript.

Model	0-Shot	Few	Self-S	ReAct	Best
GPT-5.4 xhigh	2	2	5	4	5 (6.2%)
O4-mini	2	3	4	3	4 (5.0%)
Gemini 3.1 Pro	2	3	4	3	4 (5.0%)
Qwen3-235B	2	1	2	1	2 (2.5%)
Kimi K2.5	0	0	0	0	0 (0%)

Appendix

Table 7: Brainfuck results by model and strategy (Easy solved / 20). Accuracy = Solved/80.

Model	0-Shot	Few	Self-S	ReAct	Best
GPT-5.4 xhigh	2	7	9	7	9 (11.2%)
O4-mini	5	6	8	6	8 (10.0%)
Gemini 3.1 Pro	4	3	6	5	6 (7.5%)
Qwen3-235B	0	0	0	0	0 (0%)
Kimi K2.5	2	1	2	0	2 (2.5%)

Appendix

Table 8: Befunge-98 results by model and strategy (Easy solved / 20). Accuracy = Solved/80.

Model	Unlambda	Shakespeare
GPT-5.4 xhigh	1 (1.2%)	2 (2.5%)
O4-mini	0 (0%)	1 (1.2%)
Gemini 3.1 Pro	0 (0%)	1 (1.2%)
Qwen3-235B	1 (1.2%)	1 (1.2%)
Kimi K2.5	0 (0%)	1 (1.2%)
Best	1 (1.2%)	2 (2.5%)

Appendix

Table 9: Whitespace, Unlambda, Shakespeare results (Best Easy solved / 20). Accuracy = Solved/80.

Panel of Figure 6 — Figure 6: Best accuracy achieved per language (across all models and strategies). Befunge-98 is the most tractable (11.2%), while Whitespace remains completely unsolved (0%).

Language	Three demonstration problems
Brainfuck	E01 Print Hello World; E02 Echo Line; E04 Sum Two Integers
Befunge-98	E01 Print Hello World; E04 Sum Two Integers; E06 Even Or Odd
Whitespace	E01 Print Hello World; E04 Sum Two Integers; E10 Sum From 1 To N
Unlambda	E01 Print Hello World; E02 Echo Line; E03 Hello Name
Shakespeare	E01 Print Hello World; E03 Hello Name; E04 Sum Two Integers

Appendix

Table 10: Three Easy-tier problems used as Few-Shot demonstrations per language. The selection is identical across all models and runs.

Prior method	Closest equivalent in our setup
Self-Refine [ Madaan et al., 2023 ]	Strict subset of Textual Self-Scaffolding
Reflexion [ Shinn et al., 2023 ]	Textual Self-Scaffolding (TSS)
Self-debugging [ Chen et al., 2023 ]	Self-Scaffolding
OpenCodeInterpreter [ Zheng et al., 2024 ]	Self-Scaffolding (interpreter-in-the-loop)
ARCS [ Bhattarai et al., 2025 ]	Few-Shot $+$ Self-Scaffolding
Execution-guided line-by-line [ Lavon et al., 2025 ]	Not applicable (see below)

Appendix

Table 11: Mapping of prior self-improvement and execution-guided methods to our prompting strategies. “Strict subset” means the prior method’s information is a subset of what our strategy receives.

Panel of Figure 7 — Figure 7: Compile error rates by model across languages. All models show near-identical patterns: low compile errors on Brainfuck/Befunge-98, complete failure (100%) on Whitespace, and high failure (88–95%) on Unlambda.

Benchmark	Languages	Scarcity	Contamination control	Verifiable	Size
HumanEval [ Chen et al., 2021 ]	Python	Low	static	exec	164
MBPP [ Austin et al., 2021 ]	Python	Low	static	exec	974
MultiPL-E [ Cassano et al., 2023 ]	18 mainstream	Low	static	exec	$\sim$ 3,000
DS-1000 [ Lai et al., 2023 ]	Python (DS libs)	Low	static	exec	1,000
SWE-bench [ Jimenez et al., 2024 ]	Python (repo)	Low	static	exec	2,294
LiveCodeBench [ Jain et al., 2025 ]	Python	Low	rolling release	exec	$>$ 600

Appendix

Table 12: EsoLang-Bench vs. existing code-generation benchmarks. “Pre-train scarcity” refers to whether target languages are likely under-represented in pre-training corpora.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language
May 15, 2026Vinayshekhar Bannihatti Kumar, Disha Makhija, Manoj Ghuhan Arivazhagan +1LLM Fine-Tuning Code Language Models
AWS AI Labs
CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models
May 28, 2026Vedant PadwalLanguage Model Generation Evaluation Code Generation
Independent
Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
Jun 18, 2026Maria Ivanova, Pavel Zadorozhny, Rodion Levichev +5Code Generation Code Generation Evaluation
Yandex School of Data Analysis, Applied AI Institute · GigaCode

May 15, 2026J/K move · Enter open · S save

Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language

Vinayshekhar Bannihatti Kumar, Disha Makhija, Manoj Ghuhan Arivazhagan, Rashmi Gangadharaiah

AWS AI Labs

LLM Fine-Tuning Code Language Models LLM Code Generation

Open paper AskPDF arXiv

Large language models (LLMs) achieve high pass rates on code generation benchmarks, yet whether they can transfer this ability to languages absent from pretraining remains poorly understood. We introduce PyLang, a minimal imperative language absent from all pretraining corpora, and evaluate frontier models zero-shot and fine-tuned Qwen3 (4B, 8B, 32B) on 352 problems. We find that fine-tuning quickly teaches syntax but fails to transfer semantic competence: Python outperforms PyLang by up to 19% across all configurations, and no intervention (multi-task learning, preference tuning, code infilling, or latent-space objectives) closes the gap. An LLM judge reveals that frontier models select an identical algorithm to Python 80% of the time, yet cannot translate it into a working PyLang implementation., and CKA analysis confirms that fine-tuned models converge to nearly identical internal representations across languages (CKA > 0.97) while diverging at the output stage. We term this the implementation fidelity gap: models possess language-agnostic algorithmic understanding but cannot express it in an unfamiliar language. Our findings highlight the need for training methods that decouple reasoning from language-specific realization.

Abstract

LLM Evaluation

Explore similar work

Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language

CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models

Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages