The IOL-AI Challenge: An Open Challenge towards Advancing Linguistic Reasoning

Eduardo Sánchez; Rita Berrada; Dan-Mircea Mirea; Sara Rajaee; Alexander Piperski; Ana Meta Dolinar; Boris Iomdin; Andrey Nikulin

cs.CLAug 18, 2026

The IOL-AI Challenge: An Open Challenge towards Advancing Linguistic Reasoning

Authors: Eduardo Sánchez, Rita Berrada, Dan-Mircea Mirea, Sara Rajaee, Alexander Piperski, Ana Meta Dolinar, Boris Iomdin, Andrey Nikulin,

Organizations: University College London & Meta · CentraleSupélec & McGill University · Cohere Labs · Princeton University ·

Abstract

Reasoning in LLMs is overwhelmingly studied in domains that provide a model with rules: mathematics and code. Linguistic puzzles invert this: the solver must first discover the system before reasoning within it. We present the IOL-AI Challenge, an open-science competition run on the unseen problems of the International Linguistics Olympiad (IOL) 2026 Individual Contest, evaluated both automatically and, for the first time, by members of the official IOL Jury under the same rubrics applied to human contestants. The challenge drew 731 submissions from 46 teams under a strict compute budget (one T4, 30 mins). We additionally benchmark 15 unconstrained frontier and open models, with Claude Opus 4.8 earning a jury score equivalent to a gold medal, while both resource-constrained systems we submitted for jury grading scored in the range of the bottom 5% of contestants. Capability was not determined by scale: 14B submissions outperform models twice their size, and gains come from decoding and output-handling rather than model capacity. We also found that automatic metrics rank systems exactly as the jury does, but compress the scale, upscoring weak systems by ~13 points and understating strong ones. Our analysis shows that while frontier models might have prior knowledge about some of the problem languages, it does not significantly help them solve the linguistic reasoning tasks, leaving linguistic reasoning as a strong benchmarking proxy for generalizable reasoning skills.

Figures & tables

Problem Structure	Prompt Type	Context	Question
Rosetta Stone	Translation	Here are some sentences in English and their Komnzo translations: …	Translate into English in all possible ways: …
Chaos-and-order	Fill-in Blanks	Here is the family tree of a family of Sakurabiat speakers. …	Fill in the blanks (i–vi). There is only one possible answer for each blank, but each answer is not necessarily just one word.
Chaos-and-order	Match Letters	Here are some words and phrases in Yélî Dnye and their English translations in arbitrary order: …	Determine the correct correspondences.

Table 1: Examples from the IOL 2026 Individual Contest, structured into context and question to prompt LLMs.

Baseline	chrF	EM	GM
Qwen2.5-14B-Instruct-AWQ	19.70	4.49	9.40
Tiny Aya Global (3.35B)	16.09	4.10	8.12
Qwen2.5-7B-Instruct (NF4)	19.65	2.56	7.10
Qwen2.5-1.5B-Instruct (starter)	12.51	1.28	4.00

Table 2: Automatic evaluation scores of organizer baselines . EM = exact match, GM = geometric mean.

#	Team	chrF	EM	GM	Model
1	arvindcr4 ⋆	31.57	12.41	19.79	Qwen2.5-14B AWQ
2	BigRatz ⋆	30.75	11.55	18.85	Qwen2.5-14B AWQ
3	hhhar	29.14	11.35	18.19	Qwen2.5-14B AWQ
4	friedspaghetti	28.98	11.07	17.91	Qwen2.5-14B AWQ
5	mastermind	27.78	10.07	16.73	Qwen2.5-14B AWQ
6	Hul	28.41	9.84	16.72	Qwen2.5-14B AWQ

Table 3: Automatic evaluation scores for the top 10 from the leaderboard of the IOL-AI challenge. Submissions with

\star

were selected for jury evaluation. Dashed lines indicate first, second, and third place winners on the leaderboard.

Proprietary Models
Model	chrF	EM	GM	LFMR
Claude Opus 4.8 $\star$	82.9	68.1	75.1	100.0
Gemini 3.6 Flash $\star$	70.6	45.0	56.4	100.0
GPT 5.6 Sol	63.3	43.1	52.2	100.0
Open Large Models
Kimi K2.6	57.1	35.8	45.2	100.0

Table 4: Automatic evaluation scores for the extended model set on the IOL 2026 problems (after custom answer parsing). LFMR = line format match rate. Model outputs that were chosen for the jury are marked with

\star

Figure 1: Distribution of automatic scores (GM) by system tier, spanning from organizer baselines and resource-constrained competition submissions to open mid-size, open large, and proprietary frontier models.

Panel of Figure 2 — Figure 2: Breakdown of scores (geomean %) across the individual tasks of the 5 IOL 2026 problems, for the top 10 challenge submissions (top) and frontier models (bottom).

	Total Score		P1		P2		P 3		P4		P5
Model Name	Jury	Aut.	Jury	Aut.	Jury	Aut.	Jury	Aut.	Jury	Aut.	Jury	Aut.
Claude Opus 4.8	79.5	75.1	18.5	14.5	10.5	16.0	15.5	12.2	19	23.2	16	9.2
Gemini 3.6 Flash	60.5	56.4	13	10.8	9	11.5	15	9.2	13	19.0	10.5	5.8
BoN Open (oracle)	20.3	33.5	4.5	3.4	2.3	6.7	4.5	7.2	3.5	7.3	5.5	8.9
arvindcr4 (#1)	5.9	19.8	0.8	1.3	2.1	5.8	0.5	2.6	2	6.3	0.5	3.8
cabanosss (#11)	2.5	14.6	1	1.0	1	5.3	0	2.2	0	2.3	0.5	3.8

Table 5: Jury evaluation results by problem for the selected AI submissions compared to the automatic GM scores ( Aut. ).

Panel of Figure 3 — Figure 3: Schematic explanations provided in official IOL solutions for P5 in comparison to the top scoring model’s solution explanation that we elicit through prompting.

Figure 4: Difference between automatic and jury scores per problem for the five systems selected for human evaluation. Cla = Claude Opus 4.8, Gem = Gemini 3.6 Flash, BoN = BoN Open (oracle), #1 = arvindcr4, #11 = cabanosss.

Figure 5: Comparison of the difficulty of criteria in the official IOL grading schemes between human contestants and models. Difficulty is measured as the percentage of solutions that meet the criterion and receive the respective mark, split into human contestants and AI models.

Table 6: Results of knowledge probing tasks for a subset of benchmarked models.

Model	chrF	EM	Geo	chrF anon	EM anon	Geo anon	$\Delta$ Geo	95% CI
GPT 5.6 Sol	65.6	47.6	55.8	63.6	39.7	50.2	-5.6	[-16.2, +3.9]
Claude Opus 4.8	69.5	52.0	60.1	64.7	50.5	57.2	-3.0	[-15.2, +10.0]
Command A+	29.6	6.6	14.0	22.9	3.4	8.0	-5.9	[-18.0, +3.3]
Qwen 3.7 Max	52.8	27.4	38.0	42.1	16.5	26.3	-11.7	[-23.3, +0.6]
Kimi K2.6	51.8	31.2	40.2	51.3	30.4	39.5	-0.7	[-16.9, +12.9]
Nemotron 3 Ultra	36.7	18.0	25.4	36.8	19.7	26.7	+1.2	[-19.8, +18.2]

Table 7: Automatic score change when language mentions and context are dropped from the task descriptions (without custom answer parsing that would raise all scores). Scores are aggregated across three repeated runs with different random seeds.

Language	Geo	Geo anon	$\Delta$ Geo	95% CI	# models $\downarrow$
Yélî Dnye	21.0	15.2	-5.8	[-9.9, -1.9]	6/7
Komnzo	48.8	45.0	-3.8	[-10.9, +4.0]	5/7
Sakurabiat	42.4	41.4	-1.0	[-7.0, +4.0]	3/7
Central Alaskan Yup’ik	51.6	51.9	+0.3	[-9.7, +9.8]	3/7
Iquito	40.8	41.7	+0.9	[-3.7, +5.6]	3/7

Table 8: Automatic score change on each problem for an average across three runs from models in Table 7 .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	Original				+ custom answer parsing
Model	chrF	EM	GM	LFMR	chrF	EM	GM	LFMR
Proprietary Models
claude-opus-4-8 $\star$	70.6	52.8	61.0	92.9	82.9	68.1	75.1	100.0
gemini-3.6-flash $\star$	65.8	37.3	49.6	92.9	70.6	45.0	56.4	100.0
gpt5.6-sol	63.3	43.1	52.2	92.9	63.3	43.1	52.2	100.0
Open Large Models

Appendix

Table 9: Evaluation scores for existing models on the IOL 2026 problem set, before and after answer parsing. The BoN Open (oracle) ensemble builds directly on custom answer parsed answers. EM = exact match, GM = geometric mean, LFMR = line format match rate. Model outputs that were chosen for the jury are marked with

\star

	Language	chrF	EM	GM
P1	Yup’ik	49.1	7.1	18.7
P2	Yélî Dnye	25.1	17.9	21.2
P3	Iquito	83.5	25.0	45.7
P4	Sakurabiat	32.8	27.8	30.2
P5	Komnzo	63.6	50.0	56.4

Appendix

Table 10: Per-problem automatic scores for the BoN Open (oracle) ensemble. Scores are on a 0–100 scale per problem, so they are directly comparable to jury scores expressed as a percentage of the 20 points available per problem.

Panel of Figure 6 — Figure 6: Format-failure prevalence for IOL 2026 frontier runs before and after light answer extraction.

Model	Provider	temp	top_p	Reasoning Effort	Max Tokens
Claude Opus 4.8	Bedrock	1.0	1.0	high	128000
GPT-5.6	OpenAI	1.0	1	high	128000
Kimi K2.6	TogetherAI	1.0	0.95	N/A	256000
GLM-5.2	TogetherAI	1.0	0.95	max	128000
DeepSeek V4 Pro	TogetherAI	1.0	1	high	128000
Qwen3.7-Max	TogetherAI	0.6	0.95	N/A	65536

Appendix

Table 11: Hyper-parameters for evaluation. Self-hosted models were served with vLLM; “Max Tokens” is the generation budget, not the model’s context window. † GLM 4.7 Flash additionally requires repetition_penalty

{=}1.05

; without it the model degenerates into repetition loops, which is reflected in its low format-compliance and near-zero scores.

Closed Models
	Original				+ custom answer parsing
Model	chrF	EM	GM	LFMR	chrF	EM	GM	LFMR
GPT 5.6 Sol	72.0	26.9	44.0	90.6	78.5	33.8	51.5	98.8
Claude Opus 4.8 (Bedrock)	54.8	6.9	19.4	81.9	71.1	26.9	43.7	94.4
Gemini 3.6 Flash	63.7	10.6	26.0	99.4	78.0	33.1	50.8	98.8
Open Large Models

Appendix

Table 12: Out-of-the-box benchmarking scores for existing models on Linguini. EM = exact match, GM = geometric mean, LFMR = line format match rate.

Task	Problem Language	Question	Answer Options
Language Context	Yup’ik	Region where it is spoken?	A. General Central Yup’ik; B. eastern Alaska; C. northern Alaska; D. southwestern Alaska
	Yélî Dnye	Island / place where it is spoken?	A. Brazil; B. New Britain; C. Rossel Island ; D. Norton Sound
	Iquito	Does the language have tones?	A. not known; B. yes ; C. no; D. in some local dialects
	Sakurabiat	Approximate number of speakers?	A. 3; B. 300000; C. 13 ; D. 13000
	Komnzo	Village where it is spoken?	A. Tupian; B. everywhere in south-eastern Papua New Guinea; C. Rouku ; D. Jipai
Lexicon	Yup’ik	Real word/phrase?	A. at’aiqnt; B. aknaepvn; C. oykey; D. nanevpak

Appendix

Table 13: Examples of knowledge probing tasks derived from the problem statements from IOL 2026. Bold-faced answers are the correct answers. Language Context = 4-way MCQ over meta-facts from the IOL blurbs; Lexicon = 4-way MCQ over attested forms (distractors: scrambled or other-language). Questions are shortened.

Inductive Reasoning in Language Models

See all 12 papers

BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models
Mar 16, 2026Yuzhe TangLanguage Model Reasoning Evaluation LLM Evaluation
Georgia Institute of Technology
MultiZebraLogic: A Multilingual Logical Reasoning Benchmark
Nov 5, 2025Sofie Helene Bruun, Dan Saattrup SmartLogical Reasoning Multilingual Language Model Evaluation
The Alexandra Institute Rued Langgaards Vej 7, 5D, 2300 Copenhagen S, Denmark
Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces
Aug 12, 2026Congchao Wang, Diwakar Singh, Qiaozi Gao +3LLM-as-a-Judge Reasoning Evaluation
Amazon AGI

Mar 16, 2026J/K move · Enter open · S save

BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models

Yuzhe Tang

Georgia Institute of Technology

Language Model Reasoning Evaluation LLM Evaluation Implicit Reasoning in Language Models Reasoning Consistency in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.

Abstract

Inductive Reasoning in Language Models

Explore similar work

BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models

MultiZebraLogic: A Multilingual Logical Reasoning Benchmark

Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces