Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture
Authors: Yunsu Kim, Kaden Uhlig, Ashwin Purohit, Milind Agarwal, Patrick Simianer, Anil Arslan, Kiarash Mokhtari, Thomas Zenkel, Johannes Mosig, Gabriel Bretschner, Shamik Bose, Joern Wuebker, John DeNero
Most evaluations for coding agents are conducted exclusively in English, which does not reflect real-world multilingual deployment. We present Terminal-Bench-LILT, a suite of 300 authentic coding tasks in ten languages: Arabic, Czech, German, Spanish, Hindi, Japanese, Korean, Serbian, Turkish, and Chinese. Each task targets issues specific to non-English software development that have no direct English equivalent, e.g., internationalization, encoding, text normalization, and cultural conventions. All tasks are authored by native-speaker programmers and validated through a multi-stage quality control pipeline. Evaluation of six frontier models reveals that even the strongest model reaches only 63.1% pass rate, with many tasks unsolved by any model. Performance varies substantially by language and does not track general coding benchmark rankings, highlighting that multilingual coding competence is a distinct and underexplored capability axis. Sample tasks are available at https://github.com/lilt/terminal-bench-lilt
Figures & tables
Figure 1: The ja-terminal-width task from Terminal-Bench-LILT. The instruction in (a) is abridged for clarity. In (b), each grid cell is one display column of the terminal.
Category
Subcategory
Scope / Examples
Internationalization
Locale-based Templating
Plurals, numbers, suffixes, date formats, Korean man numbering
Character Rendering
RTL, Arabic shaping, Simplified vs. Traditional Chinese, Cyrillic
Case augmentation, tokenizer training, typo simulation
Table 1: Challenge taxonomy for Terminal-Bench-LILT tasks.
Category
ar
cs
de
es
hi
ja
ko
sr
tr
zh
Total
Internationalization
10
7
3
2
7
5
6
6
5
1
52
Interaction with Environment
2
2
5
3
3
3
1
1
2
3
25
Text Conversion
10
3
3
11
6
7
2
15
12
7
76
Text Handling
1
1
2
2
4
2
1
3
2
0
18
Cultural & Other
7
17
17
12
10
13
20
5
9
19
129
Total
30
30
30
30
30
30
30
30
30
30
300
Table 2: Distribution of tasks by challenge category and language.
Figure 2: Instructions for two sample tasks with different challenge scopes: (a) a bounded challenge and (b) an unbounded one fenced to a finite range.
Figure 3: Quality control workflow for task submissions. ✓ and ✗ indicate whether a task passes or fails each stage, respectively. The responsible party is shown above each stage.
Figure 4: Workflow management tool for Terminal-Bench-LILT.
Figure 5: Task pass rate per model on Terminal-Bench-LILT, sorted by rate. Error bars show 95% confidence intervals.
Difficulty
ar
cs
de
es
hi
ja
ko
sr
tr
zh
Total
Easy
16
17
13
12
8
13
12
16
17
9
133
Medium
9
4
6
12
12
8
9
7
6
13
86
Hard
5
9
11
6
10
9
9
7
7
8
81
Total
30
30
30
30
30
30
30
30
30
30
300
Table 3: Distribution of tasks by language and empirical difficulty.
Model
ar
cs
de
es
hi
ja
ko
sr
tr
zh
σlang
GPT-5.5
69.3
60.7
51.3
74.0
51.3
70.7
58.0
72.7
69.3
53.3
8.7
Gemini 3.5 Flash
62.7
64.0
56.7
60.7
52.0
57.3
58.7
66.0
69.3
64.7
4.9
Claude Opus 4.8
59.3
68.7
50.0
68.7
52.7
52.7
57.3
61.3
66.7
64.7
6.6
σmodel
4.2
3.3
2.9
5.5
0.5
7.6
0.5
4.7
1.3
5.3
–
Table 4: Per-language pass rates. σlang and σmodel are the standard deviations (in percentage points) across languages and across models, respectively.
International-
Interaction w/
Text
Text
Cultural &
Model
ization
Environment
Conversion
Handling
Other
GPT-5.5
63.5
70.4
71.3
53.3
58.0
Gemini 3.5 Flash
60.4
69.6
69.7
55.6
55.7
Claude Opus 4.8
60.8
64.8
65.0
53.3
57.2
Table 5: Per-category pass rates for the top three models.
Figure 6: Native language vs. English instruction ablation. Every bar pair covers the same 30 tasks, scored in both conditions. The number above each pair is the difference in pass rate, English minus native.
Figure 7: Reasoning ablation. Solid lines mark significant changes with p<.05 in a two-sided paired permutation test.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Task ID
Category
Description
Difficulty
cs-implicit-alphabet-frequency
Cultural & Other
Count occurrences of each Czech alphabet letter in a text without being told what the Czech alphabet is — the agent must implicitly know all 42 letters including accented characters and the ‘ch’ digraph.
Medium
cs-nbsp-justify
Cultural & Other
Split Czech sentences into xelatex-justified lines with no Overfull hbox, while honoring Czech orthotypographic non-breaking-space and soft-hyphen rules.
Hard
cs-rozuctovani-tepla
Cultural & Other
Fix a script that audits a Czech SVJ heating-cost allocation against vyhlaska 269/2015 Sb.
Medium
de-bgb-citation-resolver
Cultural & Other
Resolve a list of German civil-code citations (e.g. § 433 Abs. 1 Satz 2) against a BGB XML file. Parse the hierarchical legal-text format, count Sätze per Absatz while respecting period-terminated legal abbreviations (Abs., Nr., S., gem., i.V.m.), and return each citation’s exact text as JSON.
Medium
de-umsatzsteuer-klassifikation
Cultural & Other
Classify German products by statutory VAT rate (7% reduced or 19% standard) under §12 UStG and Anlage 2 (Stand 1.1.2026), fixing a broken keyword draft. Many names are German compounds where the Grundwort sets the rate while the Bestimmungswort misleads, so decompose the compound first.
College of Computer Science and Software Engineering, Shenzhen University, China · Guangzhou Institute of Technology, Xidian University, China · Guangzhou University of Software, China +2