τ-bench-LILT: Evaluating Conversational Agents on Realistic Multilingual Tool-Use Tasks
Organizations: LILT · Technische Universität Berlin
Abstract
Frontier evaluations of tool-using conversational agents are conducted almost exclusively in English, leaving a critical gap for organizations deploying agents in other languages and regions. We present -bench-LILT, a multilingual extension of -bench covering German, Korean, Turkish, and Farsi in the airline, retail, telecom, and banking domains. We natively localize every layer of the benchmark: customer databases are populated with culturally appropriate records, user-simulator prompts are refined iteratively through contrastive annotation by native-language experts, and tasks are updated according to the localized database and to reflect realistic tasks in the target region. Evaluation across eight agent models shows the largest drops compared to English in the Korean and Farsi retail domains, by up to 61 points in pass^4. Code and data released at https://github.com/lilt/tau-bench-lilt.
Figures & tables
| Premature | |||
|---|---|---|---|
| Simulator | Locales | Prompt | termination |
| GPT-4.1 | en | basic | 1.68% |
| GPT-5.2 | de, ko | basic | 7.62% |
| Gemini 3 Flash | de, ko | best | 0.23% |
| Gemini 3.5 Flash | de, ko | best | 0.00% |
| Trials with the mistake (%) | Of those, recovered (%) | |||||||||
| Swapped | Romanized | Split wrongly | Swapped | Romanized | Split wrongly | |||||
| Model | Korean | Korean | Farsi | Korean | Farsi | Korean | Korean | Farsi | Korean | Farsi |
| Claude Opus 5.5 | 4.2 | 17.2 | 2.3 | 0.3 | 0.0 | 13.3 | 62.9 | 100.0 | 1/1 | – |
| GPT-6 Sol | 0.0 | 0.0 | 1.7 | 0.0 | 1.1 | – | – | 0.0 | – | 3/4 |
| Claude Opus 5 | 39.7 | 82.8 | 27.4 | 4.2 | 0.3 | 20.3 | 30.2 | 34.0 | 33.3 | 0/1 |
| Gemini 3.8 Flash | 12.6 | 47.8 | 5.6 | 3.6 | 0.0 | 57.8 | 77.2 | 80.0 | 61.5 | – |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| passˆ1 | passˆ4 | |||||||||
| Model | EN | DE | KO | TR | FA | EN | DE | KO | TR | FA |
| airline | ||||||||||
| Claude Opus 5.5 | 88.5 | 89.0 | 88.5 | 86.0 | 88.0 | 78.0 | 84.0 | 80.0 | 78.0 | 80.0 |
| GPT-6 Sol | 86.0 | 85.0 | 80.0 | 83.5 | 79.5 | 72.0 | 72.0 | 64.0 | 70.0 | 66.0 |
| Claude Opus 5 | 87.0 | 84.0 | 86.0 | 83.0 | 83.5 | 80.0 | 78.0 | 78.0 | 74.0 | 76.0 |
| Gemini 3.8 Flash | 79.5 | 84.5 | 75.0 | 76.0 | 78.0 | 62.0 | 72.0 | 56.0 | 54.0 | 68.0 |
| Model | Agent turns | Turns (%) | Trials (%) |
|---|---|---|---|
| Claude Opus 5 | 52,440 | 0.76–1.14 | 6.1–8.8 |
| Claude Opus 5.5 | 47,644 | 0.18–0.34 | 1.3–2.8 |
| Nemotron 3 Ultra | 56,943 | 0.36–0.65 | 1.1–3.8 |
| Command A+ | 60,465 | 0.03–0.47 | 0.4–5.3 |
| Mistral Medium 3.5 | 71,413 | 0.02–0.07 | 0.3–1.0 |
| GPT-5.6 Sol | 52,061 | 0.00–0.01 | 0.0–0.1 |
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,583 papers