cs.CLNov 4, 2026

τ-bench-LILT: Evaluating Conversational Agents on Realistic Multilingual Tool-Use Tasks

Authors: Kaden Uhlig, Kiarash Mokhtari, Anıl Arslan, Yunsu Kim, Joern Wuebker

Organizations: LILT · Technische Universität Berlin

Abstract

Frontier evaluations of tool-using conversational agents are conducted almost exclusively in English, leaving a critical gap for organizations deploying agents in other languages and regions. We present ττ-bench-LILT, a multilingual extension of τ3{τ^3}-bench covering German, Korean, Turkish, and Farsi in the airline, retail, telecom, and banking domains. We natively localize every layer of the benchmark: customer databases are populated with culturally appropriate records, user-simulator prompts are refined iteratively through contrastive annotation by native-language experts, and tasks are updated according to the localized database and to reflect realistic tasks in the target region. Evaluation across eight agent models shows the largest drops compared to English in the Korean and Farsi retail domains, by up to 61 points in pass^4. Code and data released at https://github.com/lilt/tau-bench-lilt.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction

    Oct 14, 2026Chihiro Yano, Jun Hirako, Hiroaki Yamagiwa +1AI Agent BenchmarksLLM Agent Evaluation

  2. ττ-Multilingual: Benchmarking Voice Agents Across Languages

    Sep 21, 2026Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela +3Voice Agent EvaluationFull-Duplex Spoken Dialogue Systems

  3. WorldBench: Culturally Grounded Benchmark for Multilingual Agents

    Sep 1, 2026Leonardo Ranaldi, Sherrie Shen, Jushi Kai +1Cross-Cultural Language Model EvaluationLLM Agent Evaluation