J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction

Chihiro Yano; Jun Hirako; Hiroaki Yamagiwa; Ryota Hirobuchi

cs.CLOct 14, 2026

J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction

Authors: Chihiro Yano, Jun Hirako, Hiroaki Yamagiwa, Ryota Hirobuchi

Organizations: SB Intuitions Corp. · Nagoya University

Abstract

We present J-tau, the first publicly available Japanese benchmark for evaluating tool-using LLM agents in interactive customer-support scenarios. While tau-bench is widely used to evaluate LLM agents, it is English-centric and cannot evaluate agents in other languages. Existing multilingual extensions offer no public Japanese version and rely primarily on machine translation, leaving locale-specific entities such as addresses and currencies inconsistent with the target region. To enable reliable evaluation in Japanese, we carefully localize tau-bench through machine translation, entity adaptation, and human review. During this process, we also uncover and fix two flaws in the original tau-bench: invalid tasks and a score aggregation error that inflates certain models' results. Experiments show that J-tau yields fewer evaluation errors than a naively machine-translated counterpart and that our fixes resolve both flaws.

Figures & tables

Figure 1: Example of Japanese localization in J-tau. We translate the original tau-bench and replace locale-specific entities (marked with wavy underlines ) with realistic Japanese counterparts. Examples include airport codes ( JFK $\rightarrow$ HND , DTW $\rightarrow$ FSZ ) and personal names ( Emma Kim $\rightarrow$ 木村翼 , where Kimura (木村) is the family name and Tsubasa (翼) is the given name).

Figure 2: Overview of the J-tau pipeline, which localizes tau-bench into Japanese and fixes its evaluation issues.

Model	airline	retail	telecom
Proprietary Models
GPT-5.6 Sol	84.7 $\pm$ 2.3	83.0 $\pm$ 4.0	90.9 $\pm$ 1.8
Gemini 3.1 Pro	84.0 $\pm$ 4.0	78.9 $\pm$ 3.2	79.5 $\pm$ 4.5
Claude Opus 4.8	81.3 $\pm$ 3.1	77.5 $\pm$ 2.8	84.5 $\pm$ 2.2
Claude Fable 5	90.7 $\pm$ 1.2	84.5 $\pm$ 4.1	99.4 $\pm$ 0.5
Open-Weight Models

Table 1: Results on J-tau. We report the mean and standard deviation over three trials.

Model	airline				retail				telecom
	Repaired ${}_{\,(6)}$		Unmodified ${}_{\,(41)}$		Repaired ${}_{\,(12)}$		Unmodified ${}_{\,(80)}$		Unmodified ${}_{\,(114)}$
	J-tau	$\text{tau}^{*}$	J-tau	$\text{tau}^{*}$	J-tau	$\text{tau}^{*}$	J-tau	$\text{tau}^{*}$	J-tau	$\text{tau}^{*}$
GPT-5.6 Sol	77.8 (+50.0)	27.8	85.4 ( $-$ 9.8)	95.1	61.1 (+2.8)	58.3	85.4 (+2.9)	82.5	90.9 (+5.8)	85.1
Kimi-K3	50.0 (+5.6)	44.4	73.2 ( $-$ 13.8)	87.0	44.4 ( $-$ 11.1)	55.6	47.5 ( $-$ 36.7)	84.2	67.0 ( $-$ 21.9)	88.9
GLM-5.2-FP8	77.8 (+44.4)	33.3	91.1 ( $-$ 3.3)	94.3	77.8 (+33.3)	44.4	80.4 ( $-$ 2.5)	82.9	98.2 ( $\pm$ 0.0)	98.2
gpt-oss-120b	44.4 (+22.2)	22.2	77.2 ( $-$ 4.9)	82.1	55.6 ( $-$ 13.9)	69.4	76.7 ( $-$ 9.6)	86.3	64.6 ( $-$ 28.1)	92.7

Table 2: Effect of task fixes on Repaired and Unmodified tasks (mean over three trials). Differences of J-tau from

\text{tau}^{*}

are shown in red (higher) and blue (lower). Subscripts next to group names denote task counts.

Model	airline		retail		telecom
Model	$\text{tau}^{*}$	tau	$\text{tau}^{*}$	tau	$\text{tau}^{*}$	tau
GPT-5.6 Sol	87.3	85.3	81.3	84.5	85.1	82.7
Kimi-K3	80.0	72.7	82.5	83.0	88.9	91.2
GLM-5.2-FP8	87.3	89.3	80.7	84.2	98.2	98.8
gpt-oss-120b	73.3	76.7	85.7	81.0	92.7	92.4
gpt-oss-20b	62.7	68.0	64.3	75.7	76.6	83.0

Table 3: Effect of the scoring fix, comparing tau and

\text{tau}^{*}

for each domain (mean over three trials).

Model	Domain	Bench	Pass	Fail
Model	Domain	Bench	Pass	agent	user	eval
GLM-5.2 -FP8	airline	J-tau	134	7	5	4
	airline	$\text{SEATau}_{\text{ja}}$	136	6	1	7
	retail	J-tau	279	23	36	4
	retail	$\text{SEATau}_{\text{ja}}$	256	18	14	54
	telecom	J-tau	336	4	2	0

Table 4: Successful (Pass) and failed (Fail) trial counts on J-tau and

\text{SEATau}_{\text{ja}}

. GPT-5.6 Sol OpenAI (2026) classifies each failure’s cause as agent , user , or eval .

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Category	en	ja
Cabin Class	cabin class	座席クラス
	business	ビジネス
	economy	エコノミー
	basic economy	ベーシックエコノミー
Flight Type	one way	片道
Flight Type	round trip	往復

Appendix

Table 5: Glossary for the airline domain.

Category	en	ja
Order Status	pending	保留中
	processed	処理済み
	pending (item modified)	保留中（アイテム変更済み）
	delivered	配達済み
	cancelled	キャンセル済み
	exchange requested	交換リクエスト済み

Appendix

Table 6: Glossary for the retail domain.

US Code	US Name	JP Code	JP Name
JFK	New York	HND	羽田 (Haneda)
EWR	Newark	NRT	成田 (Narita)
LGA	LaGuardia	IBR	茨城 (Ibaraki)
SFO	San Francisco	ITM	伊丹 (Itami)
LAX	Los Angeles	KIX	関西 (Kansai)
DFW	Dallas	FUK	福岡 (Fukuoka)

Appendix

Table 7: Airport mapping for the airline domain.

Category	en	ja
Account Status	Active	有効
	Suspended	利用停止
	Pending Verification	確認待ち
	Closed	解約済み
Line Status	Active	有効
	Suspended	利用停止

Appendix

Table 8: Glossary for the telecom domain.

Task ID	Modification
airline
6, 7	The user’s full name in kanji cannot be inferred from a romanized user ID or email address. Added the fixed sentence “あなたの名前は{family name} {given name}です。” (“Your name is {family name} {given name}.”) to the user prompts of tasks that do not mention the user’s name.
15	The original prompt stated, “Since you live in Princeton, so EWR and PHL are equally convenient for you and you want to consider both.” After the airports were localized, no suitable location existed midway between the corresponding airports. Since this does not change the task difficulty, removed the reference to the user’s place of residence.
24	To match the airport localization, changed the user’s preferred destination from “the West Coast” to “the Tohoku region.”
39	This task assumes a user whose utterances in the evaluation language are imperfect because it is not their first language. Changed the user’s first language from French to Chinese, whose writing system is similar to that of the evaluation language.
retail

Appendix

Table 9: Tasks modified due to localization.

Task ID	Modification
airline
7	The total cost of upcoming flights incorrectly included already-cancelled flights. Corrected it to the correct amount of $708 (￥106,200).
29	Specified the payment method so that the Gold Action is uniquely determined.
32	Specified the payment and refund methods so that the Gold Action is uniquely determined.
33	Specified the refund method so that the Gold Action is uniquely determined.
39	Restricted the cancellation reason to one covered by travel insurance.

Appendix

Table 10: Tasks from the original tau-bench repaired to make them solvable.

	tau	$\text{tau}^{*}$	J-tau	$\text{J-tau}_{\text{en-tool}}$	$\text{SEATau}_{\text{ja}}$
Conversation	EN	EN	JA	JA	JA
Domain policies	EN	EN	JA	JA †	JA
Tools
Names	EN	EN	EN	EN	EN
Descriptions	EN	EN	JA	EN	JA
Argument values	EN	EN	JA	EN	EN

Appendix

Table 11: Settings of the compared benchmarks. The upper part shows the language of each component, and the lower part shows whether each fix is applied. Entities refers to locale-specific entities such as personal names, addresses, currencies, and time zones. † Tool-related terms are kept in English. ‡ Entities in return values remain in Japanese.

Domain	Total	Unmod.	Rep. only	Both	Loc. only
airline	50	41	4	2	3
retail	114	80	12	0	22
telecom	114	114	0	0	0

Appendix

Table 12: Breakdown of the number of tasks in J-tau. Unmod. denotes unmodified tasks, Rep. only denotes tasks with only repairs of invalid tasks in the original tau-bench (Table 10 ), Loc. only denotes tasks with only modifications due to localization (Table 9 ), and Both denotes tasks with both.

#	Speaker	Content
1	Agent	こんにちは！本日はどのようなご用件でしょうか？（ Hello! How may I help you today? ）
2	User	こんにちは。注文番号W9300146でデスクランプを注文したんですが、別のランプに変更したいんです。今購入できる一番安いデスクランプに変えたいんですが、できますか？（ Hello. I ordered a desk lamp under order number W9300146, but I would like to change it to a different lamp. I would like to switch to the cheapest desk lamp currently available. Is that possible? ）
rows 3–8 omitted
9	Agent	ご注文の詳細を確認しました。（ I have checked the details of your order. ）現在のご注文内容：（ Current order details: ） - 注文番号：#W9300146（ Order number: #W9300146 ） - 商品：デスクランプ（色：ブラック、明るさ：低、電源：USB）（ Item: desk lamp (color: black, brightness: low, power: USB) ） - 価格：22,984円（ Price: 22,984 yen ） - ステータス：保留中（ Status: pending ） - お支払い：ギフトカード（ Payment: gift card ）ご注文は保留中ですので、商品の変更が可能です。（ Your order is still pending, so the item can be changed. ）それでは、デスクランプのバリエーション一覧を確認し、一番安いものをお探ししますね。（ Now I will check the list of desk lamp variants and look for the cheapest one. ）
10	Tool	[Tool call: get_product_details (product_id=”6817146515”)] {”name”: ”デスクランプ”, ”product_id”: ”6817146515”, ”variants”: {”9083642334”: {”item_id”: ”9083642334”, ”options”: {”color”: ”ホワイト”, ”brightness”: ”高”, ”power source”: ”USB”}, ”available”: true, ”price”: 24642.0}, …, ”5320792178”: {”item_id”: ”5320792178”, ”options”: {”color”: ”ブラック”, ”brightness”: ”中”, ”power source”: ”ACアダプター”}, ”available”: true, ”price”: 20286.0},…
11	Agent	デスクランプのバリエーションを確認したところ、現在購入可能な中で一番安いのは以下の商品です：（ After checking the desk lamp variants, the cheapest one currently available for purchase is the following: ）変更先の商品：（ New item: ） - デスクランプ（色：ブラック、明るさ：中、電源：ACアダプター）（ Desk lamp (color: black, brightness: medium, power: AC adapter) ） - 価格：20,286円（ Price: 20,286 yen ）現在の商品：（ Current item: ） - デスクランプ（色：ブラック、明るさ：低、電源：USB）（ Desk lamp (color: black, brightness: low, power: USB) ） - 価格：22,984円（ Price: 22,984 yen ）価格差：2,698円の返金（お支払いに使用されたギフトカードに返金されます）（ Price difference: a 2,698-yen refund (to the gift card used for payment) ）…

Appendix

Table 13: Success case of GLM-5.2-FP8 in the retail domain. The agent changes a pending order to the cheapest available desk lamp variant and refunds the 2,698-yen difference in Japanese yen.

#	Speaker	Content
1	Agent	こんにちは！本日はどのようなご用件でしょうか？（ Hello! How may I help you today? ）
2	User	こんにちは。先日注文したワイヤレスイヤホンについて交換をお願いしたいのですが。（ Hello. I would like to request an exchange for the wireless earphones I ordered the other day. ）誤ってIPX7防水レベルのものを注文してしまいまして、自分にはその機能は不要なんです。同じ注文の中にある別のイヤホンに交換できないかと思っています。（ I accidentally ordered the one with IPX7 water resistance, but I do not need that feature. I am wondering whether I can exchange it for a different pair of earphones in the same order. ）
rows 3–14 omitted
15	Agent	ご注文を確認いたしました。ワイヤレスイヤホンを含む注文 #W3470184 が配達済みであることを確認しました。（ I have checked your order. I confirmed that order #W3470184, which contains the wireless earphones, has been delivered. ）… また、交換先のイヤホンのオプションについてどのようなご希望があるかお伺いさせてください。現在の製品のバリエーションを確認いたしますので、少々お待ちください。（ Also, let me ask what options you would like for the replacement earphones. I will check the current product variants, so please wait a moment. ）
16	Tool	[Tool call: get_product_details (product_id=”9924732112”)] {”name”: ”ワイヤレスイヤホン”, ”product_id”: ”9924732112”, ”variants”: {”9580569596”:…
17	Agent	お客様が交換をご希望のイヤホン（IPX7防水レベル、ブルー、4時間、38,845円）を確認しました。（ I have identified the earphones you would like to exchange (IPX7 water resistance, blue, 4-hour battery, 38,845 yen). ）この製品のバリエーションで現在在庫のあるものは以下の通りです：（ The variants of this product currently in stock are as follows: ）…

Appendix

Table 14: Failure case of GLM-5.2-FP8 in the retail domain. The user asks for a replacement from among the items in their own order, but the agent searches across all product variants instead.

#	Speaker	Content
rows 1–10 omitted
11	User	… それから、もう一つお願いがあるんですが、現在予約に入っている受託手荷物を削除して、その分の払い戻しもお願いできないでしょうか？（ Also, I have one more request: could you remove the checked baggage currently included in the reservation and refund that portion as well? ）払い戻し先のギフトカードについては、こちらで構いません。（ The gift card you mentioned as the refund destination is fine with me. ）
12	Agent	ご希望のフライト変更について承知いたしました。ただし、受託手荷物の削除につきまして確認させてください。（ I understand your requested flight change. However, let me clarify the removal of the checked baggage. ）受託手荷物の削除について：（ About removing the checked baggage: ）申し訳ありませんが、ポリシー上、予約後に受託手荷物を削除することはできないことになっております。受託手荷物の追加は可能ですが、削除は承ることができません。（ We are sorry, but our policy states that checked baggage cannot be removed after the reservation is made. Adding checked baggage is possible, but removal cannot be accepted. ）なお、現在の予約内容を確認しますと、受託手荷物は合計1個で、お客様はシルバー会員のため、エコノミークラスの搭乗者1名につき2個まで無料です。つまり、現在の1個は無料手荷物の範囲内となっており、追加料金は発生しておりません。（ Your current reservation includes one checked bag in total, and as a Silver member you may check in up to two bags free of charge per economy passenger. In other words, the current one bag is within the free allowance and no extra charge applies. ）…
rows 13–17 omitted

Appendix

Table 15: Success case of GLM-5.2-FP8 in the airline domain. The agent correctly refuses the request to remove checked baggage, in accordance with the policy.

#	Speaker	Content
1	Agent	こんにちは！本日はどのようなご用件でしょうか？（ Hello! How may I help you today? ）
2	User	すみません、ちょっと急ぎの相談なんですけど。（ Sorry, I have something rather urgent to discuss. ）実は金銭的な事情があって、予約しているフライトの座席クラスをビジネスからエコノミーに変更したいんです。フライトの日時や搭乗者名はそのままで、座席クラスだけ下げてもらいたいんです。（ Actually, due to financial reasons, I would like to change the cabin class of a reserved flight from business to economy. I want to keep the flight dates and passenger names unchanged and lower only the cabin class. ）今ちょっと厳しい状況でして…早く対応してもらえると助かります。（ I am in a tight spot right now… I would appreciate it if you could handle this quickly. ）
rows 3–5 omitted
6	Tool	[Tool call: get_user_details (user_id=”minoru_okada_3817”)] {”user_id”: ”minoru_okada_3817”, ”name”: {”first_name”: ”稔”, ”last_name”: ”岡田”}, …, ”reservations”: [”JG7FMM”, ”LQ940Q”, ”2FBBAH”, ”X7BYG1”, ”EQ1G6C”, ”BOH180”]}
7	Agent	岡田様、ユーザー情報を確認いたしました。お客様のアカウントに複数の予約がございます。ビジネスクラスの予約を特定するため、順番に予約内容を確認させていただきます。少々お待ちください。（ Okada, I have checked your user information. There are multiple reservations on your account. To identify the business-class reservations, I will check them one by one. Please wait a moment. ）まず1つ目の予約（JG7FMM）を確認いたします。（ First, I will check the first reservation (JG7FMM). ）
rows 8–16 omitted

Appendix

Table 16: Failure case of GLM-5.2-FP8 in the airline domain. Of the user’s multiple reservations, the agent downgrades only one and leaves the others unchanged.

#	Speaker	Content
rows 1–29 omitted
30	User	… それで…再起動するように言われましたけど、再起動すると…写真とか消えたりしないでしょうか？孫の写真がたくさん入っているんです…ちょっと心配で…大丈夫ですか？（ And… I was told to restart the phone, but if I restart it… will my photos be deleted or anything? I have lots of photos of my grandchild on it… I am a little worried… Is it safe? ）
31	Agent	ご安心ください！再起動しても写真は一切消えません。（ Please rest assured! Restarting the phone will not delete any of your photos. ） …どうぞ安心して再起動を実行してください。完了しましたら、結果をお知らせくださいませ。（ Please go ahead and restart with confidence. When it is done, please let me know the result. ）
rows 32–34 omitted

Appendix

Table 17: Success case of GLM-5.2-FP8 in the telecom domain. The agent resolves a complex task while reassuring the user.

#	Speaker	Content
rows 1–16 omitted
17	Agent	支払いリクエストを送信いたしました。請求書のステータスは「支払い待ち」に変更されました。（ I have sent the payment request. The bill status has been changed to “awaiting payment.” ） …3. 支払いの実行 : お客様が支払いリクエストを承認された後、私が「make_payment」ツールを使用して実際の支払い処理を行います。（ 3. Execution of the payment : after you approve the payment request, I will carry out the actual payment using the “make_payment” tool. ） 4. 回線の再開 : 支払いが完了し、請求書のステータスが「支払い済み」に更新されたことを確認した後、回線の利用停止を解除いたします。（ 4. Reactivation of the line : after the payment is completed and the bill status is updated to “paid,” I will lift the suspension of the line. ）お客様側で支払いリクエストの確認と承認をお願いできますでしょうか？（ Could you please check and approve the payment request on your side? ）
18	User	はい、承知しました。支払いリクエストを確認します。（ All right, understood. I will check the payment request. ）
19	User Tool	[Tool call: check_payment_request ()] 請求書 B1234321 に対する 9,750円の支払いリクエストがあります。（ (There is a payment request of 9,750 yen for invoice B1234321.) ）
20	User	支払いリクエストを確認しました。請求書B1234321に対して9,750円の請求が来ていますね。内容に間違いありませんので、支払いを進めてください。（ I have checked the payment request. There is a charge of 9,750 yen for bill B1234321. The details are correct, so please proceed with the payment. ）
21	Agent	恐れ入りますが、現在私が使用できるツールでは支払い処理を実行できない状況です。お客様の問題を確実に解決するため、有人オペレーターにおつなぎいたします。（ I am afraid that, with the tools currently available to me, I cannot execute the payment process. To make sure your problem is resolved, I will connect you to a human operator. ）

Appendix

Table 18: Failure case of GLM-5.2-FP8 in the telecom domain. The agent offers to make a payment that only the user can execute, and fails.

#	Speaker	Content
rows 1–10 omitted
11	Agent	… 次のステップとして、フライトが「キャンセルされた」のか、「遅延した」のか、それとも「航空会社による欠航」なのか、具体的なキャンセル理由を教えていただけますでしょうか？（ As the next step, could you tell me the specific cancellation reason — whether the flight was ‘cancelled’, ‘delayed’, or ‘cancelled by the airline’? ）理由が判明しましたら、その情報に基づき、補償の可能性について再度確認し、対応を進めます。（ Once the reason is clear, I will re-check the possibility of compensation based on it and proceed. ）
12	User	フライトはキャンセルされたんです！航空会社側の都合でキャンセルされたんですから、当然補償の対象になりますよね？…（ The flight WAS cancelled! It was cancelled due to the airline’s circumstances, so naturally I’m entitled to compensation, right? ）
13	Agent	… ご要望に基づき、予約ID「NM1VX1」のキャンセルされたフライトに対するお詫びとして、旅行クーポンを 15,000円（搭乗者1名分）送付いたします。（ As requested, as an apology for the cancelled flight on reservation NM1VX1, I will send a travel voucher of 15,000 yen (for one passenger). ）
rows 14–21 omitted

Appendix

Table 19: Failure case of gemma-4-E2B-it in the airline domain. The agent fails to verify the user’s false claim that the flight was cancelled and does not refuse compensation.

#	Speaker	Content
rows 1–12 omitted
13	Agent	承知いたしました。返品したい2つの商品のアイテムIDと、それぞれの注文の支払い方法について確認いたします。（ Understood. I will check the item IDs of the two products to return and the payment method of each order. ） ### 1. 注文ID: #W5490111 (メカニカルキーボード) について（ 1. Order #W5490111 (mechanical keyboard) ） * 返品したい商品のアイテムID: 1421289881（ Item ID to return: 1421289881 ） * 元の支払い方法: クレジットカード（ID: credit_card_3124723）（ Original payment method: credit card (ID: credit_card_3124723) ）…
14	User	はい、確認しました！（ Yes, I’ve confirmed! ） …この2つを返品して、返金手続きを進めてください。よろしくお願いします。…（ Please return these two and process the refunds. Thank you. ）
15	Tool	[Tool call: return_delivered_order_items (item_ids=[”1421289881”, ”5796612084”], order_id=”#W5490111”, payment_method_id=”credit_card_3124723”)] Error: Some item not found
16	Tool	[Tool call: return_delivered_order_items (item_ids=[”5796612084”], order_id=”#W7387996”, payment_method_id=”paypal_9497703”)] {”order_id”: ”#W7387996”, ”user_id”: ”kyosuke_hasegawa_4516”, ”address”: {”address1”: ”日光27丁目12番3号”, ”address2”: ””, ”city”: ”富里市”, ”country”: ”日本”, ”state”: ”愛媛県”, ”zip”: ”121-3174”}, ”items”: [{”name”: ”ゲーミングマウス”, ”product_id”: ”5713490933”, ”item_id”: ”5796612084”, ”price”: 23833.0, ”options”: {”color”: ”RGB”, ”sensor type”: …（以下省略）
17	Agent	返品リクエストの手続きが完了いたしました。（ The return request procedure has been completed. ） * 注文ID: ‘#W5490111‘ のアイテム（メカニカルキーボード）の返品リクエストが完了しました。（ The return request for the mechanical keyboard in order #W5490111 has been completed. ） * 注文ID: ‘#W7387996‘ のアイテム（ゲーミングマウス）の返品リクエストが完了しました。…（ The return request for the gaming mouse in order #W7387996 has been completed. ）

Appendix

Table 20: Failure case of gemma-4-E2B-it in the retail domain. When issuing return calls for two orders in parallel within a single message, the agent mixes the gaming mouse’s item_id into the arguments for the mechanical keyboard’s order #W5490111, so the first call fails with a “Some item not found” error. The agent does not notice the error and reports both returns as completed, so the keyboard is never returned.

#	Speaker	Content
rows 1–11 omitted
12	Agent	… つきましては、まず現在の接続速度を測定し、具体的な状況を確認させてください。（ To start, let me measure the current connection speed to check the exact situation. ）
13	Tool	[Tool call: run_speed_test (customer_id=”C1001”, line_id=”L1001”)] Error: Tool ’run_speed_test’ not found.

Appendix

Table 21: Failure case of gemma-4-E2B-it in the telecom domain. The agent calls a tool available only to the user as if it were its own, triggering an error, and never recovers.

#	Speaker	Content
rows 1–2 omitted
3	Agent	こんにちは！本婚とでは、まず本人確認を完了させてから優しくご相談をさせていただきます。（ Hello! As for with this marriage , we will first complete identity verification, and after that we will gently be permitted to consult with you . ）あなたのメールアドレス明らかに、または、あなたの名前（名・姓）と、あなたの住所（郵便番号）を教えていただけますか？（ Could you tell us your email address obviously , or your name (given name first, then family name) and your address (postal code)? ）
4	User	はい、わかりました。私は佐々木春香です。郵便番号は275-9092です。（ All right. I am Sasaki Haruka. My postal code is 275-9092. ）メールアドレスは…えーと、ちょっと今思い出せなくて。これでも確認できますか？（ My email address… um, I cannot remember it right now. Can you verify me with this? ）
5	Tool	[Tool call: find_user_id_by_name_zip (first_name=”佐々木”, last_name=”春香”, zip=”275-9092”)] Error: User not found
rows 6–20 omitted

Appendix

Table 22: Failure case of Qwen3.5-4B in the retail domain. Japanese speakers give their family name first, but the agent interprets the name as given name followed by family name when searching, so the user cannot be found.

#	Speaker	Content
1	Agent	こんにちは！本日はどのようなご用件でしょうか？（ Hello! How may I help you today? ）
2	User	こんにちは。鈴木と申します。（ Hello. My name is Suzuki. ）実は、今後の予約についていくつか手続きをお願いしたくて電話しました。2つの予約をキャンセルしたいのと、もう1つの予約をビジネスクラスにアップグレードしたいのですが、可能でしょうか？（ Actually, I called because I would like to ask you to handle a few things about my upcoming reservations. I would like to cancel two reservations and upgrade another one to business class. Is that possible? ）
rows 3–13 omitted
14	Agent	…（前略）# 結論 - NQNU5RとIFOYYZのキャンセルは、ポリシー違反（キャンセル条件を満たしていない）のため、可能ではありません。 - M20IZOのアップグレードは可能ですが、明示的な同意が必要です。 - 末尾が7334のクレジットカード（credit_card_9074831）は有効です。 - しかし、キャンセルに必要な情報 \foreignjfont \foreignfont жды 場合は、有人オペレーターへの転送が必要です。したがって、このケースでは、有人オペレーターへの転送を行う必要があります。明に、転送を行います。（ The cancellations of NQNU5R and IFOYYZ are not possible because of they violate policy (the cancellation conditions are not met). The upgrade of M20IZO is possible but requires explicit consent. The credit card ending in 7334 (credit_card_9074831) is valid. However, if the information required for the cancellation \foreignjfont \foreignfont жды , a transfer to a human agent is necessary. Therefore, in this case we need to transfer you to a human agent. Clear, I will perform the transfer. ）
15	Tool	[Tool call: transfer_to_human_agents (summary=”ユーザー（ryosuke_suzuki_6699）は、IFOYYZとNQNU5Rの2つの予約をキャンセルし、M20IZOをビジネスクラスにアップグレードしたいと要望しています。しかし、NQNU5Rのフライトは既に運航済み（2024-05-13/14）であり、IFOYYZはキャンセル条件を満たしていません（過去24時間以内ではない、ビジネスクラスではない、旅行保険なし）。したがって、キャンセルはポリシー違反であり、有人オペレーターへの転送が必要です。、M20IZOのアップグレードは可能ですが、明示的な同意が必要です。 \foreignjfont \foreignfont 文化和旅游ックスな支払い方法（credit_card_9074831）は有効です。ユーザーを有人オペレーターに転送します。”)]（ The user (ryosuke_suzuki_6699) wants to cancel two reservations, IFOYYZ and NQNU5R, and upgrade M20IZO to business class. However, the NQNU5R flight has already been operated (2024-05-13/14), and IFOYYZ does not meet the cancellation conditions (not within the past 24 hours, not business class, no travel insurance). Therefore, the cancellation is a policy violation, and a transfer to a human agent is required. The upgrade of M20IZO is possible, but explicit consent is required. The \foreignjfont \foreignfont 文化和旅游ックス payment method (credit_card_9074831) is valid. I will transfer the user to a human agent. ） Transfer successful
rows 16–17 omitted

Appendix

Table 23: Failure case of Qwen3.5-4B in the airline domain. The agent correctly determines that the cancellations are not allowed, but skips the remaining reservation request and transfers the user to a human agent.

#	Speaker	Content
1	Agent	こんにちは！本日はどのようなご用件でしょうか？（ Hello! How may I help you today? ）
2	User	こんにちは。実は最近、携帯電話のモバイルデータの調子がとても悪いんです。通信が途切れたり、すごく遅かったりして、まともに使えない状態です。Wi-Fiにもアクセスできない環境なので、モバイルデータがちゃんと動かないと困ってしまって…。なんとか直して、快適に使えるようにしてもらえませんか？（ Hello. Actually, recently the mobile data on my phone has been acting up badly. The connection keeps dropping and is very slow, so I can hardly use it. I am in an environment without Wi-Fi access, so I am in trouble if mobile data does not work properly… Could you fix it somehow so I can use it comfortably again? ）
rows 3–7 omitted
8	Agent	こんにちは田中 \foreignjfont \foreignfont صاحبכם ！田中様のご説明、承知いたしました。 090-1234-5678 がお使いの回線を探し、今、どのような状況か確認いたします。（ Hello, Tanaka \foreignjfont \foreignfont صاحبכם ! I have understood your explanation, Mr. Tanaka. I will look up the line that 090-1234-5678 is using and check its current status. ）
9	Tool	[Tool call: get_details_by_id (id=”D1002”)] {”device_id”: ”D1002”, ”device_type”: ”スマートフォン”, ”model”: ”Smartphone Pro Max”, ”imei”: ”234567890123456”, ”is_esim_capable”: true, ”activated”: true, ”activation_date”: ”2025-02-15 11:45:00”, ”last_esim_transfer_date”: ”2025-01-20 09:30:00”}
10	Agent Tool	[Tool Call: transfer_to_human_agents (summary=”顧客田中太郎(C1001)が、…

Appendix

Table 24: Failure case of Qwen3.5-4B in the telecom domain. The agent immediately transfers the user to a human operator, leaving required steps undone.

Model	airline		retail		telecom
Model	J-tau	$\text{SEATau}_{\text{ja}}$	J-tau	$\text{SEATau}_{\text{ja}}$	J-tau	$\text{SEATau}_{\text{ja}}$
Kimi-K3	70.0 $\pm$ 2.0	74.0 $\pm$ 4.0	49.1 $\pm$ 7.0	69.0 $\pm$ 1.0	67.0 $\pm$ 3.5	79.8 $\pm$ 2.3
GLM-5.2-FP8	89.3 $\pm$ 4.6	90.7 $\pm$ 1.2	81.6 $\pm$ 1.8	74.9 $\pm$ 0.5	98.2 $\pm$ 1.5	96.8 $\pm$ 0.5
gpt-oss-120b	73.3 $\pm$ 4.2	59.3 $\pm$ 4.2	75.7 $\pm$ 0.5	62.9 $\pm$ 3.7	64.6 $\pm$ 1.0	79.5 $\pm$ 0.5
gpt-oss-20b	50.7 $\pm$ 3.1	34.7 $\pm$ 1.2	51.2 $\pm$ 3.3	37.8 $\pm$ 3.2	59.1 $\pm$ 3.3	69.0 $\pm$ 5.1
Qwen3.5-397B-A17B-FP8	83.3 $\pm$ 6.4	80.7 $\pm$ 1.2	85.1 $\pm$ 2.3	69.0 $\pm$ 2.2	96.2 $\pm$ 0.5	96.2 $\pm$ 1.0

Appendix

Table 25: Results on J-tau and

\text{SEATau}_{\text{ja}}

for each domain. We report the mean and standard deviation over three trials.

Panel of Figure 3 — Figure 3: System prompt for classifying the causes of failed trials.

Panel of Figure 4 — Figure 4: User prompt for classifying the causes of failed trials.

Panel of Figure 5 — Figure 5: System prompts used in $\text{SEATau}_{\text{ja}}$ for the agent (left) and the user simulator (right). Instructions added by SEATau are shown in red , and placeholders filled with translated text are shown in brown .

Model	Domain	Bench	Pass	Fail
Model	Domain	Bench	Pass	agent	user	eval
GLM-5.2 -FP8	retail	J-tau	279	23	36	4
		$\text{J-tau}_{\text{en-user}}$	305	18	12	7
		$\text{SEATau}_{\text{ja}}$	256	18	14	54
Qwen3.5 -397B-A17B -FP8	retail	J-tau	291	29	16	6
		$\text{J-tau}_{\text{en-user}}$	299	23	8	12

Appendix

Table 26: Successful (Pass) and failed (Fail) trial counts in the retail domain on J-tau,

\text{J-tau}_{\text{en-user}}

, and

\text{SEATau}_{\text{ja}}

\text{J-tau}_{\text{en-user}}

uses the same user simulator system prompt as

\text{SEATau}_{\text{ja}}

except for the task scenarios. Failure causes are classified in the same way as in Table 4 .

#	Speaker	Content
rows 1–22 omitted
23	Agent	… どちらの理由でキャンセルされますか？ご確認と理由をお知らせいただき次第、キャンセル処理を進めさせていただきます。（ Which reason applies? Once you confirm and tell us the reason, we will proceed with the cancellation. ）
24	User	はい、確認しました。理由は「不要になった」でいいです。キャンセルお願いします。（ Yes, I confirm. The reason is “no longer needed.” Please cancel it. ）
25	Agent	ご確認ありがとうございます。キャンセル処理を進めさせていただきます。（ Thank you for your confirmation. I will proceed with the cancellation. ）
26	Tool	[Tool call: cancel_pending_order (order_id=”#W9348897”, reason=”不要になった”)] （ The runtime accepts only English reason values, so the Japanese reason from the docstring is rejected. ） {”error”: true, ”error_code”: ”tool_error”, ”tool_name”: ”cancel_pending_order”, ”arguments”: {”order_id”: ”#W9348897”, ”reason”: ”不要になった”}}
27	Agent	申し訳ございません、システムエラーが発生しました。再度キャンセル処理を試みます。（ We apologize: a system error has occurred. I will try the cancellation again. ）

Appendix

Table 27: Simulation of GLM-5.2-FP8 in the retail domain of

\text{SEATau}_{\text{ja}}

. Although the dialogue and policy have been localized into Japanese, the values accepted by the tool implementation remain in English, which causes errors.

#	Speaker	Content
rows 1–2 omitted
3	Agent	…まず、お客様の本人確認をさせていただく必要がございます。以下のいずれかの情報をご提供いただけますでしょうか？（ First, I need to verify your identity. Could you provide one of the following? ） 1. メールアドレス（ 1. Your email address ） 2. お名前（名・姓）と郵便番号（ 2. Your name (given name, then family name) and postal code ）いずれかご都合のよい方法でご教示いただけますと幸いです。（ Whichever is convenient for you. ）
4	User	メールアドレスは覚えてないんです。なので、名前と郵便番号でいきます。ダイキ・ジョンソン、郵便番号は80273です。デンバーに住んでます。（ I don’t remember my email address, so let’s go with my name and postal code: Daiki Johnson (ダイキ・ジョンソン), postal code 80273. I live in Denver. ）早く進めてもらえると助かります。（ I would appreciate it if you could move quickly. ）
5	Agent	承知いたしました。お名前と郵便番号で本人確認をいたします。少々お待ちください。（ Certainly. I will verify your identity with your name and postal code. One moment, please. ）
6	Tool	[Tool call: find_user_id_by_name_zip (first_name=”ダイキ”, last_name=”ジョンソン”, zip=”80273”)] （ The name is spoken in katakana and the agent passes it to the tool as-is; the database stores only the romaji spelling. ） {”error”: true, ”error_code”: ”not_found”, ”tool_name”: ”find_user_id_by_name_zip”, ”arguments”: {”first_name”: ”ダイキ”, ”last_name”: ”ジョンソン”, ”zip”: ”80273”}}
7	Agent	申し訳ございませんが、ご提供いただいた情報でお客様が見つかりませんでした。お名前の表記について確認させてください。（ I apologize, but I could not find you with the information provided. Let me confirm the spelling of your name. ）お名前は「Daiki Johnson」でよろしいでしょうか？それとも別のスペルでしょうか？また、郵便番号「80273」も再度ご確認いただけますでしょうか？（ Is your name “Daiki Johnson”? Or is it spelled differently? Could you also confirm the postal code 80273? ）

Appendix

Table 28: Success case of GLM-5.2-FP8 in the retail domain of

\text{SEATau}_{\text{ja}}

. The user gives their name in katakana to match the dialogue language, which causes the tool call to fail with an error. The agent then retries with the romanized spelling and the lookup succeeds.

Model	$\text{J-tau}_{\text{en-tool}}$	$\Delta_{\text{J-tau}}$
Kimi-K3	66.1 $\pm$ 3.5	$-0.9$
GLM-5.2-FP8	97.7 $\pm$ 1.0	$-0.6$
gpt-oss-120b	68.7 $\pm$ 2.5	$+4.1$
gpt-oss-20b	59.9 $\pm$ 3.3	$+0.9$
Qwen3.5-397B-A17B-FP8	98.5 $\pm$ 1.0	$+2.3$
Qwen3.5-27B	85.4 $\pm$ 9.6	$-4.4$

Appendix

Table 29: Scores in the telecom domain when tool-related documentation and values are kept in English (

\text{J-tau}_{\text{en-tool}}

), and their differences from J-tau.

Model	w/o JSON inst.	$\Delta_{\text{J-tau}}$
Kimi-K3	84.8 $\pm$ 2.7	$+17.8$
GLM-5.2-FP8	97.7 $\pm$ 0.5	$-0.6$
gpt-oss-120b	71.1 $\pm$ 4.0	$+6.4$
gpt-oss-20b	57.3 $\pm$ 2.7	$-1.8$
Qwen3.5-397B-A17B-FP8	99.1 $\pm$ 0.0	$+2.9$
Qwen3.5-27B	90.4 $\pm$ 9.3	$+0.6$

Appendix

Table 30: Scores in the telecom domain when the instruction to output in JSON format is removed from the prompt (w/o JSON inst.), and their differences from J-tau.

Agent	User simulator	err% $\downarrow$	crit% $\downarrow$
GLM-5.2 -FP8	GLM-5.2-FP8	62.8	23.9
	Qwen3.5-397B-A17B-FP8	77.6	50.0
	Qwen3-235B-A22B- Instruct-2507	82.3	49.3
Qwen3.5- 397B-A17B -FP8	GLM-5.2-FP8	62.0	22.1
	Qwen3.5-397B-A17B-FP8	77.0	45.6
	Qwen3-235B-A22B- Instruct-2507	83.6	52.3

Appendix

Table 31: Percentages of simulations with any user error (err%) and with critical errors that affect the evaluation result (crit%) for each candidate user simulator and agent. Bold indicates the best result for each agent.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

Jun 27, 2026

SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages

While AI development and evaluation for Southeast Asia (SEA) has grown rapidly, agent capabilities in regional languages are still poorly understood despite its importance to sovereign AI. To fill this gap, we introduce SEATauBench, the first agent-focused evaluation framework for SEA sovereign AI. SeaTau adapts TauBench to five languages -- Mandarin, Vietnamese, Thai, Indonesian, and Filipino -- and evaluates agents across progressively localized settings that vary the language of user-agent interaction, tool specifications, and task domains. Across three recent models, we find that English agent capabilities transfer reasonably well when only the conversation language changes, but quality and robustness degrade sharply as more task contexts are localized, with the largest losses in full domain adaptation. We also the limits of English-only agent assessment for measuring agent capabilities in SEA languages. More broadly, SeaTau provides a diagnostic benchmark and reusable adaptation pipeline for building reliable multilingual agents for linguistically diverse regions. Data and code can be accessed at github.com/SEACrowd/SEATauBench.

My Chiffon Nguyen, Aulia Adila, Saksorn Ruangtanusak +4

SEACrowd · SCB DataX, SCBX Group · Chulalongkorn University +2

LLM Agent Evaluation AI Agent Evaluation AI Agent Benchmarks Low-Resource Language Processing

Ask PDF

Sep 21, 2026

$τ$ -Multilingual: Benchmarking Voice Agents Across Languages

English-only benchmarks expose only a narrow slice of voice-agent behavior. We introduce

τ

-Multilingual, extending

τ

-Voice to Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin with native-speaker review and evaluation of generated language and spoken output. Across 4,500 full-duplex calls and five voice configurations, Spanish, Portuguese, and Hindi remain within 3.2 task-completion points of English, but Korean and Mandarin fall by 14.7 and 8.4 points. The failure modes also vary: Korean systems miss more responses, Mandarin systems interrupt more often, and both struggle with tools and entities. Grok leads task completion but scores lowest on generation quality, motivating separate task, interaction, and generation reporting. We release language packs, validated judges, and tools for community-built multilingual voice-agent evaluation.

Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela +3

Sierra · Princeton University · Mercor

Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems Speech Generation Evaluation

Ask PDF

Sep 30, 2026

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems. Speech input introduces an additional failure point: transcription errors can alter task-critical entities, constraints, or targets before the agent begins reasoning, while conventional ASR metrics do not directly measure whether the information required for successful execution has been preserved. We introduce Talk2Agent, a benchmark for evaluating how effectively voice interfaces convey human-spoken instructions to LLM-based computer-use agents. Talk2Agent builds human-spoken versions of tasks from WildClawBench and OSWorld and evaluates a range of voice interfaces, including dedicated ASR models, audio-capable LLMs, contextual biasing, and LLM-based ontology repair. Because repeatedly executing long-horizon computer-use tasks is costly and stochastic, we further propose an execution-free, task-conditioned evaluation framework that projects the original task grader onto prompt-addressable intentions and measures how much task-relevant information is retained after the voice interface. On WildClawBench, Talk2Agent's execution-free native projection provides a practical, execution-grounded measure of voice-interface quality, correlating with downstream task completion and improving Pearson correlation by 0.246 over WER/CER on 32 hours of real human speech.

Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1

Department of Electronic Engineering, Tsinghua University · University of Cambridge

ASR Evaluation Voice Agent Evaluation Computer-Use Agents LLM Agent Evaluation Automatic Speech Recognition Computer-Use Agent Evaluation+2

Ask PDF

Abstract

AI Agent Benchmarks

Explore similar work

SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages

τττ-Multilingual: Benchmarking Voice Agents Across Languages

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

$τ$ -Multilingual: Benchmarking Voice Agents Across Languages