Multilingual intent classification is central to customer-service systems on global logistics platforms, where models must process noisy user queries across languages and hierarchical label spaces. Yet most existing multilingual benchmarks rely on machine-translated text, which is typically cleaner and more standardized than native customer requests and can therefore overestimate real-world robustness. We present a public benchmark for hierarchical multilingual intent classification constructed from real logistics customer-service logs. The dataset contains approximately 30K de-identified, stand-alone user queries curated from 600K historical records through filtering, LLM-assisted quality control, and human verification, and is organized into a two-level taxonomy with 13 parent and 17 leaf intents. English, Spanish, and Arabic are included as seen languages, while Indonesian, Chinese, and additional test-only languages support zero-shot evaluation. To directly measure the gap between synthetic and real evaluation, we provide paired native and machine-translated test sets and benchmark multilingual encoders, embedding models, and small language models under flat and hierarchical protocols. Results show that translated test sets substantially overestimate performance on noisy native queries, especially for long-tail intents and cross-lingual transfer, underscoring the need for more realistic multilingual intent benchmarks.
Figures & tables
Figure 1: Construction pipeline for the benchmark.
Figure 2: Top- k label coverage for the parent and leaf tasks. The 17-way leaf setting is substantially more long-tailed than the 13-way parent setting.
Model
13-way
17-flat
17-hier
Multilingual encoders
mBERT
92.62
91.38
N/A
XLM-R
94.00
91.85
N/A
mmBERT
94.69
93.15
N/A
Embedding-based classifiers
EmbeddingGemma
94.85
93.38
N/A
Table 1: Seen-language supervised benchmark results on English, Spanish, and Arabic (Accuracy/Micro-F1, %).
Model
Lang.
13-way Parent
17-way Flat
17-way Hier
Trans.
Native
Trans.
Native
Trans.
Native
mBERT
ZH
64.84
68.10
68.40
65.13
N/A
N/A
ID
54.60
53.41
52.82
51.63
N/A
N/A
TH
19.29
17.80
20.03
18.84
N/A
N/A
XLM-R
ZH
90.36
87.54
88.43
85.61
N/A
N/A
ID
92.58
90.36
90.80
88.72
N/A
N/A
Table 2: Synthetic-to-native robustness gap under same-language evaluation (Accuracy/Micro-F1, %). Models are evaluated on translated (Trans.) versus native user queries across varying task granularities. Bold values indicate the best performance within each architectural category (Masked LMs vs. Causal LLMs). Underlined values with ↑ explicitly highlight instances where the lightweight Gemma 3 270M outperforms the similarly-sized, classification-specialized mmBERT . Notably, Causal LLMs support the 17-way hierarchical (Hier) formulation natively, whereas implementing hierarchy-aware constrained decoding for standard BERT-style classifiers is typically more involved than flat classification.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Value
Raw source records
∼ 600K historical human customer-service records
Released real labeled queries
∼ 30K
Primary split ratio
train/dev/test = 20:1:1
Seen benchmark languages
English, Spanish, Arabic
Unseen test-only languages
Indonesian, Chinese, and additional held-out languages
Native multilingual evaluation set
1,300 examples, with ES:EN:AR = 2:1:1
Appendix
Table 3: Benchmark overview. Precise per-language counts can be inserted once the final release manifest is fixed.
Leaf ID
Leaf intent
Parent ID
Parent intent
0
Human Agent
0
Human Agent
1
Shipment Tracking / Delay
1
Shipment Tracking / Delay
2
Delivery Failure / Status Dispute
2
Delivery Failure / Status Dispute
3
Change Delivery Details / Method / Time
3
Delivery Change / Intercept / Return
4
Issue Report (No Compensation Request)
4
Damage / Missing / Confirmed Loss
5
Service Scope / Transit Time / Working Hours
5
Service Scope / Transit / Branch
Appendix
Table 4: Classification label names and leaf-to-parent mapping for the released hierarchy.
Figure 3: Comprehensive evaluation of clustering configurations. ( Left ) Normalized overlay of six clustering metrics across K∈{10,…,19} . All metrics are scaled to [0,1] and direction-aligned so that higher values indicate better clustering quality. The black line represents the aggregate score, which peaks at K=13 (indicated by the red dashed line). ( Right ) Radar chart comparing the multivariate performance profiles of the top-six K candidates. K=13 achieves the highest aggregate score ( 0.672 ) and exhibits the most balanced, well-rounded performance across the evaluated dimensions.
Model
13-way Macro
17-flat Macro
17-hier Macro
Multilingual encoders
mBERT
90.85
88.75
N/A
XLM-R
92.86
90.15
N/A
mmBERT
93.82
92.10
N/A
Embedding-based classifiers
EmbeddingGemma
94.12
92.35
N/A
Appendix
Table 5: Companion Macro-F1 table for Table 1 on seen languages (%).
Model
17-hier Parent Macro
Instruction-tuned SLMs
Gemma 3 270M
93.22
Qwen 3 0.6B
93.55
Gemma 3 1B
94.61
closed-source LLMs
Seed-1.8
93.63
Appendix
Table 6: Parent-level Macro-F1 in 17-way hierarchical evaluation for models with hierarchical decoding (%).
Model
Lang.
13-way Parent
17-way Flat
17-way Hier
Trans.
Native
Trans.
Native
Trans.
Native
mBERT
ZH
57.35
61.89
61.89
59.93
N/A
N/A
ID
53.56
52.93
51.78
50.64
N/A
N/A
TH
17.62
15.66
18.04
17.93
N/A
N/A
XLM-R
ZH
87.13
82.29
84.74
80.99
N/A
N/A
ID
90.73
88.85
88.14
85.96
N/A
N/A
Appendix
Table 7: Companion F1 table for Table 2 under same-language translated/native evaluation. Each cell is Macro-F1 (%).
Figure 4: Representative XML-style system prompt used for generation-based intent classification.