XDecideBench: Evaluating Typed Decision Models and Confidence Policies Across Languages
Organizations: Comenius University in Bratislava · Cisco Systems · Kempelen Institute of Intelligent Technologies
Abstract
Typed Decision Models such as Jev, Clef and pplx-decider return a probability for each answer an application provides. Applications use them in many languages, but are compared mostly in English. We introduce XDecideBench, a multilingual benchmark of nine suites in 122 languages. On our twelve-language suite, three open models fine-tuned for typed decisions beat their Qwen base models in every language. On new test data with local base models, their accuracy gain is larger in languages with less web text than in those with more. They fix a similar share of base-model errors in both, so the larger gain may reflect weaker base models. The best model on the twelve-language suite is best on only two other suites. The best English model on each suite, chosen in hindsight, is significantly beaten in about a quarter of suite and language combinations. On Belebele, Decision-1 is best in English but falls far behind Jev in Burmese. A confidence threshold fitted on English to allow 5% errors gives a higher error rate on XNLI, even in German and French. XDecideBench shows that results from one suite or from English are a poor guide to how a model and its threshold behave in other languages.
Figures & tables
| Study | Langs | DMs | Types | Same | P/N | H–B | Transf. |
|---|---|---|---|---|---|---|---|
| Deußer et al. | + | CNS | – | – | – | – | |
| yanng981 | C | ✓ | – | – | – | ||
| Convai Innovations | CNS | – | – | – | – | ||
| Tateno | CNS | ✓ | – | – | – | ||
| Rafe and Das | CN | ✓ | – | ✓ | – | ||
| Lu and Wu | C | ✓ | – | – | task |
| Typed Decision Model | Access | Backbone | Precision |
|---|---|---|---|
| Clef | local, API | Qwen3.8-27B † | fp32/4 dp |
| Clef-flash | local, API | Qwen3.5-9B † | fp32/4 dp |
| pplx-decider -v1.1 | API | Qwen3.8-27B ‡ | full |
| Microsoft- Decision-1 | API | Qwen3.5-9B ‡ | full |
| Kev-4B | API | Qwen3.5-4B ‡ | 4 dp |
| Laya -multilingual | local | mmBERT-base | full |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Source | Revision | Licence (XDecideBench treatment) | Provenance of non-English text |
|---|---|---|---|
| MASSIVE 1.1 | ff6bd8e | CC-BY-4.0 | Localized from English SLURP utterances |
| XNLI | b8dd5d7 | CC-BY-NC-4.0 | Translated MultiNLI-style pairs; labels propagated |
| Belebele | 7899cdf | CC-BY-SA-4.0 | Questions on FLORES-200 passages, translated |
| MMLU-ProX (full) | 8e6106a | MIT (declared) | Machine translation of MMLU-Pro with expert review |
| SIB-200 | 38977a6 | CC-BY-SA-4.0 | English topic labels propagated through FLORES-200 |
| INCLUDE | d2e1f60 | Apache-2.0 (card only) | Native regional exams, not translated |
| Rubric (levels) | Agree | Locale en | Differ 2 |
|---|---|---|---|
| toxicity (5) | |||
| insult (3) | |||
| bias (3) | |||
| identity attack (3) | |||
| violence (3) | |||
| microaggression (3) |
| Lang | Test rows | Eligible | Frozen |
|---|---|---|---|
| ar | |||
| de | |||
| es | |||
| fr | |||
| hi | |||
| ru |
| Rule | Models | |
|---|---|---|
| two decimals | Jev, Luna | |
| four decimals | Kev-4B; Clef, Clef-flash (API) | |
| six decimals | Solar | |
| strict | pplx-decider, d1, Mercury, Decision-1 | |
| fp32 | strict | Clef, Clef-flash (local) |
| Model | Brier skill | Rank | Worst | Gap to en | Cov. | ||
| Hosted Typed Decision Models, by Core12 skill | |||||||
| pplx-decider | [.693, .731] | 1–3 | sw | ||||
| Jev | [.667, .710] | 1–7 | sw | ||||
| Clef, API | [.655, .695] | 2–5 | sw | ||||
| d1 | [.642, .682] | 2–6 | sw | ||||
| Decision-1 | [.633, .669] | 1–7 | sw | ||||
| NAT: INCLUDE | DESC: described bare | |||||
| Arm | MMLU-ProX | MASSIVE | SIB-200 | |||
| Typed Decision Models | ||||||
| pplx-decider | [ .005, .109] | [ .001, .019] | [ .049, .023] | |||
| Jev | [ .136, .028] | [.020, .066] | [ .028, .025] | |||
| Clef, API | [.045, .144] | [ .037, .006] | [ .051, .032] | |||
| Clef, local | [.045, .143] | [ .037, .005] | [ .051, .031] | |||
| Native instr. | Fully native | |||
| Model | XNLI | MASSIVE | XNLI | MASSIVE |
| Hosted Typed Decision Models | ||||
| Jev | ∗ | ∗ | ∗ | |
| pplx-decider | ∗ | |||
| d1 | ∗ | ∗ | ||
| Luna | ∗ | ∗ | ∗ | |
| ECE raw | ECE scaled | |||||||
|---|---|---|---|---|---|---|---|---|
| Pair | MASSIVE | XNLI | MASSIVE | XNLI | lowest to | highest | lowest to | highest |
| Clef | to | to | ||||||
| Clef-flash | to | to | ||||||
| pplx-decider | to | to | ||||||
| Head | Gain | CI | Median gain | Median share | with web share ( ) | Slope | |
|---|---|---|---|---|---|---|---|
| Clef | ( ) | ||||||
| Clef-flash | ( ) | ||||||
| pplx-decider | ( ) | ||||||
| Risk at 5% (admitted %) | 90% set: coverage/size | |||||||||||
| Model | En | High | Low | En | High | Low | ||||||
| XNLI | ||||||||||||
| Jev | ( | ) | ( | ) | ( | ) | / | / | / | |||
| pplx-decider | ( | ) | ( | ) | ( | ) | / | / | / | |||
| d1 | ( | ) | ( | ) | ( | ) | / | / | / | |||
| Mercury | ( | ) | ( | ) | ( | ) | / | / | / | |||
| Time (UTC, 2026) | Event |
|---|---|
| 10-04 00:56 | Statistical protocol frozen, with accuracy as the primary endpoint |
| 10-04 01:23–03:58 | Core12 and English development manifests |
| 10-06 20:25 | Held-out cohort manifest frozen |
| 10-07 14:42 | Described-label manifest |
| 10-07 14:43–10-08 20:41 | Model registrations (Laya’s on 10-04), each before that model’s first benchmark request |
| 10-07 14:44–10-08 20:38 | Core12 inference |
| Spend (US$) | Tokens (M) | ||||||
|---|---|---|---|---|---|---|---|
| Hosted arm | Bench. | Held-out cohort | Explor. | Pilot | Total | In | Out |
| Qwen3.8-27B, hosted | |||||||
| Luna | |||||||
| Clef | |||||||
| Jev | |||||||
| Solar | |||||||
| (a) | Aligned, shared labels | Locale labels | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Core12 | Belebele | MASSIVE | NLI | RTP-LX | ||||||
| Model | Fail. | Zero | Fail. | Zero | Fail. | Zero | Fail. | Zero | Fail. | Zero |
| Hosted Typed Decision Models | ||||||||||
| pplx-decider | ||||||||||
| Jev | ||||||||||
| Clef | ||||||||||
| Aligned, shared labels | Locale labels | Native-authored | |||||||||
| Model | Core12 | Belebele | MASSIVE | NLI | RTP-LX | INCLUDE | Injongo | Moder. | BRIGHTER | ||
| Hosted Typed Decision Models | |||||||||||
| pplx-decider | .712 | .756 | .397 | .536 | .578 | .673 | .144 | ||||
| Jev | .688 | .785 | .561 | ||||||||
| Clef | .167 | ||||||||||
| d1 | .315 | .070 | |||||||||
| (a) | Core12 | |||||||||
| Model | MASSIVE | XNLI | Belebele | MMLU-ProX | SIB-200 | |||||
| Hosted Typed Decision Models | ||||||||||
| pplx-decider | [.729, .803] | .663 | [.619, .700] | .887 | [.861, .910] | .450 | [.402, .498] | [.735, .844] | ||
| Jev | [.608, .693] | [.424, .536] | [.851, .911] | .584 | [.535, .634] | .839 | [.783, .888] | |||
| Clef | .776 | [.739, .811] | [.525, .615] | [.819, .880] | [.318, .404] | [.749, .868] | ||||
| d1 | [.676, .748] | .601 | [.550, .645] | [.824, .883] | [.313, .406] | [.733, .834] | ||||
Multilingual Language Model Evaluation
Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.
See all 527 papers