cs.CLNov 1, 2026

XDecideBench: Evaluating Typed Decision Models and Confidence Policies Across Languages

Authors: Marek Šuppa, Jaroslav Kopčan

Organizations: Comenius University in Bratislava · Cisco Systems · Kempelen Institute of Intelligent Technologies

Abstract

Typed Decision Models such as Jev, Clef and pplx-decider return a probability for each answer an application provides. Applications use them in many languages, but are compared mostly in English. We introduce XDecideBench, a multilingual benchmark of nine suites in 122 languages. On our twelve-language suite, three open models fine-tuned for typed decisions beat their Qwen base models in every language. On new test data with local base models, their accuracy gain is larger in languages with less web text than in those with more. They fix a similar share of base-model errors in both, so the larger gain may reflect weaker base models. The best model on the twelve-language suite is best on only two other suites. The best English model on each suite, chosen in hindsight, is significantly beaten in about a quarter of suite and language combinations. On Belebele, Decision-1 is best in English but falls far behind Jev in Burmese. A confidence threshold fitted on English to allow 5% errors gives a higher error rate on XNLI, even in German and French. XDecideBench shows that results from one suite or from English are a poor guide to how a model and its threshold behave in other languages.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Do We Still Need Fine-Tuned Classifiers? Swapping Typed Decision Models into 24 ACL-Venue Papers

    Nov 15, 2026Marek Šuppa, Jaroslav KopčanTyped Decision ModelsZero-Shot Text Classification

  2. Zuse: A Decision Model for European Languages

    Nov 4, 2026Umair Mehmood ImamMultilingual Language Model EvaluationLanguage Model Calibration