Organizations: Microsoft Corporation, India · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Indian Institute of Technology Hyderabad
We study predictive multilingual evaluation: estimating how well a model will perform on a task in a target language when direct benchmark results are missing. This problem is common in multilingual deployment, where evaluation coverage is sparse and published evidence is uneven across languages, tasks, and model families. We introduce a controlled benchmark of 1,500 questions spanning six tasks and five evidence scenarios. The benchmark separates accessible evidence from ground truth, enabling evaluation of systems that must infer missing results from incomplete literature evidence. We also present Litmus (Re)Agent, a DAG-orchestrated agentic system that decomposes queries into hypotheses, retrieves evidence, and synthesises predictions through feature-aware aggregation. Across six systems, Litmus (Re)Agent achieves the best overall performance, with the largest gains in transfer-heavy scenarios where direct evidence is weak or absent. These results show that structured agentic reasoning is a promising approach to multilingual performance estimation under incomplete evidence.
Figures & tables
Figure 1: Overview of the benchmark and Litmus (Re)Agent . Top: six tasks, five controlled scenarios (S1–S5), two query types, and restricted paper-corpus access. Bottom: dynamic DAG orchestration in which specialised agents spawn and prune hypotheses, gather evidence, and aggregate results into the final response.
Scenario
Target language
Target model
Available evidence source
S1: Same L + Same M
Observed
Observed
Direct evidence
S2: Same L + Diff M
Observed
Missing
Same language, other models
S3: Similar L + Same M
Missing
Observed
Typologically close language
S4: Distant L + Same M
Missing
Observed
Typologically distant language
S5: Diff L + Diff M
Missing
Missing
Other languages and model families
Table 1: Summary of the five benchmark scenarios. “Observed” means directly available in the reduced evidence corpus at inference time.
Figure 2: Illustrative questions from the two benchmark subsets: PredSet for numeric prediction and QnASet for comparative reasoning.
Figure 3: Combined versus reduced corpus statistics per task.
PredSet MAE ( ↓ )
QnASet Accuracy (%) ( ↑ )
Task
LRA
L++
TA
SA
GPT
M1
LRA
L++
TA
SA
GPT
M1
Code Gen.
14.2
21.1
18.9
17.8
39.8
27.0
55.3
25.4
39.5
51.8
27.2
41.2
Math Reas.
20.0
28.8
27.5
29.6
18.2
26.8
15.4
10.6
13.0
9.8
18.7
7.3
QA/VQA
9.5
8.7
14.7
12.4
9.6
14.1
8.0
22.4
8.0
16.8
16.0
5.6
Class./NLI
7.9
9.6
11.6
11.9
9.9
12.0
16.0
7.2
6.4
13.6
9.6
2.4
Text Sum.
5.1
6.9
7.0
4.7
6.8
8.3
21.0
13.4
10.9
21.8
15.1
1.7
Table 2: Task-level results on the benchmark. Left: PredSet mean absolute error (MAE; lower is better). Right: QnASet accuracy (higher is better). LRA = Litmus (Re)Agent, L++ = LITMUS++, TA = ThoughtAgent, SA = Single Agent, GPT = GPT-4.1, and M1 = Magentic-One.
MAE ( ↓ )
Accuracy % ( ↑ )
Scenario
LRA
L++
TA
SA
GPT
M1
LRA
L++
TA
SA
GPT
M1
S1: Same L+M
9.4
13.5
17.9
14.6
14.5
15.1
29.3
17.3
26.7
22.7
12.0
22.7
S2: Same L, Diff M
10.3
12.8
15.7
15.7
16.0
16.9
14.7
12.0
9.3
22.0
12.7
6.7
S3: Similar L+M
9.0
10.3
11.2
12.5
15.4
14.7
33.3
21.1
19.7
27.9
29.9
14.3
S4: Distant L+M
11.7
12.2
16.3
15.0
19.1
23.5
13.8
13.1
11.7
13.8
11.0
2.1
S5: Diff L+M
11.9
15.7
14.7
15.6
17.7
16.7
16.6
11.5
11.5
13.7
13.7
3.6
Table 3: PredSet MAE and QnASet accuracy by scenario. Abbreviations as in Table 2 . Best per row in bold .
Metric
Value
Thought faithfulness
93.9%
Capability compliance
70.0%
Web search relevance
83.5%
Feature correctness
91.6%
Code execution success
30.6%
Table 4: Agent reasoning quality metrics for Litmus (Re)Agent , computed over 1,730 conversations and 5,184 generated hypotheses.
Figure 5: LLM-judge quality metrics (1–5) averaged across all tasks for each system. Litmus (Re)Agent leads on predictive plausibility and feature selection; GPT-4.1 leads on coherence and citation emphasis.
Backbone
Scen.
Pred.
Feat.
Coh.
Cite.
GPT-4.1
S1
4.8
4.9
4.9
3.2
S3
4.7
4.7
4.9
2.9
S5
4.5
4.9
5.0
3.0
o4-mini
S1
4.0
4.4
5.0
2.8
S3
4.2
4.7
4.8
2.7
S5
4.4
4.8
5.0
2.6
Table 5: Backbone LLM ablation on code generation (S1, S3, S5). Quality metrics from LLM-judge evaluation (1–5 scale) computed on common questions across all models. Pred. = Predictive Plausibility, Feat. = Feature Selection, Coh. = Coherence, Cite. = Citation Emphasis.
Figure 6: Human evaluation quality metrics by participant type (novice vs. expert) and phase (baseline vs. Litmus (Re)Agent ).
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
PredSet MAE ( ↓ )
QnASet Accuracy % ( ↑ )
Quality (1–5) ( ↑ )
Task
Sys
S1
S2
S3
S4
S5
Avg
S1
S2
S3
S4
S5
Avg
Pr
Ft
Co
Ci
Avg
Code Gen.
LRA
7.9
14.4
8.5
20.2
20.4
14.2
88.0
32.0
73.9
38.1
40.0
55.3
4.7
4.9
4.9
3.0
4.4
L++
10.5
29.9
22.7
18.4
24.6
21.1
32.0
16.0
34.8
19.0
25.0
25.4
3.1
3.3
4.6
2.1
3.3
TA
16.8
23.6
12.0
24.7
18.6
18.9
60.0
20.0
43.5
42.9
30.0
39.5
4.6
4.7
4.8
3.0
4.3
SA
12.0
23.5
12.0
21.8
21.1
17.8
76.0
48.0
65.2
47.6
15.0
51.8
4.4
3.9
4.9
4.2
4.4
GPT
28.4
43.7
39.5
47.7
39.8
39.8
24.0
20.0
43.5
23.8
25.0
27.2
4.4
3.3
5.0
4.6
4.3
Appendix
Table 6: Detailed results by task: PredSet MAE ( ↓ ), QnASet accuracy % ( ↑ ), and LLM-judge quality (1–5, ↑ ). Quality: Pr = Predictive Plausibility, Ft = Feature Selection, Co = Coherence, Ci = Citation Emphasis. Abbreviations as in Table 2 .
Figure 7: Per-metric MAE for Litmus (Re)Agent . Accuracy-based metrics show higher prediction error than text-overlap metrics such as ROUGE, BLEU, and chrF.
Figure 8: Per-scenario MAE trends across all five systems. Litmus (Re)Agent maintains the lowest and most stable MAE, while Magentic-One degrades sharply in S4 (distant language transfer).
Figure 9: Litmus (Re)Agent: mean MAE by scenario and task. Each group of bars corresponds to one scenario (S1–S5), with tasks shown as different colours. Code generation and mathematical reasoning show higher and more variable error than other tasks.
Figure 10: Distribution of per-question absolute errors for Litmus (Re)Agent , grouped by scenario–task pairs (log scale). Diamonds indicate means; horizontal lines indicate medians. Wider boxes and more outliers appear in code generation and math reasoning, especially in transfer-heavy scenarios.
Figure 11: Sample screenshot of the Litmus (Re)Agent interface showing the conversational interaction, structured evidence retrieval, and prediction output presented to participants during the Litmus (Re)Agent phase.
Figure 12: Sample screenshot of the Litmus (Re)Agent interface showing the final prediction summary with confidence intervals, reasoning trace, and supporting evidence.
Figure 13: Screenshot of the human evaluation form used after each prediction question. Participants rate quality dimensions on Likert scales and report their reasoning strategies and challenges.
#
Phase
Model
Language
1
Baseline
GPT-4
Hindi
2
Baseline
GPT-4o
Odia
3
Baseline
GPT-3.5
Romanian
4
Baseline
DeepSeek-Coder-V2
Shona
5
Baseline
Claude-3.5-Opus
Bosnian
6
LITMUS++
CodeT5+-2B
Malay
Appendix
Table 7: The 10 code generation prediction questions used in the human evaluation.
Overall
Novice
Expert
Metric
B
L
Δ
B
L
Δ
B
L
Δ
Confidence
2.6
3.4
+.8
2.4
3.3
+1.0
2.8
3.5
+.7
Interpretability
3.1
3.7
+.7
3.1
4.0
+.8
3.0
3.5
+.5
Explainability
3.2
3.9
+.7
3.2
4.0
+.8
3.4
3.9
+.5
Justification
2.8
3.8
+1.0
3.0
3.8
+.9
2.8
3.8
+1.1
Actionability
2.6
3.5
+.9
2.8
3.7
+1.0
2.5
3.3
+.8
Appendix
Table 8: Human evaluation: baseline (B) vs. Litmus (Re)Agent (L). Higher is better except for time.
Figure 14: Human evaluation results: (a) mean quality dimension ratings (baseline vs. Litmus (Re)Agent ) across all participants, (b) self-assessed prediction quality distribution showing the proportion of responses rated from “much worse” to “much better” relative to unaided prediction.
Figure 15: Regression algorithms selected by the Coder agent, broken down by task. Darker cells indicate higher usage frequency.
Figure 16: Usage of lang2vec typological features by scenario. Cross-lingual scenarios (S3, S4) show the highest feature utilisation.
Figure 17: Breakdown of lang2vec feature categories used by the Coder agent. Syntactic and geographic features dominate across all tasks.
Figure 18: Unique language and model-family coverage by task and scenario.
Figure 19: Language frequency across all benchmark questions. Left: top 20 languages by question frequency. Right: the same distribution with remaining languages grouped as Others.
Figure 20: Model-family frequency across all benchmark questions. Left: top model families by question frequency. Right: the same distribution with remaining families grouped as Others.
Figure 21: Prompt template used to generate illustrative questions for PredSet (numeric prediction) and QnASet (comparative reasoning).
Figure 22: LLM-as-judge prompt for evaluating system responses on four quality dimensions (predictive plausibility, feature selection, coherence, citation emphasis). Each metric is scored 1–5 with structured rationale and indicators.
Figure 23: Prompt used to extract structured predictions from system response reports. The extracted predictions are compared against ground truth to compute PredSet MAE (numeric prediction) and QnASet accuracy (comparative reasoning).
Figure 24: Prompts used to evaluate ThoughtCreatorAgent output: faithfulness reflection (alignment with expert guidance) and system capability adherence (avoidance of prohibited operations).
Figure 25: Prompt used to evaluate generated Python code quality across seven dimensions: task alignment, algorithms, features, methodology, code quality, research soundness, and sophistication.
Figure 26: Prompt used to evaluate web search and crawl tool call relevance against the active hypothesis context.
Large language model (LLM) agents increasingly execute multi-step workflows through tool use and interaction with users and environments. However, current agent evaluations are largely English-centric, limiting our understanding of agent capabilities in multilingual settings. We introduce BabelFlow, a benchmark-general agentic workflow that adapts existing agent benchmarks to new languages by analyzing runtime dependencies, coordinating structure-preserving translation, and combining multi-layer verification with human review to preserve task and evaluation semantics. Using BabelFlow, we construct BabelArena, a task-aligned benchmark comprising 16,146 instances derived from 702 canonical tasks across four benchmark families, 13 domains, and 23 languages. Experiments with five frontier models show that no single model dominates across benchmark families and that cross-language disparities extend well beyond task success. Lower-resource languages exhibit distinct failure patterns, with larger shares of tool-use and control-flow errors rather than answer-quality errors alone, pointing to gaps in reliable task execution across the resource levels of these languages. On the same tasks, agents in low-resource languages also consume substantially more tokens than in English (up to roughly twice the input) without proportional increases in interaction length, and language consistency degrades further on tasks requiring structured output, where switches are directed overwhelmingly toward English. We believe BabelArena provides a foundation for advancing research on reliable and efficient multilingual agents.
Peng Kuang, Yuchun Fan, Jiangnan Li +7
Alibaba Token Hub, Alibaba Group · University of Illinois Urbana-Champaign · Northeastern University +2
Agentic benchmarks aim to measure how well AI agents plan, search, execute, and recover within realistic multi-tool environments, but they are almost exclusively in English. As AI agents are globally deployed to a linguistically diverse user base, whether agentic competence measured in English transfers to other languages remains an open question. We introduce OmnilingualGAIA2, a machine-translated expansion (with partial human- expert validation) of the GAIA2 agentic benchmark, covering ten target languages spanning five writing systems, paired with a localised and human-calibrated multilingual verifier. Evaluating seven frontier and open-weight agents, we find a universal cross-lingual gap of 8.8-18.4 pass@3 points that is agent-asymmetric in magnitude, concentrates on tool-orchestration rather than quantitative reasoning, and does not close with model scale. A stratified error attribution decomposes the gap as predominantly model-driven (55%), with a bounded translation-contamination floor of only 6.4% of scenario-language pairs. Human-expert linguistic analysis further identifies morphological cue loss and amplified ambiguity as the primary failure mechanisms in non-Latin-script languages. Our results argue that multilingual agentic evaluation must become a standard part of the reporting protocol for globally deployed agents.
Andrea Caciolai, Pere-Lluís Huguet Cabot, Chierh Cheng +11
Automatic evaluators such as reward models play a central role in the alignment and evaluation of large vision-language models (LVLMs). Despite their growing importance, these evaluators are almost exclusively assessed on English-centric benchmarks, leaving open the question of how well these evaluators generalize across languages. To answer this question, we introduce MM-JudgeBench, the first large-scale benchmark for multilingual and multimodal judge model evaluation, which includes over 60K pairwise preference instances spanning 25 typologically diverse languages. MM-JudgeBench integrates two complementary subsets: a general vision-language preference evaluation subset extending VL-RewardBench, and a chart-centric visual-text reasoning subset derived from OpenCQA, enabling systematic analysis of reward models (i.e., LVLM judges) across diverse settings. We additionally release a multilingual training set derived from MM-RewardBench, disjoint from our evaluation data, to support domain adaptation. By evaluating 22 LVLMs (15 open-source, 7 proprietary), we uncover substantial cross-lingual performance variance in our proposed benchmark. Our analysis further shows that model size and architecture are poor predictors of multilingual robustness, and that even state-of-the-art LVLM judges exhibit inconsistent behavior across languages. Together, these findings expose fundamental limitations of current reward modeling and underscore the necessity of multilingual, multimodal benchmarks for developing reliable automated evaluators.
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5
York University · University of Alberta · Nanyang Technological University +1