Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models

Avni Mittal; Shanu Kumar; Sandipan Dandapat; Monojit Choudhury

cs.CLApr 10, 2026

Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models

Authors: Avni Mittal, Shanu Kumar, Sandipan Dandapat, Monojit Choudhury

Organizations: Microsoft Corporation, India · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Indian Institute of Technology Hyderabad

Abstract

We study predictive multilingual evaluation: estimating how well a model will perform on a task in a target language when direct benchmark results are missing. This problem is common in multilingual deployment, where evaluation coverage is sparse and published evidence is uneven across languages, tasks, and model families. We introduce a controlled benchmark of 1,500 questions spanning six tasks and five evidence scenarios. The benchmark separates accessible evidence from ground truth, enabling evaluation of systems that must infer missing results from incomplete literature evidence. We also present Litmus (Re)Agent, a DAG-orchestrated agentic system that decomposes queries into hypotheses, retrieves evidence, and synthesises predictions through feature-aware aggregation. Across six systems, Litmus (Re)Agent achieves the best overall performance, with the largest gains in transfer-heavy scenarios where direct evidence is weak or absent. These results show that structured agentic reasoning is a promising approach to multilingual performance estimation under incomplete evidence.

Figures & tables

Figure 1: Overview of the benchmark and Litmus (Re)Agent . Top: six tasks, five controlled scenarios (S1–S5), two query types, and restricted paper-corpus access. Bottom: dynamic DAG orchestration in which specialised agents spawn and prune hypotheses, gather evidence, and aggregate results into the final response.

Scenario	Target language	Target model	Available evidence source
S1: Same L + Same M	Observed	Observed	Direct evidence
S2: Same L + Diff M	Observed	Missing	Same language, other models
S3: Similar L + Same M	Missing	Observed	Typologically close language
S4: Distant L + Same M	Missing	Observed	Typologically distant language
S5: Diff L + Diff M	Missing	Missing	Other languages and model families

Table 1: Summary of the five benchmark scenarios. “Observed” means directly available in the reduced evidence corpus at inference time.

Panel of Figure 2 — Figure 2: Illustrative questions from the two benchmark subsets: PredSet for numeric prediction and QnASet for comparative reasoning.

Figure 3: Combined versus reduced corpus statistics per task.

	PredSet MAE ( $\downarrow$ )						QnASet Accuracy (%) ( $\uparrow$ )
Task	LRA	L++	TA	SA	GPT	M1	LRA	L++	TA	SA	GPT	M1
Code Gen.	14.2	21.1	18.9	17.8	39.8	27.0	55.3	25.4	39.5	51.8	27.2	41.2
Math Reas.	20.0	28.8	27.5	29.6	18.2	26.8	15.4	10.6	13.0	9.8	18.7	7.3
QA/VQA	9.5	8.7	14.7	12.4	9.6	14.1	8.0	22.4	8.0	16.8	16.0	5.6
Class./NLI	7.9	9.6	11.6	11.9	9.9	12.0	16.0	7.2	6.4	13.6	9.6	2.4
Text Sum.	5.1	6.9	7.0	4.7	6.8	8.3	21.0	13.4	10.9	21.8	15.1	1.7

Table 2: Task-level results on the benchmark. Left: PredSet mean absolute error (MAE; lower is better). Right: QnASet accuracy (higher is better). LRA = Litmus (Re)Agent, L++ = LITMUS++, TA = ThoughtAgent, SA = Single Agent, GPT = GPT-4.1, and M1 = Magentic-One.

	MAE ( $\downarrow$ )						Accuracy % ( $\uparrow$ )
Scenario	LRA	L++	TA	SA	GPT	M1	LRA	L++	TA	SA	GPT	M1
S1: Same L+M	9.4	13.5	17.9	14.6	14.5	15.1	29.3	17.3	26.7	22.7	12.0	22.7
S2: Same L, Diff M	10.3	12.8	15.7	15.7	16.0	16.9	14.7	12.0	9.3	22.0	12.7	6.7
S3: Similar L+M	9.0	10.3	11.2	12.5	15.4	14.7	33.3	21.1	19.7	27.9	29.9	14.3
S4: Distant L+M	11.7	12.2	16.3	15.0	19.1	23.5	13.8	13.1	11.7	13.8	11.0	2.1
S5: Diff L+M	11.9	15.7	14.7	15.6	17.7	16.7	16.6	11.5	11.5	13.7	13.7	3.6

Table 3: PredSet MAE and QnASet accuracy by scenario. Abbreviations as in Table 2 . Best per row in bold .

Metric	Value
Thought faithfulness	93.9%
Capability compliance	70.0%
Web search relevance	83.5%
Feature correctness	91.6%
Code execution success	30.6%

Table 4: Agent reasoning quality metrics for Litmus (Re)Agent , computed over 1,730 conversations and 5,184 generated hypotheses.

Figure 4: Feature usage heatmap across tasks. Rows represent feature categories (typological, preprocessing, model-based); columns represent tasks. Darker cells indicate higher usage.

Figure 5: LLM-judge quality metrics (1–5) averaged across all tasks for each system. Litmus (Re)Agent leads on predictive plausibility and feature selection; GPT-4.1 leads on coherence and citation emphasis.

Backbone	Scen.	Pred.	Feat.	Coh.	Cite.
GPT-4.1	S1	4.8	4.9	4.9	3.2
	S3	4.7	4.7	4.9	2.9
	S5	4.5	4.9	5.0	3.0
o4-mini	S1	4.0	4.4	5.0	2.8
	S3	4.2	4.7	4.8	2.7
	S5	4.4	4.8	5.0	2.6

Table 5: Backbone LLM ablation on code generation (S1, S3, S5). Quality metrics from LLM-judge evaluation (1–5 scale) computed on common questions across all models. Pred. = Predictive Plausibility, Feat. = Feature Selection, Coh. = Coherence, Cite. = Citation Emphasis.

Figure 6: Human evaluation quality metrics by participant type (novice vs. expert) and phase (baseline vs. Litmus (Re)Agent ).

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

		PredSet MAE ( $\downarrow$ )						QnASet Accuracy % ( $\uparrow$ )						Quality (1–5) ( $\uparrow$ )
Task	Sys	S1	S2	S3	S4	S5	Avg	S1	S2	S3	S4	S5	Avg	Pr	Ft	Co	Ci	Avg
Code Gen.	LRA	7.9	14.4	8.5	20.2	20.4	14.2	88.0	32.0	73.9	38.1	40.0	55.3	4.7	4.9	4.9	3.0	4.4
	L++	10.5	29.9	22.7	18.4	24.6	21.1	32.0	16.0	34.8	19.0	25.0	25.4	3.1	3.3	4.6	2.1	3.3
	TA	16.8	23.6	12.0	24.7	18.6	18.9	60.0	20.0	43.5	42.9	30.0	39.5	4.6	4.7	4.8	3.0	4.3
	SA	12.0	23.5	12.0	21.8	21.1	17.8	76.0	48.0	65.2	47.6	15.0	51.8	4.4	3.9	4.9	4.2	4.4
	GPT	28.4	43.7	39.5	47.7	39.8	39.8	24.0	20.0	43.5	23.8	25.0	27.2	4.4	3.3	5.0	4.6	4.3

Appendix

Table 6: Detailed results by task: PredSet MAE (

\downarrow

), QnASet accuracy % (

\uparrow

), and LLM-judge quality (1–5,

\uparrow

). Quality: Pr = Predictive Plausibility, Ft = Feature Selection, Co = Coherence, Ci = Citation Emphasis. Abbreviations as in Table 2 .

Figure 7: Per-metric MAE for Litmus (Re)Agent . Accuracy-based metrics show higher prediction error than text-overlap metrics such as ROUGE, BLEU, and chrF.

Figure 8: Per-scenario MAE trends across all five systems. Litmus (Re)Agent maintains the lowest and most stable MAE, while Magentic-One degrades sharply in S4 (distant language transfer).

Figure 9: Litmus (Re)Agent: mean MAE by scenario and task. Each group of bars corresponds to one scenario (S1–S5), with tasks shown as different colours. Code generation and mathematical reasoning show higher and more variable error than other tasks.

Figure 10: Distribution of per-question absolute errors for Litmus (Re)Agent , grouped by scenario–task pairs (log scale). Diamonds indicate means; horizontal lines indicate medians. Wider boxes and more outliers appear in code generation and math reasoning, especially in transfer-heavy scenarios.

Figure 11: Sample screenshot of the Litmus (Re)Agent interface showing the conversational interaction, structured evidence retrieval, and prediction output presented to participants during the Litmus (Re)Agent phase.

Figure 12: Sample screenshot of the Litmus (Re)Agent interface showing the final prediction summary with confidence intervals, reasoning trace, and supporting evidence.

Figure 13: Screenshot of the human evaluation form used after each prediction question. Participants rate quality dimensions on Likert scales and report their reasoning strategies and challenges.

#	Phase	Model	Language
1	Baseline	GPT-4	Hindi
2	Baseline	GPT-4o	Odia
3	Baseline	GPT-3.5	Romanian
4	Baseline	DeepSeek-Coder-V2	Shona
5	Baseline	Claude-3.5-Opus	Bosnian
6	LITMUS++	CodeT5+-2B	Malay

Appendix

Table 7: The 10 code generation prediction questions used in the human evaluation.

	Overall			Novice			Expert
Metric	B	L	$\Delta$	B	L	$\Delta$	B	L	$\Delta$
Confidence	2.6	3.4	+.8	2.4	3.3	+1.0	2.8	3.5	+.7
Interpretability	3.1	3.7	+.7	3.1	4.0	+.8	3.0	3.5	+.5
Explainability	3.2	3.9	+.7	3.2	4.0	+.8	3.4	3.9	+.5
Justification	2.8	3.8	+1.0	3.0	3.8	+.9	2.8	3.8	+1.1
Actionability	2.6	3.5	+.9	2.8	3.7	+1.0	2.5	3.3	+.8

Appendix

Table 8: Human evaluation: baseline (B) vs. Litmus (Re)Agent (L). Higher is better except for time.

Figure 14: Human evaluation results: (a) mean quality dimension ratings (baseline vs. Litmus (Re)Agent ) across all participants, (b) self-assessed prediction quality distribution showing the proportion of responses rated from “much worse” to “much better” relative to unaided prediction.

Figure 15: Regression algorithms selected by the Coder agent, broken down by task. Darker cells indicate higher usage frequency.

Figure 16: Usage of lang2vec typological features by scenario. Cross-lingual scenarios (S3, S4) show the highest feature utilisation.

Figure 17: Breakdown of lang2vec feature categories used by the Coder agent. Syntactic and geographic features dominate across all tasks.

Panel of Figure 18 — Figure 18: Unique language and model-family coverage by task and scenario.

Figure 19: Language frequency across all benchmark questions. Left: top 20 languages by question frequency. Right: the same distribution with remaining languages grouped as Others.

Figure 20: Model-family frequency across all benchmark questions. Left: top model families by question frequency. Right: the same distribution with remaining families grouped as Others.

Panel of Figure 21 — Figure 21: Prompt template used to generate illustrative questions for PredSet (numeric prediction) and QnASet (comparative reasoning).

Panel of Figure 22 — Figure 22: LLM-as-judge prompt for evaluating system responses on four quality dimensions (predictive plausibility, feature selection, coherence, citation emphasis). Each metric is scored 1–5 with structured rationale and indicators.

Panel of Figure 23 — Figure 23: Prompt used to extract structured predictions from system response reports. The extracted predictions are compared against ground truth to compute PredSet MAE (numeric prediction) and QnASet accuracy (comparative reasoning).

Panel of Figure 24 — Figure 24: Prompts used to evaluate ThoughtCreatorAgent output: faithfulness reflection (alignment with expert guidance) and system capability adherence (avoidance of prohibited operations).

Panel of Figure 25 — Figure 25: Prompt used to evaluate generated Python code quality across seven dimensions: task alignment, algorithms, features, methodology, code quality, research soundness, and sophistication.

Panel of Figure 26 — Figure 26: Prompt used to evaluate web search and crawl tool call relevance against the active hypothesis context.

Multilingual Language Model Evaluation

Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.

See all 485 papers

BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents
Sep 20, 2026Peng Kuang, Yuchun Fan, Jiangnan Li +7Multilingual Language Model Evaluation LLM Agent Evaluation
Alibaba Token Hub, Alibaba Group · University of Illinois Urbana-Champaign · Northeastern University +2
OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents
Aug 9, 2026Andrea Caciolai, Pere-Lluís Huguet Cabot, Chierh Cheng +11AI Agent Evaluation AI Agent Benchmarks
Meta Superintelligence Labs
Lost in Translation: Do LVLM Judges Generalize Across Languages?
Apr 21, 2026Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5Multimodal Model Evaluation Multilingual VLM Evaluation
York University · University of Alberta · Nanyang Technological University +1

Sep 20, 2026J/K move · Enter open · S save

BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents

Peng Kuang, Yuchun Fan, Jiangnan Li, Minghao Wu+6

Alibaba Token Hub, Alibaba Group · University of Illinois Urbana-Champaign · Northeastern University+2

Multilingual Language Model Evaluation LLM Agent Evaluation AI Agent Benchmarks Cross-Lingual Consistency in Language Models

Open paper AskPDF arXiv

Large language model (LLM) agents increasingly execute multi-step workflows through tool use and interaction with users and environments. However, current agent evaluations are largely English-centric, limiting our understanding of agent capabilities in multilingual settings. We introduce BabelFlow, a benchmark-general agentic workflow that adapts existing agent benchmarks to new languages by analyzing runtime dependencies, coordinating structure-preserving translation, and combining multi-layer verification with human review to preserve task and evaluation semantics. Using BabelFlow, we construct BabelArena, a task-aligned benchmark comprising 16,146 instances derived from 702 canonical tasks across four benchmark families, 13 domains, and 23 languages. Experiments with five frontier models show that no single model dominates across benchmark families and that cross-language disparities extend well beyond task success. Lower-resource languages exhibit distinct failure patterns, with larger shares of tool-use and control-flow errors rather than answer-quality errors alone, pointing to gaps in reliable task execution across the resource levels of these languages. On the same tasks, agents in low-resource languages also consume substantially more tokens than in English (up to roughly twice the input) without proportional increases in interaction length, and language consistency degrades further on tasks requiring structured output, where switches are directed overwhelmingly toward English. We believe BabelArena provides a foundation for advancing research on reliable and efficient multilingual agents.

Abstract

Multilingual Language Model Evaluation

Explore similar work

BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents

OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents

Lost in Translation: Do LVLM Judges Generalize Across Languages?