MARCA: A Checklist-Based Benchmark for Multilingual Web Search

Thales Sales Almeida; Giovana Kerche Bonás; Ramon Pires; Celio Larcher; Hugo Abonizio; Marcos Piau; Roseval Malaquias Junior; Rodrigo Nogueira

cs.CLApr 15, 2026

MARCA: A Checklist-Based Benchmark for Multilingual Web Search

Authors: Thales Sales Almeida, Giovana Kerche Bonás, Ramon Pires, Celio Larcher, Hugo Abonizio, Marcos Piau, Roseval Malaquias Junior, Rodrigo Nogueira,

Organizations: Maritaca AI · Jusbrasil

Abstract

Large language models (LLMs) are increasingly used as sources of information, yet their reliability depends on the ability to search the web, select relevant evidence, and synthesize complete answers. While recent benchmarks evaluate web-browsing and agentic tool use, multilingual settings, and Portuguese in particular, remain underexplored. We present MARCA, a bilingual (English and Portuguese) benchmark for evaluating LLMs on web-based information seeking. MARCA consists of 52 manually authored multi-entity questions, paired with manually validated checklist-style rubrics that explicitly measure answer completeness and correctness. We evaluate 14 models under two interaction settings: a Basic framework with direct web search and scraping, and an Orchestrator framework that enables task decomposition via delegated subagents. To capture stochasticity, each question is executed multiple times and performance is reported with run-level uncertainty. Across models, we observe large performance differences, find that orchestration often improves coverage, and identify substantial variability in how models transfer from English to Portuguese. The benchmark is available at https://github.com/maritaca-ai/MARCA

Figures & tables

Figure 2: Overview of the two evaluation frameworks. In the basic framework, the model directly issues web searches and scrapes pages. In the orchestrator framework, a top-level model delegates sub-questions to subagents that perform the web interactions.

Model	English		Portuguese
	Basic	Orchestrator	Basic	Orchestrator
GPT-5.2 [ 19 ]	$0.837\pm 0.062$	$0.881\pm 0.039$	$0.855\pm 0.036$	$0.872\pm 0.022$
GPT-5-mini [ 20 ]	$0.837\pm 0.015$	$0.827\pm 0.027$	$0.852\pm 0.022$	$0.819\pm 0.083$
Gemini-3-pro [ 10 ]	$0.885\pm 0.020$	$0.897\pm 0.042$	$0.896\pm 0.017$	$0.889\pm 0.016$
Gemini-3-flash [ 10 ]	$0.828\pm 0.008$	$0.831\pm 0.013$	$0.733\pm 0.016$	$0.820\pm 0.005$
GPT-4.1 [ 18 ]	$0.815\pm 0.022$	$0.779\pm 0.026$	$0.826\pm 0.023$	$0.825\pm 0.029$

Table 1: Checklist Accuracy (mean

\pm

run-level standard deviation) across languages and inference frameworks.

Figure 3: English Checklist Accuracy under Basic vs. Orchestrator. Error bars correspond to run-level standard deviation.

Figure 4: Dumbbell plot comparing Checklist Accuracy between English and Portuguese in the Orchestrator setting (mean with run-level uncertainty).

Figure 5: Performance vs. cost in English Orchestrator using Checklist Accuracy (mean with run-level uncertainty). Cost is measured as total token usage per question.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Disentangling Language Roles in Multilingual LLM Task Execution
May 26, 2026Qishi Zhan, Minxuan Hu, Seoyeon Jang +7Multilingual Language Models Multilingual Language Model Evaluation
Marquette · Cornell · UC San Diego +7
BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents
Sep 20, 2026Peng Kuang, Yuchun Fan, Jiangnan Li +7Multilingual Language Model Evaluation LLM Agent Evaluation
Alibaba Token Hub, Alibaba Group · University of Illinois Urbana-Champaign · Northeastern University +2
Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models
Apr 10, 2026Avni Mittal, Shanu Kumar, Sandipan Dandapat +1Multilingual Language Model Evaluation Benchmark Design
Microsoft Corporation, India · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Indian Institute of Technology Hyderabad

May 26, 2026J/K move · Enter open · S save

Disentangling Language Roles in Multilingual LLM Task Execution

Qishi Zhan, Minxuan Hu, Seoyeon Jang, Lei Zhao+6

Marquette · Cornell · UC San Diego+7

Multilingual Language Models Multilingual Language Model Evaluation Instruction Following

Open paper AskPDF arXiv

Multilingual LLMs are increasingly used when instruction, source content, and required response languages do not coincide. Existing benchmarks have expanded multilingual instruction-following evaluation, but they rarely isolate these three roles within a fully crossed design. We introduce MTM-Bench, a controlled benchmark for language-conditioned task execution in which each instance is defined by a triplet

(L_{\text{instr}}, L_{\text{content}}, L_{\text{resp}})

. Across English, Spanish, and Chinese, MTM-Bench enumerates all 27 triplets and contains 2{,}430 instances per model across semantic reversal, final-state extraction, and language purity with update realization. We evaluate 20 frontier and open-weight LLMs using decomposed metrics for semantic correctness, target-language adherence, constraint satisfaction, contamination ratio, and joint success, with scoring validated by a targeted human audit. The fully crossed design reveals that degradation is organized by the role a language occupies in the task structure, not merely by mismatch count. The response-language role is the dominant axis of variation, and a single response-slot mismatch accounts for most degradation. The response-only and full-mismatch comparison suggests that mismatch count is not a monotonic predictor of difficulty, with model-level ordering varying across systems. Task families fail through distinct channels, showing that semantic correctness alone does not capture reliable multilingual task execution.

Abstract

LLM Evaluation

Explore similar work

Disentangling Language Roles in Multilingual LLM Task Execution

BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents

Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models