HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents
Authors: Alham Fikri Aji, Faiz Rizki Ramadhan, Zayd M. K. Zuhri, Seung Hun Eddie Han, Ryandito Diandaru, Qinrong Cui, Jan Christian Blaise Cruz, Badrinath Chandana, Peerawat Chomphooyod, Ahmed Attia, Jonibek Mansurov, Emilio Villa-Cueva, Canh Duong Nguyen, Imran Turganov, Minghao Wu, Peerat Limkonchotiwat, Irina Nikishina
Organizations: Mohamed bin Zayed University of Artificial Intelligence · Mila – Quebec Artificial Intelligence Institute · Inception AI · Alibaba Group · AI Singapore
We introduce HyperBrowseComp, a multilingual and multimodal browsing benchmark comprising 423 manually authored and human-validated questions across 13 languages, written by native or highly proficient speakers. Questions are designed to be extremely challenging. Each question targets a concise, publicly verifiable answer whose discovery requires locating obscure evidence, following multi-step clue chains, or inspecting heterogeneous sources such as videos, scanned documents, images, or maps. Easier questions are filtered out by evaluating them with models without internet access to reduce the likelihood that they can be answered with parametric knowledge alone. We evaluate several models using provider-native search and a shared external retrieval harness under a common agent protocol. To contextualize model performance and effort, we also conduct a human evaluation on a sample of the questions. HyperBrowseComp provides a challenging testbed for persistent information seeking across languages and evidence modalities, with difficulty arising from discovering and connecting evidence on the open web.
Figures & tables
Figure 1: Example question and evidence route from HyperBrowseComp . Solving it requires locating a specific moment in a video, grounding the scene to a real-world location, and retrieving information from a financial report.
Benchmark
Open web
Multimodal
# Langs
Data provenance
BrowseComp ( Wei et al., 2025 )
✔
✗
1
Human
BrowseComp-Plus ( Chen et al., 2025 )
✗
✗
1
Human
BrowseComp-ZH ( Zhou et al., 2025 )
✔
✗
1
Human
K-BrowseComp ( Lee et al., 2026 )
✔
✗
1
Mixed
XBCP ( Lu et al., 2026 )
✗
✗
12
Translation-based
MM-BrowseComp ( Li et al., 2025 )
✔
✔
1
Human
Table 1: Comparison with other browsing and search benchmarks.
Figure 2: Distributions of (a) languages, (b) modalities, and (c) primary domains in the 423 HyperBrowseComp questions. Labels report counts and percentages of all questions. Modalities may overlap; each question has one language and one primary domain.
Setting
Model
Correct.
Accuracy
Total tokens (M)
Total time (h)
Built-in
Gemini 3.7 Flash
134
31.68%
14.1
129.5
Built-in
Gemini 3.1 Pro
108
25.53%
10.2
45.3
Built-in
GPT-5.6 Sol
81
19.15%
43.6
19.9
Built-in
GPT-5.6 Terra
67
15.84%
33.1
11.1
Built-in
GPT-5.6 Luna
49
11.58%
27.5
8.5
Exa
GPT-5.6 Sol
113
26.71%
928.8
75.1
Table 2: Performance and total resource use across retained questions. Tokens are rounded to 0.1 million; time is cumulative end-to-end sample time, rounded to 0.1 hour.
Language
n
Accuracy
Gave up
Mean (min)
Median (min)
Gemini 3.7 Flash
Accuracy
Indonesian
10
6/10
1
40.23
34.68
3/10
Thai
10
3/10
2
159.70
60.33
3/10
Vietnamese
10
6/10
1
68.56
57.92
7/10
All
30
15/30
4
89.50
54.57
13/30
Table 3: Human evaluation on 30 questions. Timing summaries include all attempts, including give-ups.
Figure 3: Question difficulty across five models with built-in internet search. (a) Counts of questions answered correctly by different numbers of models; cumulative success groups overlap. (b) Selected examples of characteristics associated with shared failures. Percentages and fractions indicate questions answered incorrectly by all five models within each category. The dashed line marks the overall shared-failure rate (57.68%); the patterns are descriptive, not causal.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Evaluation epochs
1
Research-step limit
25 ReAct iterations (built-in search and Exa); not applicable to OWL
Model API retries
Up to 2 per ReAct request; 1 per OWL request
Sample-level retries
Up to 3 after ReAct execution errors; no automatic OWL rerun within a shard
Temperature
Provider default (ReAct); 0 (OWL)
Maximum output length
Provider default (ReAct); 8,192 tokens per OWL call
Appendix
Table 4: Evaluation configuration by harness.
Setting
Available tools
No tools
None
Built-in search
Provider-specific web_search
Exa
Exa-backed web_search and web_fetch
OWL
DuckDuckGo and Wikipedia search, screenshot-based visual browsing, and image, video, rendered-PDF, and audio analysis
Appendix
Table 5: Tools exposed under each retrieval setting.
Judge
Correct / N
Acc.
Δ pp
Agree.
C → I / I → C
Self-judge
723 / 3,384
21.37%
—
—
—
Gemini 3.7 Flash
740 / 3,384
21.87%
+0.50
99.32%
3 / 20
GLM-4.7
706 / 3,384
20.86%
−0.50
98.85%
28 / 11
Kimi K2 Thinking
696 / 3,384
20.57%
−0.80
98.79%
34 / 7
Appendix
Table 6: Sensitivity of aggregate accuracy to the grading model. The final column gives changes from correct to incorrect and from incorrect to correct relative to the original self-judge.
Threshold ( k )
Excluded
Retained
Gemini 3.1 Pro
Gemini 3.7 Flash
DeepSeek V3.2
GPT-OSS 120B
Kimi K2
MiniMax M2
GLM 4.7
1
121
379
67
68
24
20
19
16
11
2
54
446
42
44
19
18
16
12
7
3
25
475
20
22
15
14
15
8
6
4
15
485
12
14
12
10
10
7
5
5
7
493
5
6
6
6
6
6
3
6
2
498
2
2
2
2
2
2
1
Appendix
Table 7: Questions excluded and retained when exclusion requires correct answers from at least k of the seven models without internet access. Each model column counts the excluded questions that model answered correctly; these counts overlap across models.
Figure 7: Shared failures across all 13 languages, sorted by shared-failure rate. The dashed line marks the overall rate (244/423; 57.68%). The languages contain different question sets, so these rates do not isolate language ability.
Figure 8: Shared failures across 12 domains (sample sizes differ), ordered by shared-failure rate. The dashed line marks the overall rate (244/423; 57.68%). These associations are descriptive, not causal.
Figure 9: Shared failures across all modality-related labels. The dashed line marks the overall rate (244/423; 57.68%). Labels overlap, Arithmetic describes an operation, and no selected modality label does not imply text-only input. These associations are descriptive, not causal.
Figure 10: Top-100 terms ranked by positive mean TF-IDF difference between shared-failure and solved questions. Dots show shared-failure rates, not TF-IDF scores. Fractions count failures among questions containing each term. Dashed lines mark 244/423 (57.68%). Terms occur in at least eight questions. The associations are descriptive, not causal.
Web agent benchmarks test how reliably AI agents browse, navigate, and act on websites, including under long-horizon, ambiguous, and adversarial conditions.