cs.CLAug 18, 2026
SaveBEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models
Organizations: Yandex · Applied AI Institute
Abstract
While Multimodal Large Language Models (MLLMs) have made significant strides in visual comprehension, their ability to reason about text-dense, professional documents remains incompletely evaluated. Existing benchmarks emphasize information extraction, require external domain knowledge, or cover professional documents only as one of many settings. They are also largely English- or Chinese-centric, leaving other languages and Russian, in particular, substantially underrepresented. To address these limitations, we introduce BEAR-Bench (Bilingual Enterprise and Academic Reasoning), a self-contained, complex English-and-Russian benchmark comprising 1000 human-annotated questions based on text-rich business and scientific documents. We evaluate 16 proprietary and open-weight MLLMs, including Gemini 3.1 Pro and Qwen3.5-397B, on BEAR-Bench and observe clear headroom even for the strongest systems. Finally, we use the resulting model outputs to compare existing hallucination detection methods, evaluating not only how often models fail on BEAR-Bench but also how reliably those failures can be identified.
Figures & tables
| Benchmark | #Langs | #QA pairs | #RU reasoning VQA | Image scope | OCR chars/img |
| DocVQA Mathew et al. (2021) | EN | 5.2K | n/a | Industry documents | 1,113.0 |
| ChartQA Masry et al. (2022) | EN | 2.5K* | n/a | Charts | 231.8 |
| CharXiv Wang et al. (2024) | EN | 11.6K | n/a | Scientific charts | 165.1 |
| OCRBench v2 Fu et al. (2025) | EN, ZH | 10K | n/a | Mixed text-rich images | 437.2 |
| OCR-Reasoning Huang et al. (2026) | EN | 1.1K | n/a | Everyday text-rich scenes | 514.3 |
| MWS Vision Bench MWS AI (2025) | RU | 1.3K** | 400 | Business/personal documents | 1,126.4 |
Table 1: Comparison of BEAR-Bench to existing multimodal reasoning benchmarks. #QA pairs refers to the evaluation split; DocVQA and ChartQA additionally provide training data (50K and 32.7K items in total, respectively), while the remaining benchmarks are evaluation-only. For CC-OCR V2, the reported count is the document QA track (2K of 7.1K items). #RU reasoning VQA : number of evaluation questions in Russian that require answering from the image, excluding OCR, parsing, grounding, and key-information extraction; n/a = not applicable (no Russian split). Image scope summarizes the principal visual sources represented in each benchmark. OCR chars/img : average number of non-whitespace OCR characters per image ( randomly sampled images per benchmark). *Evenly split between human-written and machine-generated questions. **Publicly available. .
Figure 1: Representative items from BEAR-Bench. Each card displays the source document image (left) alongside the human-authored multi-hop question and ground-truth answer (right).
Figure 2: BEAR-Bench dataset statistics. (a) Item distribution across the four domain language cells; the central numeral indicates the total count. (b) Distribution of complexity scores over all 1,000 items, where each score reflects the total number of reasoning and computational steps required to solve the corresponding question. (c) Prevalence of visual content types, diagrams, equations, code, and figures, disaggregated by subset; bars show absolute counts. Items may carry multiple content-type labels simultaneously.
| Model | Overall | Business | Science | RU | EN | Figures | Diagrams | Equations |
|---|---|---|---|---|---|---|---|---|
| ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | |
| Proprietary models | ||||||||
| Qwen3.5-397B-A17B | 75.4 | 78.2 | 72.2 | 71.4 | 81.9 | 65.7 | 76.9 | 74.9 |
| Gemini 3.1 Pro | 75.1 | 79.5 | 70.1 | 70.2 | 83.0 | 65.5 | 76.3 | 71.4 |
| Gemini 2.5 Pro | 67.4 | 72.7 | 61.3 | 62.8 | 74.9 | 62.4 | 66.8 | 62.3 |
| Qwen 3.6 Plus | 65.0 | 59.4 | 71.4 | 62.5 | 69.1 | 62.7 | 62.4 | 69.3 |
Table 2: BEAR-Bench leaderboard. Results are reported as accuracy (%) across all evaluation subsets. Bold denotes the best result in each column.
Figure 3: Accuracy on BEAR-Bench versus model size for the Qwen3.5 and Qwen3-VL-Instruct families (parameters on a log axis), reported separately for English (solid) and Russian (dashed) items. The English-over-Russian gap persists across scales.
| Model | No CoT | CoT | |
|---|---|---|---|
| Qwen3.5-9B | 49.6 | 49.3 | 0.3 |
| Qwen3.5-4B | 48.6 | 47.2 | 1.4 |
| Qwen3-VL-4B-Thinking | 34.5 | 34.9 | 0.4 |
| Qwen3-VL-8B-Instruct | 25.4 | 39.1 | 13.7 |
Table 3: Overall accuracy (%) with and without an explicit chain-of-thought prompt. is CoT minus the default (no-CoT) protocol used in Table 2 .
| Downsampling factor | |||||
|---|---|---|---|---|---|
| Accuracy (%) | 49.6 | 45.0 | 33.0 | 12.6 | 4.7 |
Table 4: Overall accuracy of Qwen3.5-9B on BEAR-Bench at different image downsampling factors. A factor of means that both the width and height are divided by .
Figure 4: Error-type distribution in a manually annotated subsample of Gemini 3.1 Pro responses ( ). Samples may receive multiple labels.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Illustrative items from the public validation split of MWS Vision Bench MWS AI (2025) . The mix of personal handwriting, receipts, and document-processing tasks differs from BEAR-Bench’s professional, text-dense pages.
Figure 6: Accuracy on BEAR-Bench as a function of reasoning depth (number of steps per item) for three frontier models.
Figure 7: Accuracy on BEAR-Bench as a function of reasoning depth (number of steps per item) for four open-weight models.
| Group | Model | Median words |
|---|---|---|
| Proprietary / API | Gemini 3.1 Pro | 3 |
| Gemini 3.1 Flash | 46 | |
| Gemini 2.5 Pro | 3 | |
| Gemini 2.5 Flash | 123 | |
| Qwen 3.6 Plus | 2 | |
| Qwen3.5 397B A17B | 2 |
Table 5: Median response length (words) by model. Words are whitespace-split tokens from each model’s stored answer field.
| Method | AUROC | AUC-PR | BalAcc |
|---|---|---|---|
| Gemini 3.1 Pro | |||
| Max prob | 0.64 | 0.38 | 0.60 |
| Avg prob | 0.48 | 0.26 | 0.48 |
| Log-likelihood | 0.50 | 0.29 | 0.46 |
| Max entropy | 0.60 | 0.41 | 0.52 |
| Avg entropy | 0.50 | 0.26 | 0.52 |
Table 6: Hallucination detection (5-fold CV): Gemini models. AUROC and AUC-PR are omitted for VLM-as-judge because the judge returns binary verdicts.
| Method | AUROC | AUC-PR | BalAcc |
|---|---|---|---|
| Qwen 3.6 Plus | |||
| Max prob | 0.72 | 0.61 | 0.66 |
| Avg prob | 0.62 | 0.48 | 0.56 |
| Log-likelihood | 0.67 | 0.58 | 0.57 |
| Max entropy | 0.62 | 0.50 | 0.56 |
| Avg entropy | 0.59 | 0.44 | 0.57 |
Table 7: Hallucination detection (5-fold CV): Qwen and Claude models. AUROC and AUC-PR are omitted for VLM-as-judge because the judge returns binary verdicts.
| Method | AUROC | AUC-PR | BalAcc |
|---|---|---|---|
| Qwen3.5-9B | |||
| Max prob | 0.63 | 0.63 | 0.58 |
| Avg prob | 0.74 | 0.74 | 0.67 |
| Log-likelihood | 0.73 | 0.74 | 0.68 |
| Max entropy | 0.69 | 0.69 | 0.64 |
| Avg entropy | 0.74 | 0.75 | 0.68 |
Table 8: Hallucination detection (5-fold CV): open-weight Qwen3.5 models (native signals). The VLM-as-a-judge results are computed over 997 valid samples for each model. AUROC and AUC-PR are omitted for VLM-as-judge because the judge returns binary verdicts.
Figure 8: Chain-of-thought system prompt used in the CoT evaluation condition.
Figure 9: Original Russian instructions given to benchmark annotators. The excerpt states the annotation objective (one multi-step question per document image with a ground-truth answer) and Section 2, which defines question requirements: at least three reasoning steps, OCR-grounded text, exemplar chain types, items to avoid, and answer constraints. The translation to English is provided in Figure 10 .
Figure 10: English translation of the annotator instructions shown in Figure 9 .
Figure 11: LLM-as-a-judge prompt used to score model responses against the ground truth.
More in this topic
Multimodal Reasoning
Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.
See all 508 papers