Recognition Is Not Grounding: Benchmarking Multilingual Financial Evidence Extraction

Lingfei Qian; Yan Wang; Xueqing Peng; Polydoros Giannouris; Nadhem Zmandar; Yi Han; Yanguang Liu; Víctor Gutiérrez-Basulto

cs.AIOct 14, 2026

Recognition Is Not Grounding: Benchmarking Multilingual Financial Evidence Extraction

Authors: Lingfei Qian, Yan Wang, Xueqing Peng, Polydoros Giannouris, Nadhem Zmandar, Yi Han, Yanguang Liu, Víctor Gutiérrez-Basulto

Organizations: The Fin AI, USA · University of Manchester, UK · Georgia Institute of Technology, USA · New Jersey Institute of Technology, USA ·

Abstract

Comparing financial disclosures across languages requires identifying shared accounting concepts and recovering all numerical facts that express them. Existing benchmarks evaluate aspects of these capabilities, but provide limited insight into whether both succeed on the same multilingual evidence. In particular, it remains unclear whether language models can recognize these concepts across languages and exhaustively recover their corresponding numerical evidence from the same contexts. We introduce MULTIFINGROUND, a benchmark designed to investigate this recognition-grounding gap in multilingual financial reports. Constructed from FY2025 disclosures of 90 issuers in English, French, and Greek, MULTIFINGROUND aligns independently authored reports through a shared accounting taxonomy and provides traceable fact-level annotations. The benchmark evaluates financial fact identification, concept recognition, exhaustive fact selection, and concept discrimination without candidate labels. Across 15 evaluated models, eight API and open-weight reference models achieve 97.3-99.4% concept recognition accuracy, yet miss at least one target fact in 26.6-41.9% of correctly recognized groups. Further analyses reveal substantial variation in concept discrimination and limited gains from the evaluated finance-specialized models. Our findings demonstrate that strong concept recognition can coexist with incomplete evidence recovery, highlighting the importance of evaluating semantic recognition and evidence completeness together.

Figures & tables

Benchmark	Concept Grounding	Fact Coverage	Multilingual	Concept Discrimination	Paired Evaluation
FiNER ( Loukas et al., 2022 )	✓	✓	✗	✗	✗
FNXL ( Sharma et al., 2023 )	✓	✓	✗	✗	✗
FinTagging ( Wang et al., 2026 )	✓	✓	✗	✗	✗
FinQA ( Chen et al., 2021 )	✗	✓	✗	✗	✗
TAT-QA ( Zhu et al., 2021 )	✗	✗	✗	✗	✗
MultiFin ( Jørgensen et al., 2023 )	✗	✗	✓	✗	✗

Table 1: Comparison of benchmarks for financial and multilingual numerical grounding. Concept grounding evaluates the mapping of numerical mentions to semantic concepts. Fact coverage assesses recovery of annotated evidence, including named entities and supporting facts, rather than only final answers; it need not concern numerical facts. Multilingual indicates evaluation across multiple languages. Concept discrimination tests whether evidence instances express the same or different concepts without candidate concept labels; it does not require an odd-one-out format. Paired evaluation examines the relationship between concept recognition and evidence coverage on matched instances, rather than simply including both tasks. Checks may refer to different tasks within a suite, including FNXL in MultiFinBen; FinQA reports retrieval recall. ✓ indicates an explicitly evaluated capability; ✗ indicates that the capability is not an explicit evaluation target.

Figure 1: MultiFinGround overview. Reports from 30 issuers per language provide evidence linked through shared accounting concepts. Four independent tasks test Fact Identification (T1), Concept Recognition (T2a), Exhaustive Fact Selection (T2b) and Concept Discrimination without candidate labels (T2c). T2a/T2b use matched contexts to relate recognition to target coverage. Examples are schematic; the finding concerns omissions among correctly recognized groups for the eight reference models.

	EN	FR	EL
Issuers (FY2025 annual reports)	30	30	30
T1 items (tables / text)	435 (368 / 67)	450 (368 / 82)	422 (415 / 7)
marked numbers per item	35.2	35.9	30.1
financial facts (gold)	11,583	12,191	10,287
monetary, instant, debit	1,277	1,360	1,216
monetary, instant, credit	2,408	2,502	2,311

Table 2: Benchmark statistics. Financial-fact annotations originate from issuer iXBRL tags, with expert corrections and exclusions incorporated for the audited subset (

\lx@sectionsign

3.8 ). T1 counts financial facts by type, period type and balance; non-financial numbers are scored in class none , while untagged amount-like numbers are unscored.

API models
	T1 (7-class macro-F1)					T2a	T2b		T2c	Invalid (%)
Model	All [95% CI]	EN	FR	EL	Text	Acc.	Macro-F1	Exact set	Acc.	T1/T2a/T2b/T2c
GPT-5.6 Luna	88.6 [87.3, 89.5]	87.2	88.5	84.7	73.6	97.3	88.3	65.8	49.4	0/0/0/0
Claude Haiku 4.5	85.9 [84.4, 87.0]	84.6	86.4	83.7	71.9	97.5	86.2	60.1	59.9	0/0/0/0
Gemini 3.8 Flash	95.4 [94.5, 96.2]	95.9	95.1	95.1	86.9	99.4	89.2	65.6	95.1	1/0/0/0
DeepSeek-V4.1-Flash	88.1 [87.1, 88.9]	87.7	86.9	88.1	72.4	98.3	86.2	59.7	35.6	0/0/0/8

Table 3: Main results (%, strict protocol: invalid outputs count as wrong). T1 intervals resample issuers within language (2,000 bootstrap samples); T2 items share issuers across groups, so no T2 intervals are reported (Section 4 ). Text: T1 text passages only.

Figure 2: Recognition and target coverage over 521 groups. The first two categories require correct T2a recognition; coverage concerns all three languages in T2b. Selecting all targets permits extra selections and is not exact-set correctness. Colours: API (blue), open-weight (orange), finance (green), base (grey).

Figure 3: T2a changes across evidence and label conditions (points). Evidence conditions compare with joint inputs; C2 compares localized labels without identifiers with C1’s English labels and identifiers; C3 adds multilingual labels. C2 does not isolate label language. * McNemar $p<0.05$ . Scores: Table 6 .

Figure 4: From recognition to discrimination. T2a accuracy (shared concept, four labelled options) and T2c accuracy (odd one out, no candidate labels) of each model; dashed line: chance. The tasks also differ in evidence and structure, so the score gap does not isolate the effect of candidate labels.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Group	Access	Reasoning in our runs
GPT-5.6 Luna	API	OpenAI gpt-5.6-luna (via Azure OpenAI)	off
Claude Haiku 4.5	API	Anthropic claude-haiku-4-5 (via Microsoft Foundry)	off
Gemini 3.8 Flash	API	Google gemini-3.8-flash	off
DeepSeek-V4.1-Flash	API	DeepSeek deepseek-flash	off
Qwen3.8-27B	Open weights	Qwen/Qwen3.8-27B	model default (thinks)
Qwen3.8-Flash-Next	Open weights	Qwen/Qwen3.8-Flash-Next	model default (thinks)

Appendix

Table 4: Evaluated models : four API models (one per provider family), four open-weight models, four finance models and the three base models they were adapted from. Open weights are served locally with SGLang in their default chat template and generation configuration; API models run with reasoning switched off. Every model receives the same prompts and an output budget of 16,384 tokens including reasoning.

API models
	T2a	Miss $\geq$ 1	T2b recall
Model	correct	language	EN	FR	EL
GPT-5.6 Luna	97.3	29.0	82.1	83.5	80.3
Claude Haiku 4.5	97.5	36.8	77.7	78.7	77.1
Gemini 3.8 Flash	99.4	34.6	81.7	82.2	75.1
DeepSeek-V4.1-Flash	98.3	38.5	74.2	76.7	73.4

Appendix

Table 5: Recognition versus grounding (%). Among groups whose T2a concept is answered correctly, the share whose T2b selection misses at least one target number in some language, and T2b recall per language on those groups.

API models
	T2a evidence languages (C1)						T2a option labels			T2c
Model	Joint	No EN	EN	FR	EL	$\Delta_{\text{EN}}$	FR (C2)	EL (C2)	Joint (C3)	Acc.	Same lang.	Unique
GPT-5.6 Luna	97.3	97.3	96.5	98.5	95.8	+0.0	96.9	95.0	96.9	49.4	46.9	52.1
Claude Haiku 4.5	97.5	98.3	95.0	97.3	94.8	$-$ 0.8	94.2	91.6	96.5	59.9	57.8	62.2
Gemini 3.8 Flash	99.4	99.2	97.5	99.8	98.3	+0.2	99.4	97.3	99.6	95.1	93.1	97.3
DeepSeek-V4.1-Flash	98.3	98.3	95.4	99.2	96.9	+0.0	96.2	93.3	98.1	35.6	30.5	40.9

Appendix

Table 6: Language and label ablations (accuracy, %). T2a: joint EN+FR+EL evidence, without English, or a single language, with English labels (C1);

\Delta_{\text{EN}}

= joint

-

no EN. C2: French or Greek official labels without concept IDs on the single-language item; C3: English, French and Greek labels on the joint item. T2c: odd-one-out accuracy, split by whether another excerpt shares the odd excerpt’s language.

API models
	Joint EN+FR+EL			No EN
Model	EN F1	FR F1	EL F1	FR F1	EL F1	Macro-F1
GPT-5.6 Luna	85.0	88.3	83.9	88.6	84.3	89.0
Claude Haiku 4.5	85.3	85.4	83.6	86.2	81.1	85.2
Gemini 3.8 Flash	89.8	90.3	85.3	90.7	86.2	89.4
DeepSeek-V4.1-Flash	83.8	85.0	82.3	83.8	79.3	84.5

Appendix

Table 7: T2b by language (F1 of the selected numbers per language, %), with and without the English excerpt.

API models
Model	Acc.	Odd EN	Odd FR	Odd EL	Invalid
GPT-5.6 Luna	49.4	40.3	49.4	63.4	0.0
Claude Haiku 4.5	59.9	57.5	62.5	60.7	0.0
Gemini 3.8 Flash	95.1	95.5	94.6	95.2	0.0
DeepSeek-V4.1-Flash	35.6	38.5	34.5	32.4	8.2
Open-weight models

Appendix

Table 8: T2c by the language of the odd excerpt (accuracy and invalid rate, %; chance 25%).

API models
	Monetary							On identified facts
Model	I/Dr	I/Cr	D/Dr	D/Cr	Per sh.	Shares	None	Period	Balance	Joint F1
GPT-5.6 Luna	93.6	95.4	66.4	78.2	97.9	91.3	97.2	99.4	81.1	80.8
Claude Haiku 4.5	89.7	91.7	58.8	78.8	97.1	90.6	94.6	98.4	80.6	79.0
Gemini 3.8 Flash	97.5	98.8	85.1	93.2	98.3	99.8	95.3	99.7	94.5	93.3
DeepSeek-V4.1-Flash	90.2	93.0	67.2	80.1	97.4	94.7	93.8	99.2	82.5	81.4

Appendix

Table 9: T1 per class (F1, %; I/D = instant/duration, Dr/Cr = debit/credit) and attribute accuracy on identified facts; joint F1 requires the ID, type, period and balance to be correct.

Figure 5: T1 by attribute class (F1, %). Monetary classes combine period (inst. instant, dur. duration) and balance (Dr debit, Cr credit); none: plainly non-financial numbers; macro: 7-class macro-F1. Models sorted by macro-F1.

	T1	T2a	T2b	T2c	T1 by language
Specialist $-$ base model	F1	Acc.	F1	Acc.	EN	FR	EL
Fin-R1 $-$ Qwen2.5-7B-Instruct	$-$ 8.4	$-$ 5.6	+9.9	$-$ 3.0	$-$ 12.1	$-$ 5.8	$-$ 7.0
Fin-o1-8B $-$ Qwen3-8B	$-$ 19.7	$-$ 6.9	$-$ 6.1	$-$ 21.3	$-$ 18.9	$-$ 20.8	$-$ 18.3
ODA-Fin-RL-8B $-$ Qwen3-8B	$-$ 0.9	$-$ 3.6	+0.6	$-$ 23.0	$-$ 2.1	$-$ 0.1	$-$ 0.5
Plutus-8B-Instruct $-$ Llama-Krikri-8B-Instruct	$-$ 24.7	$-$ 6.5	$-$ 34.2	$-$ 0.2	$-$ 27.6	$-$ 19.6	$-$ 25.4

Appendix

Table 10: Finance specialization : score of each finance model minus its base model (percentage points; T1 7-class macro-F1, T2a/T2c accuracy, T2b macro-F1).

Figure 6: Finance models versus their base models (points; T1 7-class macro-F1, T2a/T2c accuracy, T2b macro-F1). Red: the specialist scores lower; blue: higher. These are observed differences between released models under our protocol, not isolated effects of specialization. T1 by language in Table 10 .

API models
	Joint EN+FR+EL			No EN
Model	4	4 hard	20	4	4 hard	20
GPT-5.6 Luna	97.3	94.8	89.4	97.3	95.6	89.8
Claude Haiku 4.5	97.5	95.6	90.6	98.3	96.2	91.2
Gemini 3.8 Flash	99.4	98.7	97.5	99.2	98.8	97.5
DeepSeek-V4.1-Flash	98.3	95.2	90.8	98.3	96.5	90.4

Appendix

Table 11: T2a with harder option sets (accuracy, %): the original four options, four harder options (calculation siblings and wording-matched distractors), and twenty options (chance 5%), on the same items. Lexical overlap picks the option whose English label shares most words with the English rows of the marked numbers.

	T1	T2a	T2b	T2c
Items reviewed (10% sample)	131	52	52	53
Units judged	4,213 numbers	156 distractors	2,108 facts	212 excerpts
Cohen’s $\kappa$ (independent reviews)	0.87	0.74	0.84	0.88
Raw agreement (%)	98.6	88.5	97.3	94.3
Items needing discussion (%)	22.1	19.2	25.0	9.4
Items with at least one correction (%)	14.5	28.8	15.4	5.7

Appendix

Table 12: Expert audit of a stratified 10% sample per task. Units are the judged elements: scored marked numbers (T1), distractors (T2a), numbered facts (T2b) and excerpts (T2c). Agreement is computed from the two independent reviews before discussion. Corrections and dropped items follow the joint final decisions. Score change: largest absolute difference, over the eight reference models, between scores on the audited subsets with original automatically derived and expert-corrected annotations (T1 7-class macro-F1, T2a and T2c accuracy, T2b macro-F1).

English issuer (filing country)	Sector	French issuer	Sector	Greek issuer	Sector
KONE Corporation (FI)	Industrials	Safran	Industrials	Elastron	Materials
Airbus SE (NL)	Industrials	Thales	Industrials	Plastika Kritis	Materials
Bekaert (BE)	Industrials	Schneider Electric	Industrials	Alumil	Materials
STRABAG SE (AT)	Industrials	Legrand	Industrials	Flexopack	Materials
Heineken N.V. (NL)	Consumer	Bureau Veritas	Industrials	GEK TERNA	Industrials
Anheuser-Busch InBev (BE)	Consumer	Vinci	Industrials	AKTOR	Industrials

Appendix

Table 13: Issuers. One FY2025 annual financial report (fiscal year ending 31 December 2025) per issuer. English: ESEF filings indexed by filings.xbrl.org; French: CAC 40 constituents, from filings.xbrl.org or, marked † , from the AMF’s officially appointed mechanism; Greek: consolidated reports of Athens Exchange issuers. Sector labels are assigned manually.

Financial Language Model Evaluation

See all 23 papers

Language-Routed RAG and Direct Option Scoring for Multilingual Financial QA: DS@GT at FinMMEval
Jul 24, 2026Justice Ayela, Kabir SahniFinance Multilingual Language Model Evaluation
Georgia Institute of Technology, North Ave NW, Atlanta, GA 30332
Overview of FinMMEval 2026 Task 1: Multilingual Financial Multiple-Choice Question Answering
Jul 22, 2026Zhuohan Xie, Yuyang Dai, Rania Elbadry +18Multilingual Language Model Evaluation Multiple-Choice Question Answering
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria · University of Arizona, Tucson, United States +6
FIND: Toward Multimodal Financial Reasoning and Question Answering for Indic Languages
May 13, 2026Sarmistha Das, Vaibhav Vishal, Syed Ibrahim Ahmad +2Financial Services Financial QA
Indian Institute of Technology Patna, India · Microsoft, India

Jul 24, 2026J/K move · Enter open · S save

Language-Routed RAG and Direct Option Scoring for Multilingual Financial QA: DS@GT at FinMMEval

Justice Ayela, Kabir Sahni

Georgia Institute of Technology, North Ave NW, Atlanta, GA 30332

Finance Multilingual Language Model Evaluation Retrieval-Augmented Generation Multilingual QA LLM Routing Multiple-Choice Question Answering+3

Open paper AskPDF arXiv

We present DS@GT's submission to FinMMEval 2026 Task 1, a multilingual financial exam question answering benchmark spanning English, Spanish, Greek, Chinese, and Hindi. Financial certification exams such as the CFA, EFPA, and CPA demand structured domain reasoning that standard NLP benchmarks do not capture, and this challenge compounds across languages where retrieval and representation infrastructure is underdeveloped. We build a retrieval-augmented pipeline on LangGraph that detects query language and retrieves semantically relevant exemplars from a 30,209-entry multilingual knowledge base using BGE-M3 embeddings and FAISS indexing. The system then scores answers via Retrieval-Augmented Direct Scoring (RADS), reading next-token log-probabilities over candidate option letters rather than generating free-form output. For low-resource languages, we fuse per-language and cross-lingual retrieval indices using weighted Reciprocal Rank Fusion. Model selection is language-routed: Qwen3-14B for Arabic, Chinese, and Hindi; Qwen2.5-14B for English; and Llama-3.1-8B for Greek, a routing derived from empirical ablations that reveal substantial language-asymmetric performance gaps. Notably, chain-of-thought prompting significantly degrades Greek accuracy (90.7% to 20.9%), and enabling Qwen3's default thinking mode collapses Arabic RADS performance to near-chance levels. Our results indicate that effective multilingual financial reasoning requires language-aware retrieval, model routing, and deliberate scoring strategy selection.

Abstract

Financial Language Model Evaluation

Explore similar work

Language-Routed RAG and Direct Option Scoring for Multilingual Financial QA: DS@GT at FinMMEval

Overview of FinMMEval 2026 Task 1: Multilingual Financial Multiple-Choice Question Answering

FIND: Toward Multimodal Financial Reasoning and Question Answering for Indic Languages