ChemCLIR-Bench: Benchmarking Cross-Lingual Information Retrieval in Multilingual Chemical Patents

Mahdi Astaraki; Mohammad Khodadad; Reza Namazi; Mohammad Arshi Saloot; Amir Reza Behzad Moghadam; Hamidreza Mahyar; Soheila Samiee

cs.CLSep 19, 2026

ChemCLIR-Bench: Benchmarking Cross-Lingual Information Retrieval in Multilingual Chemical Patents

Authors: Mahdi Astaraki, Mohammad Khodadad, Reza Namazi, Mohammad Arshi Saloot, Amir Reza Behzad Moghadam, Hamidreza Mahyar, Soheila Samiee

Organizations: Faculty of Engineering, McMaster University, Canada · BASF Canada Inc., Canada

Abstract

Cross-lingual information retrieval (CLIR) is increasingly important in multi-national industries, where critical technical evidence may exist in a different language than the query. However, existing benchmarks do not adequately capture domain-specific cross-lingual retrieval or the retrieval-depth and recoverability failures that aggregate recall hides. In this work, we benchmark CLIR in the chemical domain, with a focus on patent data. We construct a multilingual dataset from Google Patents and the European Patent Office (EPO) data, spanning five languages (covering major Eastern and Western languages) and reflecting the diversity and complexity of real-world industrial documentation. Using this dataset, we systematically evaluate eight state-of-the-art embedding models for cross-lingual retrieval. Our results show a substantial performance gap between monolingual and cross-lingual settings: for the best-performing model, Recall@10 drops from 0.72 to 0.53 in cross-lingual setting. Retrieval depth also degrades significantly, with relevant documents ranked lower across languages in cross-lingual scenarios. Furthermore, some multilingual embedding models that perform strongly in monolingual settings exhibit sharp declines when queries and documents are in different languages, providing practical insights for model selection in cross-lingual use cases. These findings highlight critical limitations of current approaches and emphasize the need for more robust cross-lingual retrieval methods in domain-specific settings. Our benchmark provides actionable insights for model selection and establishes a controlled diagnostic evaluation framework for CLIR over industrial technical text. Data and code are publicly available at https://github.com/MohammadKhodadad/Multi-Lingual-QAC.

Figures & tables

Figure 1: Overview of the QAC generation pipeline: source documents are converted into grounded candidate questions, scored, and exported as retrieval-ready records. The pipeline optimized with three rounds of domain expert feedback collection.

Source	Corpus	Queries	Qrels	Cross-lang qrels	Q tokens	Q vocab	C tokens	C vocab
Google Patents	23,787	524	1,284	1,023	11.3 $\pm$ 6.3	2,742	145.0 $\pm$ 63.8	83,333
EPO	11,315	198	594	396	14.1 $\pm$ 5.4	1,371	178.2 $\pm$ 71.1	72,256

Table 1: Statistics of the two released benchmarks. A corpus document (C) is a title plus abstract (Google Patents) or title, abstract, and first claim (EPO) in one language; Q denotes queries. Token columns give the mean

\pm

standard deviation per query or per document; vocabulary columns give the number of distinct case-folded tokens over all queries or all documents of a benchmark, pooled across languages.

		Google Patents		EPO		Optimum K for retrieval
Model	Size	R@10 same	R@10 cross	R@10 same	R@10 cross	$k^{\star}_{80}$	$k_{\mathrm{80same}}$	$k_{\mathrm{80cross}}$
embeddinggemma	300M	0.74	0.54	0.70	0.52	147 $\pm$ 38	36 $\pm$ 13	121 $\pm$ 40
bge-m3	568M	0.64	0.47	0.71	0.47	304 $\pm$ 75	66 $\pm$ 24	215 $\pm$ 45
qwen3	600M	0.62	0.45	0.63	0.39	425 $\pm$ 93	63 $\pm$ 16	310 $\pm$ 82
nomic-v2-moe	475M	0.65	0.41	0.67	0.43	367 $\pm$ 77	55 $\pm$ 14	321 $\pm$ 68
granite	278M	0.54	0.39	0.43	0.36	$>$ 1000	641 $\pm$ 167	569 $\pm$ 153

Table 2: Retrieval performance (Recall@10) on both patent datasets, split by language route: R@10 same is same-language (monolingual) and R@10 cross is cross-language, each language-balanced. The

k

columns are right-censored 80th-percentile retrieval depths (lower is better;

\pm

values are bootstrap standard errors over queries, 2000 resamples; omitted where the depth is censored at

>

1000). Best per column in bold. Size is total parameters.

Figure 2: Recall@10 by query language $\times$ document language, per model (diagonal = same-language; off-diagonal = cross-lingual).

Figure 3: Deployment cost of cross-lingual retrieval (e5-large-instruct excluded by the R@10 ${}_{\mathrm{cross}}<0.10$ gate). (a) XRC50 reading cost; (b) RRC@ $K$ re-ranker recoverability with knee $K^{\star}$ ( $\bullet$ ); (c) the shortfall split into re-rankable (top-10), deeper-pool (top-1000), and an alignment-only floor.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Domain	Lang.	Setting	Corpus	Relevance	Objective
ChemCLIR-Bench (ours)	Chem. patents	5	Mono + CL	Patents (GP, EPO)	Parallel-doc	CLIR diagnostics
MIRACL	General	18	Mono	Wikipedia	Human	Retrieval
MMTEB	General	250+	Mono (some CL)	Aggregated	Inherited	Emb. suite
CLIRMatrix	General	139 / 8	CL	Wikipedia	Synthetic	Retrieval
NeuCLIR	News (+tech.)	EN $\rightarrow$ 3	CL, MLIR	News, abstracts	Human	Retrieval
ChemTEB	Chemistry	1 (EN)	Mono	PubChem, Wiki	DB-derived	Emb. suite

Appendix

Table S.1: ChemCLIR-Bench alongside general multilingual IR benchmarks and chemistry-focused benchmarks. Column values are abbreviated; Appendix A.1 defines them and gives the details behind each cell.

Figure S.1: Per query language, Recall@10 splits into same-language gold (R@10 same ) and cross-language gold (R@10 cross ), pooled over the non-degenerate models. The home advantage holds in every language.

Figure S.2: Distribution of human-annotator overall scores on the 106-item audited QAC sample (mean 8.33/10; no item rated below 6).

		Google Patents				EPO
		R@25		R@50		R@25		R@50
Model	Size	same	cross	same	cross	same	cross	same	cross
embeddinggemma	300M	0.80	0.63	0.85	0.68	0.76	0.61	0.82	0.68
bge-m3	568M	0.71	0.54	0.76	0.59	0.74	0.58	0.80	0.67
qwen3	600M	0.70	0.54	0.75	0.61	0.72	0.49	0.77	0.57
nomic-v2-moe	475M	0.75	0.49	0.82	0.56	0.74	0.53	0.78	0.60

Appendix

Table S.2: Recall at deeper cutoffs (

R@25

R@50

) complementing Table 2 , split by language route: same is same-language monolingual recall and cross is cross-language recall, each language-balanced as the mean of per-language means. Best per column in bold. Size is total parameters; nomic-v2-moe is a mixture-of-experts with 475M total and 305M active parameters. † e5-large-instruct fails the cross-lingual degeneracy gate; its bold same-language entries reflect strong monolingual retrieval despite collapsed cross-lingual recall.

Figure S.3: Cross-lingual reading-cost decomposition for embeddinggemma . On a log-compressed ranked list, the same-language gold typically surfaces shallow while its cross-language counterpart sits about $5\times$ deeper (the cross-lingual reading cost XRC; the example ranks 2 and 10 illustrate this median multiplier). Reading down the list partitions where the cross-language gold is recovered, with measured shares over the pooled GP+EPO queries that have both gold types: a re-ranker over the top-100 reaches $\mathrm{RRC}@100=78\%$ , a further $+15\%$ require a deeper top-1000 pool ( $\mathrm{RRC}@1000-\mathrm{RRC}@100$ ), and a $7\%$ residual is never retrieved within the top-1000 ( $L_{\infty}$ floor) and can be closed only by better cross-lingual alignment ( $\mathrm{ARI}@100=0.32$ ). Percentages are measured; the two ranked positions illustrate the reported XRC.

Figure S.4: IPC class distribution of the ChemCLIR-Bench corpus (25 most frequent classes), stacked by source. Counts are at the IPC class level (three-character codes) and are not mutually exclusive, since a document may carry several codes. Classes without a description on the axis: H01 basic electric elements (including electrochemical cells, H01M), G01 measuring and testing, B32 layered products, B29 working of plastics, B22 casting and powder metallurgy, H10 semiconductor devices, B23 machine tools.

Multilingual IR

Multilingual information retrieval (IR) finds and ranks relevant information across languages, scripts, and language-mismatched queries and documents.

See all 53 papers

Benchmarking Patent Embeddings: A Multi-Task Evaluation of 22 Models Across Retrieval, Classification, and Clustering
May 22, 2026Amirhossein Yousefiramandi, Ciaran CooneyLegal IR Clustering
Clarivate, Intellectual Property Barcelona, Spain 08025
Citation-Driven Multi-View Training for Patent Embeddings: QaECTER and Sophia-Bench
Apr 24, 2026Younes Djemmal, You Zuo, Kim Gerdes +1Legal IR Text Embedding Models
ALMAnaCH · 1AI Lab, Questel, Paris, France · 3Inria, Paris, France +3
MIMO: Multilingual Information Retrieval via Monolingual Objectives
May 29, 2026Youngjoon Jang, Seongtae Hong, Heuiseok LimContrastive Learning Cross-Lingual Representation Alignment
Department of Computer Science and Engineering, Korea University

May 22, 2026J/K move · Enter open · S save

Benchmarking Patent Embeddings: A Multi-Task Evaluation of 22 Models Across Retrieval, Classification, and Clustering

Amirhossein Yousefiramandi, Ciaran Cooney

Clarivate, Intellectual Property Barcelona, Spain 08025

Legal IR Clustering Text Embedding Evaluation Text Classification Information Retrieval IR Evaluation

Open paper AskPDF arXiv

Two questions regarding practitioners' use of patent embeddings arise: (i) Does one fine-tuning recipe suffice for all downstream applications? (ii) Is fine-tuning on one patent landscape sufficient for downstream application on other landscapes? By evaluating 22 pre-trained embedding models (ranging from 22M to 12B parameters) on three tasks -- information retrieval, classification, and clustering -- on 113,148 WIPO patents for assistive technology (46,069 citation queries) and on an external DAPFAM dataset, we find that two results cast doubt on the prevailing wisdom. (i) The optimal fine-tuning recipe depends on the downstream task: cross-sectional alignment (recipe R3) provides the largest improvements to retrieval performance (+7.1% nDCG@10), whereas a combined signal recipe (recipe R4) is better suited to classification (+7.1 F1) and clustering (+10.9 V-measure); a matched data control confirms that differences in training dataset size are not a contributing factor. (ii) Single-landscape fine-tuning hampers cross-landscape information retrieval: fine-tuning on one landscape significantly degrades cross-domain retrieval for 5 of 8 model-recipe combinations on the DAPFAM corpus, with the stronger zero-shot models suffering most. While within-family scaling is consistent (Qwen3 0.6B->4B->8B; Llama-Nemotron 1B->8B), cross-family scaling is erratic; the 12B KaLM-Gemma3 is ranked 8th on TAC retrieval performance, following prefix modification. Title+Abstract+Claims is the ubiquitous best text view, and all models suffer from a 55-65% gap between IN and OUT-of-domain performance which cannot be mitigated by hybrid BM25-dense fusion. Code and evaluation framework are publicly available.

Abstract

Multilingual IR

Explore similar work

Benchmarking Patent Embeddings: A Multi-Task Evaluation of 22 Models Across Retrieval, Classification, and Clustering

Citation-Driven Multi-View Training for Patent Embeddings: QaECTER and Sophia-Bench

MIMO: Multilingual Information Retrieval via Monolingual Objectives