ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings

Ali Shiraee Kasmaee; Mohammad Khodadad; Mahdi Astaraki; Mohammad Arshi Saloot; Nicholas Sherck; Hamidreza Mahyar; Soheila Samiee

cs.IRAug 3, 2025

ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings

Authors: Ali Shiraee Kasmaee, Mohammad Khodadad, Mahdi Astaraki, Mohammad Arshi Saloot, Nicholas Sherck, Hamidreza Mahyar, Soheila Samiee

Organizations: Department of Computational Science and Engineering, McMaster University, Canada · BASF Canada Inc., Canada · BASF Corporation, USA

Abstract

Retrieval-Augmented Generation (RAG) systems in chemistry heavily depend on accurate and relevant retrieval of chemical literature. However, general-purpose text embedding models frequently fail to adequately represent complex chemical terminologies, resulting in suboptimal retrieval quality. Existing embedding models for chemistry are outdated, and none is tailored to chemical literature retrieval, leaving a substantial performance gap. To address this challenge, we introduce ChEmbed, the first purpose-built family of domain-adapted text embedding models engineered for chemical literature retrieval. These models are fine-tuned via contrastive learning on a dataset comprising chemistry-specific text from the PubChem, Semantic Scholar, and ChemRxiv corpora. To create effective training data, we employ large language models to synthetically generate queries, resulting in approximately 1.7 million high-quality query-passage pairs. Additionally, we augment the tokenizer by adding 900 chemically specialized tokens to previously unused slots, which reduces the fragmentation of chemical entities, such as IUPAC names. ChEmbed also maintains an 8192-token context length, enabling retrieval of longer passages than many open-source embedding models allow. Evaluated on our newly introduced ChemRxiv Retrieval benchmark, ChEmbed outperforms state-of-the-art general embedding models, raising MRR@10 from 0.781 to 0.882 (+10.1 pp). It also substantially outperforms domain-specific embedding models such as Chemical-BERT, improving MRR@10 from 0.096 to 0.882. A role-based retrieval analysis using PubChem descriptions and ChEBI annotations shows that the improvement extends to chemical-role queries. ChEmbed represents a practical, lightweight, and reproducible embedding solution that effectively improves chemical literature retrieval.

Figures & tables

Idx	Dataset	Description	# Samples	Usage	LLM used for query synthesis
1	PubChem compounds	Title, IUPAC, SMILES, synonyms	2,087,164	Tokenizer	N/A
2	PubChem descriptions	Compound descriptions	393,321	Training	gpt-4.1-nano
3	S2ORC chemistry	Filtered chemistry paragraphs	1,187,726	Training	gpt-4.1-nano
4	ChemRxiv paragraphs	Extracted ChemRxiv paragraphs (CC-BY)	139,057	Training	o3-mini
5	ChemRxiv paragraphs	Extracted ChemRxiv paragraphs (CC-BY-NC)	69,457	Evaluation	Claude 3.7 Sonnet
6	ChemRxiv metadata	Title and abstract ChemRxiv preprints	30,378	Training	N/A

Table 1 : Summary of chemistry domain datasets used for training and evaluation

Figure 1 : Overview of the ChEmbed pipeline. Chemistry-related paragraphs are extracted from multiple sources and paired with synthetic queries generated by prompting large language models (LLMs). A held-out subset of 69,000 paragraphs is reserved for evaluation, with distinct LLMs used for generating training and evaluation queries. In parallel, a ChemVocab tokenizer is trained from more than two million IUPAC names, and 900 unique tokens are injected into the bert-base-uncased unused tokens. The base encoder (nomic-embed-text-v1) is then fine-tuned on the generated query-passage pairs to produce the final ChEmbed family of models.

Figure 2 : Expert-review interface for Task A. The reviewer inspected a generated query alongside its real source chemistry passage and recorded answerability, specificity, query quality, standalone clarity, scientific validity, and an optional qualitative note.

Term	ChemVocab (ours)	bert-base-uncased
chlorofluorocarbon	chloro/fluoro/carbon	ch/lor/of/lu/oro/carbon
acetaminophen	acet/amino/phen	ace/tam/ino/ph/en
secbutylamine	sec/butyl/amine	sec/bu/ty/lam/ine
nitrophenylphosphate	nitrophenyl/phosphate	ni/tro/ph/en/yl/ph/os/phate

Table 2: Qualitative comparison of chemical nomenclature segmentation: Our augmented ChemVocab maintains semantic sub-units (e.g., functional groups) compared to the over-segmentation observed in the general-purpose bert-base-uncased tokenizer.

Variant	$MRR@10$	$\Delta$ vs. baseline (pp)
nomic-embed-text-v1 (baseline)	0.781	–
ChEmbed vanilla	0.879	+9.8
ChEmbed full	0.869	+8.8
ChEmbed plug	0.878	+9.7
ChEmbed progressive	0.882	+10.1

Table 3: Impact of different tokenizer-adaptation variants on ChemRxiv retrieval performance

Figure 3 : Model efficiency on the ChemRxiv retrieval task. The x-axis represents retrieval speed, the y-axis shows ranking effectiveness measured by Mean Reciprocal Rank (

MRR@10

), and circle sizes represent the number of model parameters.

Open-Source Models
Model Name	Emb. size	#Params (M)	MRR@10	NDCG@10
chemical-bert-uncased	768	110.0	0.096	0.110
matscibert	768	110.0	0.117	0.137
nomic-bert-2048	768	137.3	0.019	0.025
ModernBERT-base	768	149.7	0.047	0.056
ModernBERT-large	1024	395.9	0.048	0.058

Table 4: Performance of embedding models on the ChemRxiv Retrieval task. ChemRxiv Retrieval has exactly one relevant document per query, so MAP@10 and MRR@10 coincide; we report MRR@10 only. “N/A” means the provider has not released parameter counts. Best scores per metric are shown in bold.

	ChemTEB					MTEB
Model	Cls	Clust	Pair	Mean (T)	Mean (T-type)	Cls	Clust	Pair	Mean (T)	Mean (T-type)
nomic-embed-text-v1-unsupervised	0.825	0.560	0.632	0.763	0.672	0.688	0.442	0.832	0.607	0.654
nomic-embed-text-v1	0.838	0.607	0.591	0.768	0.679	0.684	0.458	0.851	0.616	0.665
ChEmbed vanilla	0.807	0.508	0.585	0.736	0.633	0.698	0.434	0.846	0.610	0.659
ChEmbed full	0.809	0.509	0.550	0.730	0.623	0.698	0.433	0.844	0.609	0.658
ChEmbed plug	0.806	0.477	0.550	0.725	0.611	0.696	0.430	0.843	0.607	0.656

Table 5: Performance comparison across shared non - retrieval task categories in MTEB and ChemTEB, covering open - domain and chemistry - specific data, respectively. Evaluation metrics are: accuracy for Classification, V - measure for Clustering, and average precision (AP) for Pair Classification. Mean (Task) reports the average score across all individual tasks, while Mean (Task Type) first averages results within each task category and then computes the mean across the category - level averages.

		Dataset statistics			Retrieval Score b $\uparrow$
Task	Dataset	#Tasks	Avg Queries	Avg Corpus	nomic-embed-text-v1	ChEmbed vanilla	ChEmbed progressive
Chemistry Retrieval	ChemTEB(latest) a	3	1682	$34,153$	0.746	0.741	0.728
Open-domain Retrieval	MTEB Retrieval	10	1482	$109,645$	0.544	0.467	0.475

Table 6 : Retrieval performance comparison across chemistry-specific and open - domain tasks. Dataset descriptions and performance metrics for each model–dataset pair are provided.

Model	Average precision	Recall at 10
Nomic	0.176	20.4%
ChEmbed vanilla	0.203	22.7%
ChEmbed progressive	0.315	33.4%

Table 7 : Retrieval of direct ChEBI role members from the 44,887-description PubChem catalog.

Figure 4 : Chemical-role retrieval across the 518-group primary cohort. (a) Group-level average precision (AP) for ChEmbed progressive versus Nomic; points above the diagonal had higher average precision in fine-tuned ChEmbed model compared to the base Nomic model. (b) Mean AP over the 518 groups, before and after masking the exact ChEBI role phrase in the descriptions. Error bars are 95% bootstrap confidence intervals. Stars are paired

t

-tests against Nomic within each condition (

{}^{*}p<0.05

{}^{***}p<0.001

Figure 5 : Retrieval ranks for the eight PubChem records directly annotated by ChEBI as coagulation factor Xa inhibitors. For the displayed query, three records appeared among Nomic’s first ten results and seven among ChEmbed progressive ’s first ten. The eight records correspond to six inhibitor identities because edoxaban, edoxaban tosylate, and edoxaban tosylate monohydrate are separate records. Structures identify the records; retrieval used description text only.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Literature and encyclopedic passages in the base model, projected onto their first two principal components. PCA calculated the projection from the embeddings alone, and we added the corpus colors afterward. Although the two components contain only a small share of the total variance, the classes already occupy substantially different regions.

Figure S2 : Distributions of pairwise cosine similarity before and after fine-tuning with ChEmbed vanilla . Mean similarity falls from 0.570 to 0.030 for the ChemRxiv literature passages and from 0.517 to 0.163 for the balanced encyclopedic pool.

Explore similar work

Jun 13, 2025cs.IR

Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation

The retrieval stage of retrieval-augmented generation (RAG) for scientific question answering depends on how documents are segmented and how chunks are represented in embedding space. This dependence is especially relevant to chemistry texts, which contain dense terminology, symbolic notation, quantitative evidence, and context associated with document structure. However, benchmark-based evidence on the interaction between chunking strategy and embedding model remains limited for chemistry-specific retrieval. Using ChemQuests, a corpus of 952 question-answer pairs from 151 ChemRxiv papers across 17 chemistry subfields, we construct chunk-level, Massive Text Embedding Benchmark (MTEB)-compatible retrieval benchmarks for controlled evaluation. We first screen 41 embedding models on the external chemistry retrieval benchmarks ChemNQRetrieval and ChemHotpotQARetrieval using a geometric-mean metric at rank 10 (Geom@10), which we validate against the full retrieval-metric profile. We then evaluate shortlisted models on ChemQuests-derived tasks across five chunking strategies, seven chunk sizes, and multiple overlap settings. Embedding choice is associated with the largest observed differences in evidence retrieval, with retrieval-tuned E5, Beijing Academy of Artificial Intelligence General Embedding (BGE), and Nomic models among the strongest overall. Within the evaluated grid, medium-to-large chunks combined with fixed-token, recursive-token, or hierarchical-section chunking provide a practical starting point for the retrieval stage of chemistry-aware RAG. Low overlap was generally favored where overlap variation was evaluated.

Mahmoud Amiri, Thomas Bocklitz

Leibniz Institute of Photonic Technology, Albert-Einstein-Strasse 9, 07745 Jena, Germany · Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich Schiller University Jena, Helmholtzweg 4, 07743 Jena, Germany

Chemistry Chunk Trade-Off

Ask PDF

Aug 4, 2026cs.LG

Bi-semantic Chemical Embedder for Joint Representation Learning of SMILES and Natural Language

Transformer models have revolutionized natural language processing (NLP), and text-based molecular representations like SMILES have successfully extended these architectures to chemistry. However, domain-adaptive pre-training often causes models to overfit to chemical syntax, catastrophically forgetting their foundational semantic capabilities. To address this challenge, we introduce CheMatE, a chemistry-oriented embedding model that jointly captures molecular structure and domain-specific natural language within the same representation space. Built on a ModernBERT backbone, CheMatE learns bi-semantic representations through a two-stage training procedure: continued masked language modeling (MLM) followed by a Matryoshka contrastive learning stage via Multiple Negative Ranking Loss (MNRL). First, we train the model using MLM on a novel, large-scale corpus of SMILES-annotated, long-context scientific documents that were constructed and curated from FineWeb and ChemPile (comprising 10.4B and 11.5B tokens, respectively). Subsequently, the model undergoes contrastive learning using a synthetic dataset of SMILES-text pairs algorithmically derived from our original training corpus. This design exposes the model to SMILES-enriched scientific literature, enabling bi-semantic understanding. We evaluate CheMatE across a range of downstream tasks covering molecular property prediction and scientific language understanding. Our results demonstrate that coupling our custom-curated datasets with this sequential training strategy yields robust, highly transferable representations. By effectively unifying structural and contextual signals within a single text-based framework, CheMatE achieves competitive performance across both specialized chemistry models and general-purpose language model baselines.

David Ming Segura, Jeremy Goumaz, Joshua W. Sin +2

Laboratory of Artificial Chemical Intelligence (LIAC), EPFL, Lausanne, Switzerland · Process Chemistry & Catalysis, Synthetic Molecules Technical Development, F. Hoffmann-La Roche AG, Basel, Switzerland · National Centre of Competence in Research (NCCR) Catalysis, EPFL, Lausanne, Switzerland

Molecular Representation Learning Language Modeling Representation Learning Natural Language

Ask PDF

Jun 4, 2026cs.LG

MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry

Large language models (LLMs) have shown promise for molecular property prediction, but their ability to reason over chemical structures remains limited, as molecular representations such as SMILES differ substantially from the natural language on which LLMs are primarily trained. To bridge this semantic and chemical knowledge gap, we propose MolE-RAG, a training-free, molecule-centric retrieval-augmented generation framework for LLM-based molecular property prediction. MolE-RAG augments each prediction with three complementary sources of inference-time context: retrieved chemistry literature, molecule-specific information including compound synonyms, identifiers, functional group annotations, and physicochemical descriptors, and structurally similar molecules retrieved from the training set. We evaluate MolE-RAG across nine molecular property prediction tasks using proprietary, chemistry-specialized, and open-source LLMs. Across general-purpose LLMs, MolE-RAG improves ROC-AUC by up to 28 percentage points on classification tasks and reduces regression RMSE by up to 67% relative to a SMILES-only baseline. We further find that the utility of each context source varies across models and tasks, with different models benefiting most from textual retrieval, molecular context, or structural retrieval. These results suggest that molecule-centric retrieval can improve LLM-based molecular property prediction without model fine-tuning while providing a flexible framework for integrating heterogeneous chemical knowledge at inference time.

Joey Chan, Wonbin Kweon, Ashley Shin +4

University of Illinois Urbana-Champaign · University of California, San Diego

Molecular Property Prediction Chemical Structures Chemistry Hievi-Rag

Ask PDF