BioBigBird: A Sparse Attention Model for Long-Range Dependency Processing in Biomedical Text

Roshan Balaji; Pavan Kumar S; Vasudev Gupta; Sreejith N; Keerthana Sridhar; Nirav Bhatt

cs.CLOct 8, 2026

BioBigBird: A Sparse Attention Model for Long-Range Dependency Processing in Biomedical Text

Authors: Roshan Balaji, Pavan Kumar S, Vasudev Gupta, Sreejith N, Keerthana Sridhar, Nirav Bhatt

Organizations: Wadhwani School of Data Science and AI, Indian Institute of Technology Madras, Chennai, India

Abstract

While domain-specific Large Language Models (LLMs) have encoded vast biomedical knowledge, their limited context windows often hinder a deep understanding of nuanced relationships within and across texts. To address this limitation, we introduce BioBigBird, a bidirectional language model pre-trained on extensive biomedical literature and clinical data, specifically designed to handle long-range dependencies. BioBigBird leverages a sparse attention mechanism to process sequences up to 4096 tokens, and its training incorporates a multi-stage process to mitigate noise from the large-scale pre-training corpus. We further enhance its performance by employing a multi-task learning (MTL) framework that jointly optimizes for Named Entity Recognition and Relation Extraction. Comprehensive evaluations on the BLURB benchmark reveal that our MTL-enhanced BioBigBird achieves highly competitive results against state-of-the-art models. Our work contributes an effective methodology for developing powerful, long-context language models for specialized domains, demonstrating the value of extended sequence processing for complex text analysis. Our models are publicly available at https://huggingface.co/collections/bisectgroup/biobigbird.

Figures & tables

Figure 1: Attention mechanism used in BigBird. The white colour indicates the absence of attention. Random, global, and sliding connections are combined to compute the attention scores for longer sequences efficiently.

Figure 2: The comparison of BERT and BigBird attention mechanisms.

Figure 3: Overview of our approach BioBigBird: We collect documents related to the biomedical corpora and use a three-stage approach to pre-train the language model using masked language modelling.

Dataset	Disk Space	# Samples	# Tokens
PubMed Articles	80 GB	2.3 M	10 B
PubMed Abstracts	31 GB	42 M	10 B
MIMIC-III	3.5 GB	2 M	2 B

Table 1: Statistics of the pre-training datasets.

Dataset	Disk Space	# Samples
Raw Articles	300 GB	4.7 M
Pre-processed Articles	80 GB	2.3 M

Table 2: Statistics of the PubMed articles before and after pre-processing.

Figure 4: Overview of our multi-task architecture: we fine-tune the BioBigBird model using this method.

Dataset	Entity Type
NCBI Disease ( Doğan et al., 2014 )	Disease
BC5CDR ( Li et al., 2016 )	Disease
BC5CDR ( Li et al., 2016 )	Drug/Chem.
BC4CHEMD ( Krallinger et al., 2014 )	Drug/Chem.
BC2GM ( Smith et al., 2008 )	Gene
JNLPBA ( Collier and Kim, 2004 )	Biological Molecules

Table 3: Biomedical named entity recognition datasets.

Dataset	Entity Type
GAD ( Bravo et al., 2015 )	Gene-disease
DDI ( Herrero-Zazo et al., 2013 )	Drug-drug
ChemProt ( Krallinger et al., 2017 )	Protein-chemical

Table 4: Biomedical relation extraction datasets.

	PubMedBERT	BioLinkBERT	BioLinkBERT	BioBigBird
	( Gu et al., 2021 )	( Yasunaga et al., 2022 )	( Yasunaga et al., 2022 )	Stage 3
	Base (110 M)	Base (110 M)	Large (340 M)	(113 M)
BC5-chem ( Li et al., 2016 )	93.33	93.75	94.04	94.06
BC5-disease ( Li et al., 2016 )	85.62	86.10	86.39	85.18
NCBI-disease ( Doğan et al., 2014 )	87.82	88.18	88.76	85.59
BC2GM ( Smith et al., 2008 )	84.52	84.90	85.18	84.37

Table 5: Comparison of performance of pre-trained models on the BLURB benchmark.

	PubMedBERT	BioLinkBERT	BioLinkBERT	BERN-2	BioBigBird	BioBigBird
	( Gu et al., 2021 )	( Yasunaga et al., 2022 )	( Yasunaga et al., 2022 )	( Sung et al., 2022 )	Stage 3	MT-Stage 3
	Base (110 M)	Base (110 M)	Large (340 M)	(365 M)	(113 M)	(120 M)
BC5-chem ( Li et al., 2016 )	93.33	93.75	94.04	N/A	94.06	95.07
BC5-disease ( Li et al., 2016 )	85.62	86.10	86.39	N/A	85.18	90.36
NCBI-disease ( Doğan et al., 2014 )	87.82	88.18	88.76	88.6	85.59	93.67
BC2GM ( Smith et al., 2008 )	84.52	84.90	85.18	83.7	84.37	91.85

Table 6: Comparison of performance of multi-task BioBigBird for NER and RE tasks.

Explore similar work

Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?
Jun 13, 2026Jakob Mraz, Tomaž Curk, Blaž ZupanRelation Extraction Few-Shot Learning
University of Ljubljana, Ljubljana, Slovenia · Baylor College of Medicine, Houston, Texas, USA
BioTool: A Comprehensive Tool-Calling Dataset for Enhancing Biomedical Capabilities of Large Language Models
May 7, 2026Xin Gao, Ruiyi Zhang, Meixi Du +2LLM Fine-Tuning LLM Tool Use
UC San Diego · MBZUAI
ANCHOR-RE: An Agentic Neuro-Symbolic Framework for Grounded Biomedical Relation Extraction
Aug 4, 2026Shufan Ming, Yikun Han, Gibong Hong +2Relation Extraction LLM Grounding
School of Information Sciences, University of Illinois Urbana-Champaign, 501 E Daniel St., Champaign, 61820, IL, USA · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 516 Delaware St SE, Minneapolis, 55455, MN, USA

Jun 13, 2026, cs.CLJ/K move · Enter open · S save

Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?

Jakob Mraz, Tomaž Curk, Blaž Zupan

University of Ljubljana, Ljubljana, Slovenia · Baylor College of Medicine, Houston, Texas, USA

Relation Extraction Few-Shot Learning Few-Shot Prompting LLM Information Extraction

Open paper AskPDF arXiv

Biomedical relation extraction (BioRE) is a key step in transforming biomedical literature into structured knowledge. However, most existing approaches rely on supervised models trained on costly annotated datasets, limiting their scalability and adaptability across relation types and domains. We investigate few-shot BioRE using prompt-based learning with large language models (LLMs) and compare two task formulations: pairwise classification, which predicts relations for individual entity pairs, and joint generation, which extracts multiple relations in a single model call. Experiments on the BioREDirect dataset reveal a clear precision-recall trade-off. Pairwise classification achieves higher recall, whereas joint generation is more precise and computationally efficient. The best-performing model achieves a micro-F1 score of 0.44, substantially outperforming previous few-shot results (0.34) while remaining below the supervised baseline (0.56). Much of this gap is attributable to a single ambiguously defined relation type. When evaluated using macro-F1, which better captures performance across relation types in an imbalanced setting, prompt-based approaches outperform the supervised baseline (0.45 vs. 0.38), particularly on rare relation types. These findings highlight the potential of LLMs for BioRE in low-resource settings and underscore the importance of well-defined relation schemas.

Abstract

Explore similar work

Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?

BioTool: A Comprehensive Tool-Calling Dataset for Enhancing Biomedical Capabilities of Large Language Models

ANCHOR-RE: An Agentic Neuro-Symbolic Framework for Grounded Biomedical Relation Extraction