A Joint Neural Baseline for Concept, Assertion, and Relation Extraction from Clinical Text

Fei Cheng; Ribeka Tanaka; Sadao Kurohashi

cs.CLMar 8, 2026

A Joint Neural Baseline for Concept, Assertion, and Relation Extraction from Clinical Text

Authors: Fei Cheng, Ribeka Tanaka, Sadao Kurohashi

Organizations: Kyoto University · Tokyo University of Technology · National Institute of Informatics

Abstract

Clinical information extraction (e.g., 2010 i2b2/VA challenge) usually presents tasks of concept recognition, assertion classification, and relation extraction. Jointly modeling the multi-stage tasks in the clinical domain is an underexplored topic. The existing independent task setting (reference inputs given in each stage) makes the joint models not directly comparable to the existing pipeline work. To address these issues, we define a joint task setting and propose a novel end-to-end system to jointly optimize three-stage tasks. We empirically investigate the joint evaluation of our proposal and the pipeline baseline with various embedding techniques: word, contextual, and in-domain contextual embeddings. The proposed joint system substantially outperforms the pipeline baseline by +0.3, +1.4, +3.1 for the concept, assertion, and relation F1. This work bridges joint approaches and clinical information extraction. The proposed approach could serve as a strong joint baseline for future research. The code is publicly available.

Figures & tables

Figure 1: The overview of the joint concept, assertion, and relation extraction model.

	Training	Test
#Doc	170	256
#Concept	16,399	31,048
#Assertion	7,058	12,568
#Relation	3,106	6,279

Table 1: Statistics of the public i2b2/VA 2010.

	In-domain pretraining data
ClinicalBERT	+ 0.3M MIMIC
BlueBERT	+ 5M PubMed + 0.2M MIMIC

Table 2: Statistics of the pretraining settings of the in-domain clinical BERTs.

	Baseline			Joint Model
Encoders	Concept	Assertion	Relation	Concept	Assertion	Relation
Glove+LSTM	82.7	74.4	36.8	83.0	75.2	40.5
BERT	86.3	81.0	49.9	86.5	82.1	53.2
ClinicalBERT	87.5	82.6	51.7	87.6	83.3	55.5
BlueBERT	89.2	84.3	56.1	89.5	85.7	59.2

Table 3: Joint evaluation results of the joint model and pipeline baseline.

Models	Concept	Assertion	Relation
i2b2/VA Best	85.2	93.6	73.7
Alsentzer 2019b	87.8	-	-
Peng 2019	-	-	76.4
Lee 2020	86.7	-	-
Baseline ${}_{\textrm{BlueBERT}}$	89.2	94.3	$71.3^{*}$

Table 4: Independent evaluation of comparing the baseline (BlueBERT) to SOTA systems. The reported results of ‘i2b2/VA Best’ are on the original dataset (394 training and 477 test reports) in the 2010 challenge. ‘*’ denotes the noisy setting on relation categories in § 5.1

Clinical Information Extraction

Clinical Information Extraction turns clinical notes, reports, and patient narratives into structured, searchable facts and relations.

See all 106 papers

MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction
Apr 5, 2026Zhichao Yang, Gregory D. Lyng, Sanjit Singh Batra +1Clinical Information Extraction Clinical Language Model Evaluation
Optum AI
Configurable Clinical Information Extraction with Agentic RAG: What Works, What Breaks, and Why
Jun 17, 2026Osman Alperen Çinar-Koraş, Marie Bauer, Sameh Khattab +7Clinical Information Extraction Agentic RAG
Institute for Artificial Intelligence in Medicine (IKIM), University Medicine Essen, Essen, Germany · Faculty of Computer Science, University of Duisburg-Essen, Essen, Germany · Department of Cardiology and Vascular Medicine, University Hospital Essen, Essen, Germany +3
ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV
May 11, 2026Alex StinardRAG Evaluation Clinical QA
Department of Clinical Sciences, College of Medicine · University of Central Florida, Orlando, FL 32816

Apr 5, 2026J/K move · Enter open · S save

MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction

Zhichao Yang, Gregory D. Lyng, Sanjit Singh Batra, Robert E. Tillman

Optum AI

Clinical Information Extraction Clinical Language Model Evaluation LLM Grounding Implicit Reasoning in Language Models

Open paper AskPDF arXiv

Medical concept extraction from electronic health records underpins many downstream applications, yet remains challenging because medically meaningful concepts are frequently implied rather than explicitly stated in medical narratives. Existing benchmarks with human-annotated evidence spans underscore the importance of grounding extracted concepts in medical text. However, they predominantly focus on explicitly stated concepts instead of implicit concepts. We present MedicalBench, a benchmark for medical concept extraction with evidence grounding that evaluates implicit medical reasoning. MedicalBench formulates medical concept extraction as a verification task over medical note-concept pairs, coupled with sentence-level evidence identification. Built from MIMIC-IV discharge summaries and human-verified ICD-10 codes, the dataset is curated through a multi-stage large language model (LLM) triage pipeline followed by medical annotation and expert review. It deliberately includes implicit positives, semantically confusable negatives, and cases where LLM judgments disagree with medical expert assessments. We define two complementary evaluation tasks: (1) medical concept extraction and (2) sentence-level evidence retrieval, enabling assessment of both correctness and interpretability. Benchmarking state-of-the-art LLMs reveals that performance remains modest, highlighting the difficulty of extracting implicitly expressed concepts. We further show that performance is largely invariant to note length, indicating that MedicalBench isolates reasoning difficulty rather than superficial confounders. MedicalBench provides the first systematic benchmark for implicit, evidence-grounded medical concept extraction, offering a foundation for developing medical language models that can both identify medically relevant concepts and justify their predictions in a transparent and medically faithful manner.

Abstract

Clinical Information Extraction

Explore similar work

MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction

Configurable Clinical Information Extraction with Agentic RAG: What Works, What Breaks, and Why

ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV