cs.AINov 17, 2026

Fine-Grained Japanese Clinical Named Entity Recognition: A Controlled Comparison of BERT and Open-Weight Large Language Models

Authors: Feng Han, Ayako Yagahara, Kenji Hirata, Atsushi Yoshida, Kohsuke Kudo

Organizations: Division of AI Support for Medical Research, Research Center for Cooperative Projects, Faculty of Medicine, Hokkaido University, Sapporo, Japan · Faculty of Health Sciences, Hokkaido University of Science, Sapporo, Japan · Faculty of Health Sciences, Hokkaido University, Sapporo, Japan · Department of Diagnostic Imaging, Faculty of Medicine, Hokkaido University, Sapporo, Japan; Present address: Department of Nuclear Medicine, Faculty of Medicine, Kanazawa University, Kanazawa, Japan · 

Abstract

Background: Generative large language models (LLMs) offer an alternative to encoder-based clinical named entity recognition (NER), but their performance on fine-grained Japanese extraction tasks and sensitivity to generation settings remain uncertain. Methods: We compared two fine-tuned BERT-CRF baselines with nine open-weight LLMs on iCorpus, which defines 113 entity categories. The evaluation used 433 sentences from 36 test documents; 1,672 sentences were used for training. LLMs were evaluated using zero-shot prompting with and without JSON Schema constraints and three-seed QLoRA fine-tuning. Fine-tuned models were assessed under different output-token limits, including 900 and larger model-specific limits of 2,600 or 4,096 tokens. Performance was measured using sentence-level label-surface matching with relaxed and exact-surface F1. Results: The highest zero-shot relaxed F1 was 0.337, compared with 0.836-0.841 for the BERT baselines. At the larger output limits, all eight fine-tuned LLMs with at least 13B total parameters exceeded both BERT baselines (relaxed F1, 0.882-0.914; exact-surface F1, 0.840-0.880). Across three-seed means, increasing the output limit improved relaxed F1 by 0.052-0.085, alongside higher recall and fewer length-limited outputs. JSON Schema constraints changed relaxed F1 by -0.014 to +0.008 among seven models outside the two reasoning-oriented configurations, while medical-general model pairs showed no consistent advantage for medical specialization. Conclusions: On this historical Japanese case-report benchmark, task-specific fine-tuning combined with adequate generation capacity was associated with stronger LLM extraction performance. Output-token limits materially affected comparisons with BERT, whereas enforcing output structure alone did not overcome low zero-shot accuracy.

Explore similar work

CardsList
  1. Assessment of Generative Named Entity Recognition in the Era of Large Language Models

    Jan 25, 2026Qi Zhan, Yile Wang, Hui HuangNamed Entity RecognitionLLM Information Extraction

  2. Beyond the Basics: Leveraging Large Language Model for Fine-Grained Medical Entity Recognition

    Apr 19, 2026Nwe Ni Win, Jim Basilakis, Steven Thomas +6Named Entity RecognitionHealthcare