Performance Evaluation of Open-Source Large Language Models for Assisting Pathology Report Writing in Japanese

Masataka Kawai; Singo Sakashita; Shumpei Ishikawa; Shogo Watanabe; Anna Matsuoka; Mikio Sakurai; Yasuto Fujimoto; Yoshiyuki Takahara

cs.CLMar 12, 2026

Performance Evaluation of Open-Source Large Language Models for Assisting Pathology Report Writing in Japanese

Authors: Masataka Kawai, Singo Sakashita, Shumpei Ishikawa, Shogo Watanabe, Anna Matsuoka, Mikio Sakurai, Yasuto Fujimoto, Yoshiyuki Takahara,

Organizations: Department of Pathology and Clinical Laboratories, National Cancer Center Hospital East, Kashiwa, Japan · Department of Pathology, University of Yamanashi, Chuo, Japan · Division of Pathology, Exploratory Oncology Research & Clinical Trial Center, National Cancer Center, Kashiwa, Japan · Department of Preventive Medicine, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan ·

Abstract

The performance of large language models (LLMs) for supporting pathology report writing in Japanese remains unexplored. We evaluated seven open-source LLMs from three perspectives: (A) generation and information extraction of pathology diagnosis text following predefined formats, (B) correction of typographical errors in Japanese pathology reports, and (C) subjective evaluation of model-generated explanatory text by pathologists and clinicians. Thinking models and medical-specialized models showed advantages in structured reporting tasks that required reasoning and in typo correction. In contrast, preferences for explanatory outputs varied substantially across raters. Although the utility of LLMs differed by task, our findings suggest that open-source LLMs can be useful for assisting Japanese pathology report writing in limited but clinically relevant scenarios.

Figures & tables

Model	Quantization	Thinking	Med. knowledge	Size (GB)	Params (B)	pp512	tg128
Gemma 3-27b-it	Q4_0	No	No	16.04	27.01	384.66	29.90
MedGemma-27b-text-it	Q4_K_XL	No	Yes	15.66	27.01	337.32	27.31
SIP-jmed-llm-3-8x13b-AC-32k-instruct	Q8_0	No	Yes	72.40	73.16	421.82	25.97
Qwen3-Next-80B-A3B-Instruct	Q8_0	No	No	78.98	79.67	647.65	24.97
Qwen3-Next-80B-A3B-Thinking	Q8_0	Yes	No	78.98	79.67	650.06	24.94
gpt-oss-20b	MXFP4	Yes	No	11.27	20.91	2293.59	120.27

Table 1: Model information and llama-bench results.

Model	Macro F1	Macro Jaccard	Micro F1	Micro Jaccard	Avg time (s)	Max time (s)
Gemma 3-27b-it	1.000	1.000	1.000	1.000	7.92	9.04
MedGemma-27b-text-it	1.000	1.000	1.000	1.000	8.51	9.76
SIP-jmed-llm-3-8x13b-AC-32k-instruct	0.989	0.980	0.987	0.974	8.20	16.28
Qwen3-Next-80B-A3B-Instruct	0.999	0.998	0.999	0.998	11.13	11.98
Qwen3-Next-80B-A3B-Thinking	1.000	1.000	1.000	1.000	119.10	170.81
gpt-oss-20b	0.997	0.994	0.997	0.994	10.94	24.51

Table 2: Benchmark A1: JSON to institutional template.

Model	Macro F1	Macro Jaccard	Micro F1	Micro Jaccard	pT	NG	HG	All correct
Gemma 3-27b-it	0.987	0.974	0.987	0.974	0.18	0.66	0.58	0.47
MedGemma-27b-text-it	0.983	0.966	0.983	0.966	0.06	0.41	0.51	0.36
SIP-jmed-llm-3-8x13b-AC-32k-instruct	0.428	0.277	0.430	0.274	0.00	0.00	0.00	0.01
Qwen3-Next-80B-A3B-Instruct	0.987	0.975	0.987	0.975	0.18	0.66	0.59	0.37
Qwen3-Next-80B-A3B-Thinking	1.000	1.000	1.000	1.000	1.00	1.00	1.00	1.00
gpt-oss-20b	0.991	0.983	0.991	0.982	0.96	0.97	0.98	0.99

Table 3: Benchmark A2: JSON to institutional template with pT determination and score calculation.

Model	Macro F1	Macro Jaccard	Micro F1	Micro Jaccard	Avg time (s)	Max time (s)
Gemma 3-27b-it	0.992	0.985	0.992	0.985	4.44	5.90
MedGemma-27b-text-it	0.993	0.986	0.993	0.986	4.65	7.07
SIP-jmed-llm-3-8x13b-AC-32k-instruct	0.418	0.301	0.390	0.242	9.06	10.34
Qwen3-Next-80B-A3B-Instruct	0.996	0.993	0.996	0.993	6.41	6.79
Qwen3-Next-80B-A3B-Thinking *	1.000	1.000	1.000	1.000	192.14	300.00
gpt-oss-20b	0.971	0.944	0.971	0.943	13.62	28.95

Table 4: Benchmark A3: JSON to Kiyaku template.

Model	Match ratio	Avg time (s)	Max time (s)
Gemma 3-27b-it	1.000	9.43	10.85
MedGemma-27b-text-it	1.000	10.27	12.19
SIP-jmed-llm-3-8x13b-AC-32k-instruct	0.532	9.64	11.42
Qwen3-Next-80B-A3B-Instruct	1.000	12.42	12.79
Qwen3-Next-80B-A3B-Thinking	0.955	178.64	456.17
gpt-oss-20b	0.984	11.27	18.73

Table 5: Benchmark A4: extraction of JSON items from a formatted report.

Model	Macro P	Macro R	Macro F1	Micro P	Micro R	Micro F1	LD
Gemma 3-27b-it	0.763	0.557	0.619	0.759	0.538	0.630	1
MedGemma-27b-text-it	0.773	0.574	0.629	0.827	0.578	0.680	0
SIP-jmed-llm-3-8x13b-AC-32k-instruct	0.723	0.686	0.640	0.713	0.713	0.713	4
Qwen3-Next-80B-A3B-Instruct	0.738	0.673	0.697	0.743	0.692	0.717	0
Qwen3-Next-80B-A3B-Thinking	0.633	0.684	0.651	0.672	0.735	0.702	0
gpt-oss-20b	0.544	0.501	0.517	0.559	0.491	0.523	1

Table 6: Benchmark B: typo correction in 31 pathology reports.P: presicion, R: Recall.

Figure 1: Distribution of explanatory-text ratings by model among five pathologists (a) and three clinicians (b).

Figure 2: Model-level inter-rater reliability for explanatory-text evaluation. Panels show ICC(2,1) and ICC(2, k) of pathologists (a and b). Those of clinicians (c and d). 95% confidence intervals are depicted with bars.

Medical Report Generation

Medical report generation turns medical images, clinical records, or conversations into structured, clinically useful reports.

See all 82 papers

How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathology
Jun 10, 2026Kian R. Weihrauch, Thomas A. Buckley, William Lotter +1Multimodal Large Language Models Histopathology Image Classification
Massachusetts Institute of Technology, Cambridge, MA · Harvard Medical School, Boston, MA · Dana-Farber Cancer Institute, Boston, MA
PathLang: A Language-Centered Benchmark for Vision-Language Models in Computational Pathology
Oct 8, 2026Fanqi Cheng, Kuo Gong, Shangke Liu +15VLM Evaluation Medical VLMs
New York University, New York, NY, USA · Columbia University, New York, NY, USA · Weill Cornell Medicine, New York, NY, USA +9
Prompting language influences diagnostic reasoning and accuracy of large language models
May 18, 2026Adrien Bazoge, Josselin Corvellec, Sofiane Djillali Sid-Ahmed +1Multilingual Language Model Evaluation LLM Prompting
Data Clinic, Nantes University Hospital, France

Jun 10, 2026J/K move · Enter open · S save

How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathology

Kian R. Weihrauch, Thomas A. Buckley, William Lotter, Arjun K. Manrai

Massachusetts Institute of Technology, Cambridge, MA · Harvard Medical School, Boston, MA · Dana-Farber Cancer Institute, Boston, MA

Multimodal Large Language Models Histopathology Image Classification Medical Image Benchmarks Whole-Slide Image Analysis

Open paper AskPDF arXiv

General-purpose large language models (LLMs) are routinely used as baselines when evaluating specialized pathology models on whole-slide images (WSIs). Because WSIs exceed contemporary model context limits, LLM baselines routinely use small, high-magnification patches processed independently via majority voting, without systematic evaluation of seemingly inconsequential design choices such as patch size, patch count, and magnification. Generalist LLMs have consistently underperformed specialized systems, reinforcing the perception that domain-specific training or architectural adaptation is necessary for pathology tasks involving WSIs. Here, we conduct a systematic factorial analysis of four input design factors: inference mode, patch size, magnification, and patch count. We demonstrate that prior studies have overstated the gap between specialized models and general-purpose LLMs by choosing non-optimized input configurations. On the MultiPathQA benchmark, switching to a single balanced configuration (large patches at lower magnification, processed jointly) raises GPT-5 from 15.1% to 39.5% on cancer-type classification (TCGA) and from 38.1% to 62.9% on organ classification (GTEx). Per-task optimization yields further gains up to 43.9% (TCGA) and 71.6% (GTEx). The same configuration generalizes to two other models and to a fully held-out CPTAC cohort, where it improves Gemini 3 Flash by 23.4 percentage points without any task-specific tuning.

Abstract

Medical Report Generation

Explore similar work

How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathology

PathLang: A Language-Centered Benchmark for Vision-Language Models in Computational Pathology

Prompting language influences diagnostic reasoning and accuracy of large language models