PET-F2I: A Comprehensive Benchmark and Parameter-Efficient Fine-Tuning of LLMs for PET/CT Report Impression Generation

Yuchen Liu; Wenbo Zhang; Liling Peng; Yichi Zhang; Yu Fu; Xin Guo; Chao Qu; Yuan Qi

cs.CVMar 11, 2026

PET-F2I: A Comprehensive Benchmark and Parameter-Efficient Fine-Tuning of LLMs for PET/CT Report Impression Generation

Authors: Yuchen Liu, Wenbo Zhang, Liling Peng, Yichi Zhang, Yu Fu, Xin Guo, Chao Qu, Yuan Qi,

Organizations: Fudan University · Shanghai Academy of Artificial Intelligence for Science · Shanghai Universal Medical Imaging Diagnostic Center · Lanzhou University

Abstract

PET/CT imaging is pivotal in oncology and nuclear medicine, yet summarizing complex findings into precise diagnostic impressions is labor-intensive. While LLMs have shown promise in medical text generation, their capability in the highly specialized domain of PET/CT remains underexplored. We introduce PET-F2I-41K (PET Findings-to-Impression Benchmark), a large-scale benchmark for PET/CT impression generation using LLMs, constructed from over 41k real-world reports. Using PET-F2I-41K, we conduct a comprehensive evaluation of 27 models across proprietary frontier LLMs, open-source generalist models, and medical-domain LLMs, and we develop a domain-adapted 7B model (PET-F2I-7B) fine-tuned from Qwen2.5-7B-Instruct via LoRA. Beyond standard NLG metrics (e.g., BLEU-4, ROUGE-L, BERTScore), we propose three clinically grounded metrics - Entity Coverage Rate (ECR), Uncovered Entity Rate (UER), and Factual Consistency Rate (FCR) - to assess diagnostic completeness and factual reliability. Experiments reveal that neither frontier nor medical-domain LLMs perform adequately in zero-shot settings. In contrast, PET-F2I-7B achieves substantial gains (e.g., 0.708 BLEU-4) and a 3.0x improvement in entity coverage over the strongest baseline, while offering advantages in cost, latency, and privacy. Beyond this modeling contribution, PET-F2I-41K establishes a standardized evaluation framework to accelerate the development of reliable and clinically deployable reporting systems for PET/CT.

Figures & tables

Figure 1: Overview of the PET-F2I-41K framework. The pipeline encompasses clinical task formalization, comprehensive benchmarking of 27 LLMs, and multi-dimensional evaluation, demonstrating the absolute superiority of our domain-adapted PET-F2I-7B (+138% ECR, -75% UER).

Figure 2: (a) Tracer distribution in PET-F2I-41K (41,191 reports). (b, c) 27 models ranked by BLEU-4 and ECR; PET-F2I-7B (

\star

) consistently outperforms the performance of frontier proprietary, large-scale open-source, and specialized medical LLMs.

Category	Model	BLEU-4 $\uparrow$	ROUGE-L $\uparrow$	METEOR $\uparrow$	BERTScore $\uparrow$	SBERT $\uparrow$	ECR $\uparrow$	UER $\downarrow$	FCR $\uparrow$
Proprietary	Claude Opus 4.5 [ 1 ]	0.2532	0.3520	0.5692	0.7590	0.6433	0.5268	0.5417	0.9814
	Claude Sonnet 4.5 [ 1 ]	0.2374	0.3587	0.5369	0.7562	0.6286	0.5222	0.5434	0.9734
	Gemini 2.5 Pro [ 7 ]	0.2112	0.4193	0.5127	0.7643	0.6597	0.4379	0.5152	0.9618
	Gemini 2.0 Flash [ 11 ]	0.1897	0.4792	0.4653	0.7516	0.6401	0.4153	0.4947	0.9628
	GPT-4o [ 21 ]	0.1876	0.3248	0.4577	0.7344	0.5990	0.4346	0.5883	0.9756
	GPT-5.1 [ 21 ]	0.1681	0.2963	0.4768	0.7327	0.5888	0.4622	0.6250	0.9898

Table 1: Quantitative evaluation of 27 LLMs on the PET-F2I-41K benchmark. Bold and underline denote the best overall and best-in-category scores, respectively. The domain-adapted PET-F2I-7B demonstrates absolute superiority across all standard NLG and proposed clinical metrics, particularly maximizing exact entity coverage (ECR).

Figure 3: Score distributions across model categories (

N=500

). Beyond superior averages, PET-F2I-7B demonstrates exceptional clinical stability. Crucially, PET-F2I-7B’s lowest quartile for exact entity coverage (ECR) strictly dominates the highest quartiles of all zero-shot baselines, underscoring its superior reliability for real-world deployment.

Figure 4: Correlation between NLG and clinical metrics. Despite strong macro-level correlations (e.g., ECR vs. BERTScore-F1:

r=0.889

), substantial intra-distribution variance reveals that high lexical overlap does not guarantee clinical accuracy. Models with identical NLG scores frequently exhibit drastically different entity omission rates, proving traditional metrics are insufficient proxies for diagnostic safety and highlighting the necessity of the PET-F2I framework.

Figure 5: Independence of clinical metrics and divergence from NLG scores. (a) Sample-level analysis demonstrates that entity coverage (ECR) and formatting compliance (FCR) are fundamentally orthogonal ( $r=0.075$ ). (b) The correlation matrix reveals that standard lexical metrics exhibit near-zero correlation with clinical factuality (e.g., BLEU-4 vs. FCR: $r=0.28$ ), highlighting their inadequacy for safety evaluation.

Figure 6: Robustness across radiopharmaceutical tracers. Heatmaps of clinical metrics (rows: models, columns: tracers) reveal that PET-F2I-7B generalizes exceptionally well, maintaining high diagnostic accuracy even on minority tracer profiles.

Radiology Report Generation

Radiology report generation uses medical images and clinical context to produce structured, clinically accurate findings and impressions.

See all 159 papers

Error Detection for PET/CT Radiology Reports: Domain-Specific vs Large Language Models
Aug 30, 2026Hermione Warr, Harry Anthony, Lilli J Freischem +3LLM Evaluation PET/CT Imaging
Department of Engineering Science, University of Oxford, UK · Department of Physics, University of Oxford, UK · Department of Oncology, University of Oxford, UK +1
Region-Grounded Report Generation for 3D Medical Imaging: A Fine-Grained Dataset and Graph-Enhanced Framework
Apr 20, 2026Cong Huy Nguyen, Son Dinh Nguyen, Guanlin Li +8Radiology Report Generation Medical Report Generation
AI4LIFE, Hanoi University of Science and Technology, Vietnam · SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris, France · Military Central Hospital, Vietnam
DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation
Mar 18, 2026Yuhe Tian, Kun Zhang, Haoran Ma +4CT Report Generation Radiology Report Generation
Department of Electronic Engineering and Information Science, School of Information Science and Technology, University of Science and Technology of China (USTC), Hefei, Anhui 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China · Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China +2

Aug 30, 2026J/K move · Enter open · S save

Error Detection for PET/CT Radiology Reports: Domain-Specific vs Large Language Models

Hermione Warr, Harry Anthony, Lilli J Freischem, Yasin Ibrahim+2

Department of Engineering Science, University of Oxford, UK · Department of Physics, University of Oxford, UK · Department of Oncology, University of Oxford, UK+1

LLM Evaluation PET/CT Imaging Clinical Language Model Evaluation Radiology Report Evaluation

Open paper AskPDF arXiv

Errors in radiology reports can adversely affect patient treatment, yet automated report quality assurance remains challenging because errors are often subtle and require domain expertise to detect. Although large language models (LLMs) have recently been proposed for radiology report verification, their ability to detect clinically meaningful errors beyond chest X-ray datasets remains under-explored. To this end, we present the first systematic evaluation of language models for PET/CT report error detection, comparing compact domain-specific models with SOTA open-weight LLMs. We collected 30,633 oncology FDG PET/CT reports from 23 radiologists over 10 years. We trained domain-specific BERT models to detect clinically motivated synthetic reporting errors and evaluated alongside zero-/few-shot Qwen3-32B, Gemma-3-27B and Llama-3.3-70B on a held-out benchmark of 11,500 reports. A 15M-parameter model achieved 94.4% balanced accuracy with a 5.8% false-positive rate, compared with 84.0% for the strongest prompted LLM. Task-specific adaptation of Llama-3.3-70B closed this performance gap (94.4%) but retained substantially greater computational requirements. Our results suggest that domain-specific training matters more than model scale for PET/CT report error detection, supporting compact models as an accurate and computationally efficient approach to automated radiology report quality assurance.

Abstract

Radiology Report Generation

Explore similar work

Error Detection for PET/CT Radiology Reports: Domain-Specific vs Large Language Models

Region-Grounded Report Generation for 3D Medical Imaging: A Fine-Grained Dataset and Graph-Enhanced Framework

DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation