A Specialized Large Multimodal Model for Interpreting PET/CT in Head and Neck Cancer
Authors: Haengbok Chung, SunGyu Kim, Joo hyun Lee, Sangjin Bae, Min Jeong Cho, Minseok Suh, Jae Sung Lee
Organizations: Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, Republic of Korea. · Department of Nuclear Medicine, Seoul National University College of Medicine, Seoul, Republic of Korea. · Department of Nuclear Medicine, Seoul National University Hospital, Seoul, Republic of Korea. · Interdisciplinary Program in Bioengineering, College of Engineering, Seoul National University Graduate School, Seoul, Republic of Korea. · Integrated Major in Innovative Medical Science, Seoul National University Graduate School, Seoul, Republic of Korea. · Department of Nuclear Medicine, Seoul National University Bundang Hospital, Seoul National University College of Medicine, Seongnam, Republic of Korea. · Institute of Radiation Medicine, Medical Research Center, Seoul National University College of Medicine, Seoul, Republic of Korea. · Brightonix Imaging Inc., Seoul, Republic of Korea.
Background: Diagnosing head and neck cancer using PET/CT is clinically challenging and time-consuming due to the anatomical complexity of the region, motivating computer-aided diagnosis (CAD). Generalist Large Multimodal Models (LMMs) remain limited in medical contexts by insufficient domain-specific knowledge, privacy and security concerns, and verbosity, motivating specialized standalone LMMs. Purpose: We evaluated the feasibility of a specialized LMM for automated PET/CT interpretation in head and neck cancer using a large-scale multi-institutional PET/CT dataset, a tailored training curriculum, and autoregressive training. Methods: LLaVA-NeXT was fine-tuned using a two-level curriculum with image-conversation pairs curated by two radiologists from public data. The dataset included clinically important annotations such as primary tumor presence and metastatic lymph node location. Level 1 used 28,000 image-conversation pairs to learn basic information, including modality type and hypermetabolism. Level 2 used 12,975 pairs to learn primary tumor presence and the existence and anatomical location of cervical lymph node metastases. External validation included four institutions with diverse imaging devices. Results: The specialized LMM substantially outperformed ChatGPT and LLaVA-NeXT. In Level-2 external validation, ROUGE-L, ROUGE-S, Cosine Similarity, Precision, Recall, and F1 were 0.8751, 0.8794, 0.8324, 0.8794, 0.8711, and 0.8751, while generalist models consistently scored below 0.1. Primary tumor classification accuracy was 83.14 +/- 1.15% internally and 69.03 +/- 0.81% externally. For lymph node localization, the corresponding scores were 0.6389, 0.6257, 0.5287, 0.5782, 0.6371, and 0.6648. Conclusion: Specialized LMMs show promising results for fast, accurate PET/CT-based diagnostic support and medical education, highlighting their potential for clinical translation.
Figures & tables
Characteristic
Type
Training (c = 28,000)
Internal Validation (c = 2,000)
External Validation (c = 1,800)
P Value
Modality
PET
11,011 (39.3)
799 (40.0)
890 (49.4)
<0.001
PET/CT
9,125 (32.6)
655 (32.8)
540 (30.0)
CT
7,864 (28.1)
546 (27.3)
370 (20.5)
View
Axial
12,799 (45.7)
936 (46.8)
613 (34.0)
<0.001
Sagittal
7,513 (26.8)
527 (26.4)
497 (27.6)
Coronal
7,688 (27.5)
537 (26.8)
690 (38.3)
Table 1: Characteristics of the Datasets for Level 1 Training, Internal Validation, and External Validation.
Characteristic
Type
Training (p = 1,333)
Internal Validation (p = 860)
External Validation (p = 169)
P Value
Age
-
65 (55-74) [11-95]
65 (56-75) [17-95]
63 (57-70) [34-90]
0.241
Gender
Male
543 (40.7)
340 (39.5)
124 (73.3)
<0.001
Female
408 (30.6)
240 (27.9)
45 (26.6)
Unknown
382 (28.7)
280 (32.6)
0 (0)
Anatomic Site
Partial Body
87 (6.5)
76 (8.8)
137 (81.0)
<0.001
Whole Body
1229 (92.2)
774 (90.0)
32 (18.9)
Table 2: Patient Characteristics of the Level 1 Training, Internal Validation, and External Validation Cohorts.
Characteristic
Type
Training (c = 12,975)
Internal Validation (c = 1,441)
External Validation (c = 1,000)
P Value
Modality
PET
6,515 (50.1)
713 (49.4)
495 (55.8)
<0.001
PET/CT
4,768 (36.7)
538 (37.3)
378 (42.6)
CT
1,692 (13.0)
190 (13.1)
13 (1.4)
View
Axial
2,556 (19.6)
306 (21.2)
257 (25.7)
<0.001
Sagittal
5,268 (40.5)
572 (39.6)
310 (31.0)
Coronal
5,151 (39.7)
563 (39.0)
433 (43.3)
Table 3: Characteristics of the Datasets for Level 2 Training, Internal Validation, and External Validation.
Characteristic
Type
Training (p = 55)
Internal Validation (p = 55)
External Validation (p = 138)
P Value
Age
-
62 (56-66) [45-81]
62 (56-66) [45-81]
63 (57-70) [34-90]
0.399
Gender
Male
43 (78.1)
43 (78.1)
101 (73.1)
0.662
Female
12 (21.8)
12 (21.8)
37 (26.8)
Anatomic Site
Partial Body
55 (100)
55 (100)
111 (80.4)
<0.001
Whole Body
0 (0)
0 (0)
27 (19.5)
Table 4: Patient Characteristics of the Level 2 Training, Internal Validation, and External Validation Cohorts.
Institution
ROUGE-L
ROUGE-S
Cosine Similarity
Precision
Recall
F1
Internal
0.6239 (± 0.0301)
0.5644 (± 0.0472)
0.5369 (± 0.0785)
0.6168 (± 0.0664)
0.6647 (± 0.0036)
0.6323 (± 0.0324)
External
0.6389 (± 0.0239)
0.6257 (± 0.0143)
0.5287 (± 0.0129)
0.5782 (± 0.0824)
0.6371 (± 0.0650)
0.6648 (± 0.0136)
Table 5: When analyzing the specific task of identifying metastatic lymph node stations—information encoded in the final five tokens of the model's output—we observed quantitatively lower scores on both validation sets compared to the holistic performance metrics. Despite this decrease, the model's ability to detect and localize lymph nodes was high enough to establish its potential clinical utility. These results demonstrate that the model can reliably extract diagnostically relevant information, confirming its feasibility for clinical support.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Modality
Hypermetabolism
Contrast Agent
View
Overall
ROUGE-L
LLaVA-NeXT
0.046 ± (0.0427)
0.1433 ± (0.1602)
0.0688 ± (0.0601)
0.0385 ± (0.0666)
0.0741 ± (0.0913)
GPT-4o
0.3714 ± (0.0027)
0.0999 ± (0.0025)
0.2228 ± (0.0011)
0.4597 ± (0.0018)
0.3044 ± (0.1319)
Ours
0.8088 ± (0.0)
0.8237 ± (0.0)
0.8824 ± (0.0)
0.7726 ± (0.0)
0.7961 ± (0.0757)
ROUGE-S
LLaVA-NeXT
0.0395 ± (0.0374)
0.1678 ± (0.0682)
0.0547 ± (0.0478)
0.0289 ± (0.05)
0.0727 ± (0.0731)
Appendix
Table 6: The performance of internal validation of level 1 tasks.
Model
Modality
Hypermetabolism
View
Overall
ROUGE-L
LLaVA-NeXT
0.0468 ± (0.0046)
0.084 ± (0.0153)
0.0846 ± (0.0097)
0.0718 ± (0.0209)
GPT-4o
0.3769 ± (0.0034)
0.0901 ± (0.003)
0.4797 ± (0.005)
0.3285 ± (0.1510)
Ours
0.8117 ± (0.0)
0.7211 ± (0.0)
0.7692 ± (0.0)
0.7673 ± (0.0393)
ROUGE-S
LLaVA-NeXT
0.0403 ± (0.0046)
0.1289 ± (0.0456)
0.0635 ± (0.0071)
0.0776 ± (0.0460)
Appendix
Table 7: The performance of external validation of level 1.
Model
ROUGE-L
ROUGE-S
Cosine Similarity
Precision
Recall
F1
Internal Validation
LLaVA-NeXT
0.0026 ± (0.0000)
0.0025 ± (0.0000)
0.0014 ± (0.0000)
0.0025 ± (0.0000)
0.0028 ± (0.0000)
0.0026 ± (0.0000)
GPT-4o
0.0503 ± (0.0009)
0.0373 ± (0.0007)
0.0243 ± (0.0006)
0.0379 ± (0.0008)
0.0810 ± (0.0022)
0.0512 ± (0.0012)
Ours
0.8856 ± (0.0000)
0.8912 ± (0.0000)
0.8413 ± (0.0000)
0.8912 ± (0.0000)
0.8804 ± (0.0000)
0.8856 ± (0.0000)
External Validation
LLaVA-NeXT
0.0093 ± (0.0001)
0.0096 ± (0.0001)
0.0069 ± (0.0001)
0.0108 ± (0.0001)
0.0103 ± (0.0001)
0.0103 ± (0.0001)
Appendix
Table 8: Internal and external validation of level 2.
Head and neck cancers (HNC) represent a significant global health burden, with accurate tumor delineation being essential for effective radiotherapy planning. The complexity of the oropharyngeal anatomy, combined with the heterogeneous appearance of tumors on imaging, makes manual segmentation time-intensive and subject to inter-observer variability. Beyond segmentation, predicting long-term clinical outcomes, such as recurrence-free survival (RFS), and determining human papillomavirus (HPV) status from noninvasive imaging, remain challenging yet clinically valuable goals. The HECKTOR 2025 challenge addresses these needs by establishing a comprehensive benchmark for automated HNC analysis using multimodal PET/CT imaging and electronic health records. Building on previous editions (2020-2022), this challenge features an expanded multi-institutional dataset comprising over 1,100 patients from 10 centers worldwide. Participants were tasked with three complementary objectives: (1) segmenting primary gross tumor volumes (GTVp) and metastatic lymph nodes (GTVn), (2) predicting recurrence-free survival, and (3) classifying HPV status. The challenge attracted 35 registered teams, with 15 final submissions evaluated on a held-out test set. Top-performing algorithms achieved a mean Dice similarity coefficient of 0.75 for segmentation, a concordance index of 0.66 for survival prediction, and a balanced accuracy of 0.56 for HPV classification. This paper presents a comprehensive analysis of the submitted methodologies, evaluates their performance across different lesion characteristics, and discusses their implications for clinical translation in automated oncology workflows and decision support systems.
Numan Saeed, Salma Hassan, Shahad Hardan +27
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE. · Amsterdam UMC, Amsterdam, The Netherlands. · The Netherlands Cancer Institute, Amsterdam, The Netherlands. +22
PET/CT imaging is pivotal in oncology and nuclear medicine, yet summarizing complex findings into precise diagnostic impressions is labor-intensive. While LLMs have shown promise in medical text generation, their capability in the highly specialized domain of PET/CT remains underexplored. We introduce PET-F2I-41K (PET Findings-to-Impression Benchmark), a large-scale benchmark for PET/CT impression generation using LLMs, constructed from over 41k real-world reports. Using PET-F2I-41K, we conduct a comprehensive evaluation of 27 models across proprietary frontier LLMs, open-source generalist models, and medical-domain LLMs, and we develop a domain-adapted 7B model (PET-F2I-7B) fine-tuned from Qwen2.5-7B-Instruct via LoRA. Beyond standard NLG metrics (e.g., BLEU-4, ROUGE-L, BERTScore), we propose three clinically grounded metrics - Entity Coverage Rate (ECR), Uncovered Entity Rate (UER), and Factual Consistency Rate (FCR) - to assess diagnostic completeness and factual reliability. Experiments reveal that neither frontier nor medical-domain LLMs perform adequately in zero-shot settings. In contrast, PET-F2I-7B achieves substantial gains (e.g., 0.708 BLEU-4) and a 3.0x improvement in entity coverage over the strongest baseline, while offering advantages in cost, latency, and privacy. Beyond this modeling contribution, PET-F2I-41K establishes a standardized evaluation framework to accelerate the development of reliable and clinically deployable reporting systems for PET/CT.
Yuchen Liu, Wenbo Zhang, Liling Peng +6
Fudan University · Shanghai Academy of Artificial Intelligence for Science · Shanghai Universal Medical Imaging Diagnostic Center +1
Accurate lesion segmentation in PET/CT is critical for oncology, yet remains challenging because physiologic tracer uptake and artifacts can mimic malignant signal. We present RADIANT-PET, a reasoning-augmented framework that couples a high-sensitivity voxel-level segmentation model with lesion-level large language model (LLM) adjudication. Candidate uptake regions are generated with a deliberately permissive segmentation stage, then converted into structured textual descriptions that summarize uptake intensity, morphology, and regional and global anatomical context. An LLM classifies each candidate as true lesion vs. false positive, optionally leveraging the radiology report as additional clinical context. To strengthen lesion-level reasoning, we further optimize a local LLM via reinforcement learning using Group Relative Policy Optimization, rewarding correct lesion classification and anatomically concordant site assignment. Across AutoPET and an OSU test cohort, RADIANT-PET consistently outperforms strong image-only baselines, with the largest improvements observed when radiology reports are provided. Overall, these results demonstrate that LLM-based lesion-level reasoning adds a novel reasoning layer beyond conventional segmentation, suppressing physiologic false positives and aligning voxel-level predictions with clinical interpretation. The project repository is available at: https://github.com/jwang-580/RADIANT-PET.
Division of Hematology, The Ohio State University Comprehensive Cancer Center, Columbus, OH, USA · Department of Radiology, Mahidol University, Bangkok, Thailand · Department of Radiology, Mettapracharak Hospital, Thailand +1