FZ-VLM: A Two Stage Florence-Zephyr Vision Language Model Framework for Pulmonary Nodule Characterization and Clinical Decision Making

Pramit Dutta; Jenita Manokaran; Richa Mittal; Ryan Appleby; Eranga Ukwatta

cs.CVAug 15, 2026

FZ-VLM: A Two Stage Florence-Zephyr Vision Language Model Framework for Pulmonary Nodule Characterization and Clinical Decision Making

Authors: Pramit Dutta, Jenita Manokaran, Richa Mittal, Ryan Appleby, Eranga Ukwatta

Organizations: College of Engineering, University of Guelph, Guelph, ON N1G 2W1, Canada · Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON M4G 1R8, Canada · Guelph General Hospital, Guelph, ON N1E 4J4, Canada · Department of Clinical Studies, Ontario Veterinary College, University of Guelph, Guelph, ON N1G 2W1, Canada

Abstract

Lung cancer remains one of the leading causes of cancer-related mortality worldwide, and Computed Tomography (CT) is a primary imaging tool for screening and followup assessment. After pulmonary nodule detection, radiologists manually assess anatomical location, diameter, margin characteristics, and attenuation type to support risk assessment and clinical decision-making. However, this post-detection workflow is time-consuming and can be affected by inter-observer variability. Existing Artificial Intelligence methods often focus on isolated tasks, limiting their use as a unified, clinically grounded interpretation framework. This study presents FZ-VLM, a two-stage Florence-Zephyr Vision Language Model framework for unified structured pulmonary nodule characterization in lung CT. The framework uses a fine-tuned Florence-2 model to extract radiological attributes from expert-annotated 2D axial CT slices, while a Zephyr-7B model uses these attributes to generate nodule descriptions, follow-up recommendations, and longitudinal analyses. Results showed that the Stage 1 model achieved 77.18% accuracy for anatomical location, 67.96% accuracy for margin characteristics, and 79.13% accuracy for attenuation type, with a Mean Absolute Error of 2.58 mm for diameter estimation, outperforming evaluated GPT-4-based baselines as well as the human baseline. Expert radiologist evaluation of Stage 2 showed 93.9% accuracy, 98.6% completeness score, 76.1% clinical relevance, and an overall score of 89.5%. Safety analysis showed that most outputs were clinically safe, although some follow-up recommendations still required expert review. To the best of our knowledge, this study presents the first two-stage Vision-Language Model framework for structured nodule characterization and clinical decision-making.

Figures & tables

Figure 1: Architecture of the FZ-VLM framework. The pipeline executes attribute extraction by a finetuned Florence-2 with explainable attention heatmaps. These parameters are processed through a hierarchical integration layer to serve as contextual input for a Zephyr-7B model that performs zero-shot inference to generate structured clinical outputs.

Figure 2: Hierarchical integration of the radiological attributes extracted by Florence-2. Individual nodule features are combined into various profiles for different assessment.

Parameter	Precision	Recall	F1-score	Accuracy
Anatomical location	0.75	0.72	0.70	77.18%
Margin characteristics	0.65	0.64	0.64	67.96%
Attenuation type	0.58	0.52	0.53	79.13%
Nodule diameter	MAE = 2.58 mm; SD = 3.73 mm; Accuracy = 21.4–87.9%

Table 1: Performance of the fine-tuned Florence-2 model on the test set. Macro average precision, recall, F1-score, and Accuracy are reported for categorical attributes. For diameter prediction, accuracy was measured using a tolerance from 0 to 5 mm.

Panel of Figure 3 — Figure 3: Qualitative explainability and error analysis using attention heatmaps. Representative cases show correct predictions and error patterns in Stage 1 radiological attribute extraction. LUL: left upper lobe; RUL: right upper lobe.

Attribute	Pre-trained GPT-4	Fine-tuned GPT-4	Human Reference	Proposed Model
Location (Accuracy)	36.31%	43.68%	–	77.18%
Diameter (MAE)	4.74 mm	3.83 mm	3.2 mm	2.58 mm
Margin (Accuracy)	38.42%	39.47%	–	67.96%
Attenuation (Accuracy)	37.36%	63.68%	58–77%	79.13%

Table 2: Performance comparison of the proposed fine-tuned Florence-2 model with GPT-4 baselines and available human reference values available in the literature for pulmonary nodule characterization tasks. For diameter, the reported value represents inter-reader variability in manual diameter measurement, whereas for attenuation it represents the reported range of radiologist performance for solid versus subsolid nodule classification.

Figure 4: Performance of the finetuned Florence-2 model across different input slice offsets relative to the expert annotated reference slice.

Figure 5: Human evaluation results for Stage 2 outputs across nodule description, follow-up recommendation, longitudinal temporal analysis, and combined average performance.

Figure 6: Representative pulmonary nodule margin characteristics and attenuation types on axial CT images. The upper row shows different margin characteristics, while the lower row shows various attenuation types. The examples illustrate the visual overlap between margin characteristics and attenuation types.

Explore similar work

NoduleFlow: Automatic 3D Lung Nodule Detection and Segmentation
Oct 12, 2026Giulia Romoli, Filippo Ruffini, Francesco Di Feola +3Medical Image Segmentation 3D Medical Image Segmentation
Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Universitetstorget 4, Umeå, Sweden. · Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Via Alvaro del Portillo 21, Rome, Italy. · Department of Naval, Electrical, Electronic, and Telecommunications Engineering, University of Genoa, Via Opera Pia 11A, Genoa, Italy.
ConfTriage: A Calibration-Aware LLM Triage Framework for Pulmonary Nodule Malignancy with Selective Specialist Deferral
Aug 11, 2026Md Rabiul Islam, Samir Abdaljalil, Erchin Serpedin +1Selective Prediction Clinical Decision Support
Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA · College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
Distilling CT Foundation Models into Editable Concept Bottlenecks for Lung Nodule Malignancy Prediction
Aug 8, 2026Fakrul Islam Tushar, Stephen Adamo, Geoffrey D. RubinMedical Imaging Foundation Models Concept Bottleneck Models
Department of Radiology and Imaging Sciences, University of Arizona, Tucson, AZ, USA

Oct 12, 2026J/K move · Enter open · S save

NoduleFlow: Automatic 3D Lung Nodule Detection and Segmentation

Giulia Romoli, Filippo Ruffini, Francesco Di Feola, Valerio Guarrasi+2

Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Universitetstorget 4, Umeå, Sweden. · Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Via Alvaro del Portillo 21, Rome, Italy. · Department of Naval, Electrical, Electronic, and Telecommunications Engineering, University of Genoa, Via Opera Pia 11A, Genoa, Italy.

Medical Image Segmentation 3D Medical Image Segmentation Medical Image Anomaly Detection Longitudinal Medical Image Analysis

Open paper AskPDF arXiv

Low dose Computed Tomography (CT) is the standard imaging modality for lung cancer screening. On CT scans, lung cancer typically manifests as pulmonary nodules, whose manual identification is time-consuming and prone to observer variability, image quality limitations, and suboptimal reading conditions. Moreover, subtle temporal changes are often difficult to quantify reliably in CT scans from the same patient across follow-up examinations. Deep learning-based approaches for the automatic detection and segmentation of lung nodules have demonstrated improved accuracy and efficiency. However, most existing approaches provide only coarse region-of-interest localisation, rely on prior lesion annotations or human supervision, or treat detection and segmentation as separate tasks. In this work, we present NoduleFlow, a framework for lung nodule detection, classification, and segmentation directly from series of CT volumes. Our proposal consists of four interconnected modules and operates in an automated manner at inference time, without requiring prior lesion annotations, manual initialisation, or region-of-interest pre-selection. We further investigate whether nodule localization benefits from the propagation of candidate regions across follow-up scans through inter-scan registration fields, as opposed to conventional independent per-scan analysis. The generalisability of the framework is systematically evaluated across four independent datasets. The results underscore the potential of automated pipelines for lung nodule analysis in screening and diagnostic settings.

Abstract

Explore similar work

NoduleFlow: Automatic 3D Lung Nodule Detection and Segmentation

ConfTriage: A Calibration-Aware LLM Triage Framework for Pulmonary Nodule Malignancy with Selective Specialist Deferral

Distilling CT Foundation Models into Editable Concept Bottlenecks for Lung Nodule Malignancy Prediction