cs.CVAug 15, 2026

FZ-VLM: A Two Stage Florence-Zephyr Vision Language Model Framework for Pulmonary Nodule Characterization and Clinical Decision Making

Authors: Pramit Dutta, Jenita Manokaran, Richa Mittal, Ryan Appleby, Eranga Ukwatta

Organizations: College of Engineering, University of Guelph, Guelph, ON N1G 2W1, Canada · Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON M4G 1R8, Canada · Guelph General Hospital, Guelph, ON N1E 4J4, Canada · Department of Clinical Studies, Ontario Veterinary College, University of Guelph, Guelph, ON N1G 2W1, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Lung cancer remains one of the leading causes of cancer-related mortality worldwide, and Computed Tomography (CT) is a primary imaging tool for screening and followup assessment. After pulmonary nodule detection, radiologists manually assess anatomical location, diameter, margin characteristics, and attenuation type to support risk assessment and clinical decision-making. However, this post-detection workflow is time-consuming and can be affected by inter-observer variability. Existing Artificial Intelligence methods often focus on isolated tasks, limiting their use as a unified, clinically grounded interpretation framework. This study presents FZ-VLM, a two-stage Florence-Zephyr Vision Language Model framework for unified structured pulmonary nodule characterization in lung CT. The framework uses a fine-tuned Florence-2 model to extract radiological attributes from expert-annotated 2D axial CT slices, while a Zephyr-7B model uses these attributes to generate nodule descriptions, follow-up recommendations, and longitudinal analyses. Results showed that the Stage 1 model achieved 77.18% accuracy for anatomical location, 67.96% accuracy for margin characteristics, and 79.13% accuracy for attenuation type, with a Mean Absolute Error of 2.58 mm for diameter estimation, outperforming evaluated GPT-4-based baselines as well as the human baseline. Expert radiologist evaluation of Stage 2 showed 93.9% accuracy, 98.6% completeness score, 76.1% clinical relevance, and an overall score of 89.5%. Safety analysis showed that most outputs were clinically safe, although some follow-up recommendations still required expert review. To the best of our knowledge, this study presents the first two-stage Vision-Language Model framework for structured nodule characterization and clinical decision-making.

Figures & tables

Figure 1: Architecture of the FZ-VLM framework. The pipeline executes attribute extraction by a finetuned Florence-2 with explainable attention heatmaps. These parameters are processed through a hierarchical integration layer to serve as contextual input for a Zephyr-7B model that performs zero-shot inference to generate structured clinical outputs.
Figure 2: Hierarchical integration of the radiological attributes extracted by Florence-2. Individual nodule features are combined into various profiles for different assessment.
ParameterPrecisionRecallF1-scoreAccuracy
Anatomical location0.750.720.7077.18%
Margin characteristics0.650.640.6467.96%
Attenuation type0.580.520.5379.13%
Nodule diameterMAE = 2.58 mm; SD = 3.73 mm; Accuracy = 21.4–87.9%
Table 1: Performance of the fine-tuned Florence-2 model on the test set. Macro average precision, recall, F1-score, and Accuracy are reported for categorical attributes. For diameter prediction, accuracy was measured using a tolerance from 0 to 5 mm.
Figure 3: Qualitative explainability and error analysis using attention heatmaps. Representative cases show correct predictions and error patterns in Stage 1 radiological attribute extraction. LUL: left upper lobe; RUL: right upper lobe.
AttributePre-trained GPT-4Fine-tuned GPT-4Human ReferenceProposed Model
Location (Accuracy)36.31%43.68%–77.18%
Diameter (MAE)4.74 mm3.83 mm3.2 mm2.58 mm
Margin (Accuracy)38.42%39.47%–67.96%
Attenuation (Accuracy)37.36%63.68%58–77%79.13%
Table 2: Performance comparison of the proposed fine-tuned Florence-2 model with GPT-4 baselines and available human reference values available in the literature for pulmonary nodule characterization tasks. For diameter, the reported value represents inter-reader variability in manual diameter measurement, whereas for attenuation it represents the reported range of radiologist performance for solid versus subsolid nodule classification.
Figure 4: Performance of the finetuned Florence-2 model across different input slice offsets relative to the expert annotated reference slice.
Figure 5: Human evaluation results for Stage 2 outputs across nodule description, follow-up recommendation, longitudinal temporal analysis, and combined average performance.
Figure 6: Representative pulmonary nodule margin characteristics and attenuation types on axial CT images. The upper row shows different margin characteristics, while the lower row shows various attenuation types. The examples illustrate the visual overlap between margin characteristics and attenuation types.

Explore similar work

CardsList
Oct 12, 2026

NoduleFlow: Automatic 3D Lung Nodule Detection and Segmentation

Low dose Computed Tomography (CT) is the standard imaging modality for lung cancer screening. On CT scans, lung cancer typically manifests as pulmonary nodules, whose manual identification is time-consuming and prone to observer variability, image quality limitations, and suboptimal reading conditions. Moreover, subtle temporal changes are often difficult to quantify reliably in CT scans from the same patient across follow-up examinations. Deep learning-based approaches for the automatic detection and segmentation of lung nodules have demonstrated improved accuracy and efficiency. However, most existing approaches provide only coarse region-of-interest localisation, rely on prior lesion annotations or human supervision, or treat detection and segmentation as separate tasks. In this work, we present NoduleFlow, a framework for lung nodule detection, classification, and segmentation directly from series of CT volumes. Our proposal consists of four interconnected modules and operates in an automated manner at inference time, without requiring prior lesion annotations, manual initialisation, or region-of-interest pre-selection. We further investigate whether nodule localization benefits from the propagation of candidate regions across follow-up scans through inter-scan registration fields, as opposed to conventional independent per-scan analysis. The generalisability of the framework is systematically evaluated across four independent datasets. The results underscore the potential of automated pipelines for lung nodule analysis in screening and diagnostic settings.

Giulia Romoli, Filippo Ruffini, Francesco Di Feola +3

Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Universitetstorget 4, Umeå, Sweden. · Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Via Alvaro del Portillo 21, Rome, Italy. · Department of Naval, Electrical, Electronic, and Telecommunications Engineering, University of Genoa, Via Opera Pia 11A, Genoa, Italy.

Medical Image Segmentation3D Medical Image SegmentationMedical Image Anomaly DetectionLongitudinal Medical Image Analysis
AskPDF
Aug 11, 2026

ConfTriage: A Calibration-Aware LLM Triage Framework for Pulmonary Nodule Malignancy with Selective Specialist Deferral

Pulmonary nodule malignancy prediction typically depends on image-trained specialist deep learning (DL) models that require substantial annotated imaging data and task-specific training. We investigate whether a generalist large language model (LLM), reading only a faithful natural-language rendering of standard nodule attributes, can serve as a calibrated triage layer. We propose ConfTriage, a confidence-calibrated method built on three pillars: language as the modality, calibration as the safety mechanism, and a selective specialist DL backstop for low-confidence cases. We prove two guarantees: a finite-sample combined-error bound yielding an explicit per-threshold operational certificate, and an oracle inequality showing that excess risk over the Bayes-optimal deferral classifier is controlled by the L1 calibration error of the LLM probability. A controlled seven-way input ablation across five frontier LLMs on LIDC-IDRI shows that natural-language descriptions dominate the diagnostic signal, while low-level image statistics are essentially diagnostically vacuous. ConfTriage achieved an F1 score of 88.22% and an AUC of 0.92, resolving 76.5% of cases using zero-shot LLM inference alone and referring only uncertain cases to the specialist DL backstop. These results demonstrate that clinically meaningful diagnostic information can be captured through structured radiological descriptions and leveraged by calibrated LLMs for selective referral. The framework suggests a practical pathway for combining generalist LLM prediction with specialist AI models in medical decision-support systems. Source code is publicly available at https://github.com/rabiul-ai/ConfTriage.

Md Rabiul Islam, Samir Abdaljalil, Erchin Serpedin +1

Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA · College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar

Selective PredictionClinical Decision SupportLanguage Model CalibrationClinical Prediction
AskPDF
Aug 8, 2026

Distilling CT Foundation Models into Editable Concept Bottlenecks for Lung Nodule Malignancy Prediction

Foundation models provide transferable CT representations, but predictions based directly on these embeddings are difficult to interpret. We developed concept bottleneck models that map two frozen CT foundation-model representations to eight radiologist-defined pulmonary-nodule attributes and predict malignancy from the estimated concepts and nodule size. The models included CT-FM, a whole-CT self-supervised encoder using a 96^3-voxel nodule-centered patch, and FMCIB, a nodule-focused contrastive encoder using a 50-mm crop. Eight ridge-regression concept heads were trained on 2,610 LIDC-IDRI nodules. Malignancy models were trained on LUNA25 and evaluated on a held-out internal test set and the external DLCS cohort. Concept fidelity was assessed using five-fold cross-validated R^2, and malignancy discrimination was assessed using AUROC with 95% confidence intervals estimated by patient-grouped bootstrap resampling. Concept fidelity was modest but higher for FMCIB than CT-FM for subtlety (R2, 0.24 vs. 0.11), spiculation (0.17 vs. 0.08), texture (0.17 vs. 0.07), and lobulation (0.15 vs. 0.05). Internally, the CT-FM and FMCIB concept+size models achieved AUROCs of 0.86 (95% CI, 0.80-0.92) and 0.86 (0.79-0.92), respectively. Externally, AUROCs were 0.72 (0.68-0.75) and 0.73 (0.70-0.76), compared with 0.73 for nodule size alone and 0.60 and 0.67 for the corresponding embedding only probes. Additive predictions could be decomposed into feature-level contributions and modified through controlled concept interventions. Concept bottlenecks provided transparent malignancy predictions with discrimination similar to nodule size alone, while differences in concept fidelity suggest that concept recovery depends on the underlying foundation-model representation.

Fakrul Islam Tushar, Stephen Adamo, Geoffrey D. Rubin

Department of Radiology and Imaging Sciences, University of Arizona, Tucson, AZ, USA

Medical Imaging Foundation ModelsConcept Bottleneck ModelsClinical PredictionOncologyExplainable Medical Image AnalysisComputed Tomography
AskPDF