Can Multimodal Decision Models Classify Breast Ultrasound Images? Benchmarking CLEF and CLEF-Flash Across Prompts and Datasets

Tran Cao Minh; Hai Tran; Ha Minh Tan

eess.IVOct 9, 2026

Can Multimodal Decision Models Classify Breast Ultrasound Images? Benchmarking CLEF and CLEF-Flash Across Prompts and Datasets

Authors: Tran Cao Minh, Hai Tran, Ha Minh Tan

Organizations: Research and Development Center, VNSO Technology JSC, Ho Chi Minh City, Vietnam · Department of Information Technology, Ho Chi Minh University of Education, Ho Chi Minh City, Vietnam · Faculty of Information Science and Engineering, University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam

Abstract

Multimodal decision models return probabilities over predefined alternatives, offering a structured interface for image classification. This study benchmarks inference with CLEF and CLEF-Flash on 2818 breast ultrasound images from Dataset B, BUSI, and BUS-BRA. Dataset B and BUS-BRA use binary benign-malignant classification, whereas BUSI includes normal, benign, and malignant categories. Model parameters remain unchanged, and original images are submitted without user-defined preprocessing. Three prompts provide minimal labels, an explicit classification task, or sonographic criteria. We report accuracy, macro F1, balanced accuracy, macro precision, throughput, and latency. CLEF-Flash with the clinical prompt achieves the highest accuracy on all three datasets: 74.85% on Dataset B, 66.28% on BUSI, and 51.63% on BUS-BRA. On BUSI, however, CLEF with the task-guided prompt achieves the highest macro F1 (64.51%) and balanced accuracy (68.72%). CLEF-Flash has lower latency in every matched comparison. Clinical instructions improve CLEF-Flash accuracy relative to the minimal prompt but reduce CLEF accuracy on all datasets. Confusion matrices reveal frequent benign-to-malignant errors, particularly on BUS-BRA, where all configurations fall below the 67.63% accuracy of a majority-class predictor. These findings demonstrate dataset-and prompt-dependent performance and motivate class-specific evaluation before applying multimodal decision models to breast ultrasound analysis.

Figures & tables

Figure 1: Conceptual comparison between conventional large language or vision language models and decision models. Generative models produce open ended outputs, whereas decision models operate over a predefined decision schema and return structured probabilities for bounded alternatives.

Panel of Figure 2 — Figure 2: Architecture of the CLEF decision models applied to breast ultrasound classification. The image is encoded by the backbone’s vision encoder, while the state and the typed decision schema are serialized as text. The multimodal backbone, held fixed during evaluation, performs a single prefill-only pass, and the joint schema head routes evidence from the input to each option, applies cross-field attention, and outputs one logit per admissible option. A per-question softmax yields the probability distribution over normal , benign and malignant . The panels show the respective backbones within the shared decision-model design. The three-class output illustrates BUSI; binary tasks use only benign and malignant options. Probabilities shown are illustrative.

Dataset	Normal	Benign	Malignant	Task
Dataset B	–	109	54	Binary
BUSI	133	437	210	Three class
BUS-BRA	–	1268	607	Binary

Table 1: Summary of the breast ultrasound datasets used in this study.

Prompt	Description
P1	Minimal classification instruction containing only the task and diagnostic alternatives.
P2	Task-guided instruction explaining the diagnostic classification task.
P3	Clinical knowledge guided instruction incorporating relevant sonographic characteristics such as lesion shape, margin, orientation, echogenicity, and posterior acoustic features.

Table 2: Prompting strategies evaluated in this study.

Panel of Figure 3 — Figure 3: Overview of the inference and evaluation pipeline across datasets, models, and prompting strategies. The available diagnostic categories depend on the dataset, as specified in Table 1 .

Prompt	Model	Acc. (%)	F1 score (%)	Bal. Acc. (%)	Precision (%)	FPS	Latency (ms/image)
Dataset B
P1	CLEF	70.55	68.76	70.51	68.54	0.125	7971.36
P1	CLEF-Flash	73.62	71.74	73.27	71.29	0.414	2415.63
P2	CLEF	65.03	64.12	67.31	65.35	0.126	7962.04
P2	CLEF-Flash	73.01	71.37	73.28	71.04	0.430	2324.27
P3	CLEF	65.03	64.95	71.52	70.23	0.126	7959.96

Table 3: Classification performance and inference speed on Dataset B, BUSI, and BUS-BRA. Acc.: accuracy; Bal. Acc.: balanced accuracy. F1 score and precision are macro averages. FPS is reported in frames per second and latency in milliseconds per image. Bold indicates the best value for each metric within each dataset (higher is better except for latency).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Confusion matrices on Dataset B. Rows indicate reference labels and columns indicate predicted labels. The upper row shows CLEF-Flash and the lower row shows CLEF; columns correspond to P1, P2, and P3. Cell values are image counts.

Figure 2: Confusion matrices on BUSI. Rows indicate reference labels and columns indicate predicted labels. The upper row shows CLEF-Flash and the lower row shows CLEF; columns correspond to P1, P2, and P3. Cell values are image counts.

Figure 3: Confusion matrices on BUS-BRA. Rows indicate reference labels and columns indicate predicted labels. The upper row shows CLEF-Flash and the lower row shows CLEF; columns correspond to P1, P2, and P3. Cell values are image counts.

Explore similar work

Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumors
Jul 2, 2026Chuxi Nan, Di Wu, Hongming Guo +4Multimodal Classification Ultrasound Imaging
School of Integrated Circuits, Wuxi Vocational College of Science and Technology · School of Advanced Technology, Xi’an Jiaotong-Liverpool University · Shanghai Gem Science and Technology Co., Ltd. +2
Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation
Apr 30, 2026Jing Zhang, Wentao Jiang, Tao Huang +8Medical Image Analysis Multimodal Reasoning
School of Computer Science, Wuhan University, China · The Central Hospital of Wuhan, China · Taizhou Hospital of Zhejiang Province, China +1
Image-Based Breast Implant Detection for Mammography Dataset Curation and Near-Real-Time Deployment: Comparing Foundation Models and Task-Specific Convolutional Models
Oct 1, 2026Vasisht Ishwar, Hari Trivedi, Young Seok Jeon +5Medical Image Classification Medical Imaging Foundation Models
Department of Radiology and Imaging Sciences, Emory University School of Medicine, Atlanta, GA, USA · University of Maryland, College Park, MD, USA

Jul 2, 2026J/K move · Enter open · S save

Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumors

Chuxi Nan, Di Wu, Hongming Guo, Ning Cao+3

School of Integrated Circuits, Wuxi Vocational College of Science and Technology · School of Advanced Technology, Xi’an Jiaotong-Liverpool University · Shanghai Gem Science and Technology Co., Ltd.+2

Multimodal Classification Ultrasound Imaging Multimodal Fusion Medical Image Benchmarks

Open paper AskPDF arXiv

Breast fibroadenoma (FA) and phyllodes tumor (PT) are fibroepithelial breast lesions with highly overlapping appearances on B-mode ultrasound, making benign and borderline PT prone to being misclassified as FA and complicating preoperative decision-making. Existing computer-aided diagnosis methods commonly rely on single-modal imaging features and insufficiently exploit complementary clinical and textual information. To address this limitation, we construct the FAPT-M Dataset, a pathology-confirmed multimodal dataset comprising 910 patients with strictly reviewed ultrasound images, structured clinical attributes, and ultrasound diagnostic descriptions. Based on this dataset, we propose a clinically guided multimodal framework that integrates DenseNet-based visual encoding, CLIP-inspired text encoding, and lightweight clinical encoding, and further introduces clinical-conditioned adaptive modulation, cross-modal Transformer fusion, and dual-path representation learning to improve feature alignment and multimodal interaction. Under patient-level five-fold cross-validation, the proposed method achieves an accuracy of 77.64%, F1-score of 73.38%, and AUC of 89.74%, outperforming representative CNN-, Transformer-, and vision-language-based baselines. Ablation studies and class-balanced evaluations further confirm the contribution of three-modality fusion and the key architectural components. Overall, this work provides an effective multimodal approach for fine-grained FA-PT classification and establishes a high-quality benchmark for multimodal breast ultrasound analysis.

Abstract

Explore similar work

Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumors

Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation

Image-Based Breast Implant Detection for Mammography Dataset Curation and Near-Real-Time Deployment: Comparing Foundation Models and Task-Specific Convolutional Models