Extraction of clinical findings from mammography and breast ultrasound reports: a comparison between specialists and Artificial Intelligence
Authors: Lorenzo Farias, Hanna Reckziegel, Daniela Duarte da Silva Bagatini, Daniel Schulz, Gabriela de Andrade Monteiro, Letícia Zanatta, Ana Laura Brill Thum, Priscila Schmidt Lora, +3 more
Organizations: Department of Engineering, Architecture and Computing, Universidade de Santa Cruz do Sul (UNISC), Santa Cruz do Sul, RS, Brazil · Graduate Program in Industrial Systems and Processes (PPGSPI), Universidade de Santa Cruz do Sul (UNISC), Santa Cruz do Sul, RS, Brazil · Graduate Program in Production and Systems Engineering (PPGEPS), Universidade do Vale do Rio dos Sinos (UNISINOS), São Leopoldo, RS, Brazil · Faculty of Medicine (FaMed), Universidade Federal do Rio Grande (FURG), Rio Grande, RS, Brazil · School of Health, Universidade do Vale do Rio dos Sinos (UNISINOS), São Leopoldo, RS, Brazil · Clínica de Mastologia Dra. Ana Paula Muller Ltda., Porto Alegre, RS, Brazil · Graduate Program in Administration (PPGA) and Graduate Program in Project Management (PPGP), Universidade Nove de Julho (UNINOVE), São Paulo, SP, Brazil
Breast cancer is the leading cause of cancer-related death among women in Brazil, and the time between the request and the release of mammography reports directly influences adherence to screening, making the agility in processing these reports a critical factor for early diagnosis. In this context, this study compares the performance of a Large Language Model (LLM) with manual extraction performed by a team of health researchers in identifying clinical findings from mammography and breast ultrasound reports written in Brazilian Portuguese. Named Entity Recognition (NER) was applied through Prompt Engineering using a few-shot strategy, employing the Gemini 2.5 Flash model, selected from preliminary exploratory tests with four candidate models. The Gemini 2.5 Flash model demonstrated the best performance, achieving a Macro F1 of 0.91 and a Micro F1 of 0.98. The subjective validation, in which 29 exams of different formats were evaluated by health researchers using a Likert scale, yielded an agreement index of 93.1%. The model outperformed human extraction in overall Macro F1 (0.91 vs. 0.72), as in four reports the model correctly identified information that had been omitted or incorrectly recorded during manual extraction, demonstrating its potential as a complementary verification tool alongside specialists. The results confirm the hypothesis that LLMs, when instructed through Prompt Engineering, can achieve performance comparable to or superior to manual extraction by health professionals.
Figures & tables
Figure 1: Mapping process flowchart. Illustrates the stages from bibliographic search for clinical variables, validation with health professionals, identification in reports, FHIR standardization, tabulation in spreadsheet, and final validation.
Model
Benchmark
Latency (s)
Input P.
Output P.
GPT-4o (OpenAI)
1408
0.29
$2.50
$10.00
Claude 3.5 Sonnet (Anthropic)
1283
0.95
$3.00
$15.00
DeepSeek R1
1359
3.40
$1.35
$5.40
Gemini 2.5 Flash (Google)
1394
8.15
$0.15
$3.50
Table 1: Characteristics of models evaluated in exploratory tests. Prices in USD per million tokens. Benchmark : LMArena score [ 22 ] .
Figure 2: Technical methodology flowchart. Shows the stages from report anonymization to prompt preparation, model processing, JSON output, and use in the final prototype.
Clinical Entity
Accuracy
Sensitiv.
Specific.
Precision
F1
Cysts
1.00
1.00
1.00
1.00
1.00
Nodules
1.00
1.00
1.00
1.00
1.00
Calcifications
0.97
0.94
1.00
1.00
0.97
Microcalcifications
0.97
1.00
0.96
0.50
0.66
Macro F1
0.98
0.98
0.99
0.87
0.91
Micro F1 a
–
0.98
–
0.98
0.98
Table 2: Performance per clinical entity of the Gemini 2.5 Flash model after gold standard adjudication.
Figure 3: Global Confusion Matrix (Binary Entities). The model recorded 75 TN and 39 TP, with only 1 FP and 1 FN across 116 evaluated instances.
Figure 4: Confusion Matrix for the BI-RADS Category. All 29 examinations were correctly classified in categories 1 to 6 (Accuracy = 1.00), with no overlap between classes.
Clinical Entity
F1-Score
Sensitivity
Human
AI
Human
AI
Cysts
1.00
1.00
1.00
1.00
Nodules
0.90
1.00
0.90
1.00
Calcifications
0.97
0.97
0.94
0.94
Microcalcifications
0.00 a
0.66
0.00
1.00
Macro F1
0.72
0.91
0.71
0.98
Table 3: Comparison of F1-Score and Sensitivity per clinical entity between human and Gemini 2.5 Flash (adjudicated ground truth as gold standard).
Figure 5: Level of professional agreement with AI. The Global Acceptance Index calculated over 58 evaluations (Likert scale, two professionals) was 93.1%.
Figure 6: Levels of agreement between professionals by examination. Dark green = full agreement (100%); light green = subtle divergence; lilac = disagreement. Only Exam 4 showed substantive divergence regarding the punctiform calcification.
Figure 7: Exam 4: comparison between AI extraction (Gemini 2.5 Flash) and human. The AI applied a dimensional criterion ( < 0.5 cm) to classify the calcification as a microcalcification, while the human adopted morphological and clinical criteria. Disagreement correctable through prompt adjustment.
Figure 8: Exam 15: comparison between AI and human extraction. Report selected for its tendency to induce hallucinations in PDF format; full agreement achieved across all variables with .txt input.
Figure 9: Exam 6: comparison between AI and human extraction. Reference report in tests, with full agreement – absence of cyst, calcification, and microcalcifications, and presence of nodule with description.
Figure 10: Exam 14: comparison between AI and human extraction, demonstrating full agreement including BI-RADS 6 classification and complete nodule description.
Department of Radiology & Nuclear Medicine, Erasmus MC, Rotterdam, the Netherlands · Department of Epidemiology, Erasmus MC, Rotterdam, the Netherlands · Department of Electrical and Electronics Engineering, ENSEEIHT, Toulouse, France +3
Sep 13, 2026·Ali Abbasian Ardakani, Afshin Mohammadi, Taha Yusuf Kuzan +11Medical VLMsMammography
Department of Medicine, Faculty of Medicine and Dentistry, Danube Private University, Krems an der Donau, Austria · Department of Radiology, Faculty of Medicine, Urmia University of Medical Science, Urmia, Iran · Department of Radiology, Yeditepe University, Istanbul, Türkiye +10