A Semi-Automated Annotation Workflow for Paediatric Histopathology Reports Using Small Language Models
Authors: Avish Vijayaraghavan, Jaskaran Singh Kawatra, Sebin Sabu, Jonny Sheldon, Will Poulett, Alex Eze, Daniel Key, John Booth, Shiren Patel, Jonny Pearson, Dan Schofield, Jonathan Hope, Pavithra Rajendran, Neil Sebire
Organizations: Division of Systems Medicine, Department of Metabolism, Digestion and Reproduction, Imperial College London, London, UK. · Department of Computing, Imperial College London, London, UK. · Data Science Team, NHS England, Leeds & London, UK. · DRE Team, GOSH DRIVE, Great Ormond Street Hospital, London, UK. · Population, Policy & Practice Department, University College London, London, UK.
Electronic Patient Record (EPR) systems contain valuable clinical information, but much of it is trapped in unstructured text, limiting its use for research and decision-making. Large language models can extract such information but require substantial computational resources to run locally, and sending sensitive clinical data to cloud-based services, even when deidentified, raises significant patient privacy concerns. In this study, we develop a resource-efficient semi-automated annotation workflow using small language models (SLMs) to extract structured information from unstructured EPR data, focusing on paediatric histopathology reports. As a proof-of-concept, we apply the workflow to paediatric renal biopsy reports, a domain chosen for its constrained diagnostic scope and well-defined underlying biology. We develop the workflow iteratively with clinical oversight across three meetings, manually annotating 400 reports from a dataset of 2,111 at Great Ormond Street Hospital as a gold standard, while developing an automated information extraction approach using SLMs. We frame extraction as a Question-Answering task grounded by clinician-guided entity guidelines and few-shot examples, evaluating five instruction-tuned SLMs with a disagreement modelling framework to prioritise reports for clinical review. Gemma 2 2B achieves the highest accuracy at 84.3%, outperforming off-the-shelf models including spaCy (74.3%), BioBERT-SQuAD (62.3%), RoBERTa-SQuAD (59.7%), and GLiNER (60.2%). Entity guidelines improved performance by 7-19% over the zero-shot baseline, and few-shot examples by 6-38%, though their benefits do not compound when combined. These results demonstrate that SLMs can extract structured information from specialised clinical domains on CPU-only infrastructure with minimal clinician involvement. Our code is available at https://github.com/gosh-dre/nlp_renal_biopsy.
Figures & tables
Figure 1: The general histopathology workflow. Our focus is on extracting key text spans from the clinician’s report that are relevant for diagnosis.
Figure 2: Annotation workflow for medical reports which three expert touchpoints highlighted with thick blue lines . The second and third touchpoints have potential to be indefinitely long and so we have specified the assumption of minimal expert time.
Entity Name
Data Type
cortex
Binary
medulla
Binary
number of glomeruli
Numerical
number of sclerosed glomeruli
Numerical
% chronic change/fibrosis
Numerical
glomeruli normal or abnormal
Binary
Table 1: Initial entities and their data types from the consultant clinician.
Figure 3: Example entity extraction prompt for two entities.
Figure 4: Development of final LM-as-a-Judge (LAAJ) query.
Prompt Format → Model Name ↓
Zero-shot without guidelines
Two-shot without guidelines
Zero-shot with guidelines
Two-shot with guidelines
Qwen-2.5
67.7% {}^{{\color[rgb]{0.84,0,0.25}\text{*1}}}
73.4% {}^{{\color[rgb]{0.84,0,0.25}\text{*8}}}
77.1% {}^{{\color[rgb]{0.84,0,0.25}\text{*1}}}
73.3% {}^{{\color[rgb]{0.84,0,0.25}\text{*4}}}
1.5B, FP16
30.9s/report
26.2s/report
27.4s/report
26.6s/report
Gemma 2
75.9% {}^{{\color[rgb]{0.84,0,0.25}\text{*1}}}
84.3% {}^{{\color[rgb]{0.84,0,0.25}\text{*1}}}
83.3% {}^{{\color[rgb]{0.84,0,0.25}\text{*1}}}
84.3% {}^{{\color[rgb]{0.84,0,0.25}\text{*2}}}
2B, FP16
35.6s/report
37.3s/report
37.9s/report
36.6s/report
Llama 3.2
45.5% {}^{{\color[rgb]{0.84,0,0.25}\text{*210}}}
83.4% {}^{{\color[rgb]{0.84,0,0.25}\text{*1}}}
64.1% {}^{{\color[rgb]{0.84,0,0.25}\text{*109}}}
83.4% {}^{{\color[rgb]{0.84,0,0.25}\text{*1}}}
3B, Q8
34.8s/report
32.5s/report
33.5s/report
35.2s/report
Table 3: Table comparing accuracy and report iteration times for different models with zero-shot and two-shot prompts over 400 annotated reports. *x signifies the number of reports with a parsing error when converting the SLM’s generated string to a JSON.
Model → Entity ↓
Qwen1.5
Gemma2
Llama3.2
Phi3.5 Q4
Phi3.5 Q8
spaCy
BioBERT
RoBERTa
GLiNER
NuExtract
cortex_present
364/400
384/400
383/400
375/400
378/400
387/400
384/400
383/400
221/400
17/20
medulla_present
300/400
325/400
341/400
303/400
334/400
389/400
332/400
271/400
346/400
18/20
n_total
385/400
379/400
389/400
377/400
378/400
217/400
345/400
295/400
342/400
19/20
n_segmental
339/400
329/400
299/400
336/400
357/400
360/400
160/400
253/400
340/400
17/20
n_global
301/400
350/400
349/400
357/400
369/400
275/400
145/400
282/400
256/400
18/20
abnormal_glomeruli
244/400
299/400
291/400
295/400
280/400
206/400
207/400
209/400
210/400
14/20
Table 4: Combined Entity Extraction Performance Comparison. Left columns show SLM models with their best-performing configurations. Right columns show alternative approaches. NuExtract results are highlighted orange as they are incomplete (only tested on 20/400 reports due to time constraints). BioBERT = BioBERT-SQuAD, RoBERTa = RoBERTa-SQuAD.
Figure 5: Heatmap showing disagreements between top-performing SLM models (Gemma 2 (Model 1) and Llama 3.2 (Model 2), both with two-shot and guidelines configuration) compared to the ground truth. All_Agree_Correct = both predictions match the ground truth, All_Disagree = both predictions are wrong and differ from each other, Both_Models_Wrong_Same = both predictions are wrong but agree with each other, ModelX_Correct_ModelY_Wrong = model X matches the ground truth but model Y does not.
Figure 1: Streamlit app used for QA annotation of reports, displaying three of the nine total entities: cortex presence , medulla presence , and number of total glomeruli . Sensitive patient information in the report has been redacted.
Figure 2: Example report in Streamlit comparison app showing the predictions of two models for cortex presence and medulla presence entities along with the comment box for qualitative error analysis.
Question To Ask
Entity Type
Entity Guidelines
Entity Code
Combined Prompt Question
Combined Guidelines
Initial questions you get from clinicians and refine from reading reports. This is displayed in Streamlit app.
Types in order of complexity: boolean, categorical, numerical, string-simple, string-complex. Can also be combinations e.g. numerical/string-simple.
Initial guidelines per entity which you get from reading the reports. Note: use ” not ’ to avoid Streamlit markdown issues.
How entity shown in JSON
Certain entities might make sense to group together in final prompt. Add combined questions in here and merge cells.
Shortened guidelines that link individual entity guidelines together for the combined prompt questions.
Is the cortex present?
binary
Cortex is the inside of the kidney. If neither medulla or cortex is mentioned, assume the cortex is there and answer True.
cortex_ present
Are the medulla and cortex present?
If they are mentioned include them unless explicitly says they’re missing. Lack of mention of either implies absence but lack of mention of both implies just cortex.
Is the medulla present?
binary
Medulla is outside of kidney. (The medulla is covered by the capsule.) If it is not mentioned, assume it is not there and answer False.
medulla_ present
Table 1: Example of first few rows of guidelines.xlsx file showing the questions that practitioners fill out - first row is the column type, the second row contains descriptions of the column type, and all following rows correspond to our entity information.
Figure 3: Boxplot showing lengths of each section of the report.
Figure 4: Top 30 words in “MICROSCOPY” section across reports.
Figure 5: Histograms of number of words (left) and characters (right) for the microscopy (top) and conclusion (bottom) sections. There are fewer reports here than in Figures 3 and 4 since not all reports with “MICROSCOPY” sections had “CONCLUSION” sections.
Figure 6: Development of final LM-as-a-Judge (LAAJ) query.
Figure 7: Results from Qwen FP16 model across different LAAJ queries. For each plot, from left to right: exact, same_concept, similar_enough, different.
Figure 8: Results from Gemma2 2B FP16 model across different LAAJ queries. For each plot, from left to right: exact, same_concept, similar_enough, different.
Figure 9: Development of final LM-as-a-Judge (LAAJ) query.
Figure 10: Results from Llama 3.2 3B FP16 across different LAAJ queries. For each plot, from left to right: exact, same_concept, similar_enough, different. (Continued below…)
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA · Division of Pulmonary, Critical Care and Sleep Medicine, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA · College of Nursing, Florida State University, Tallahassee, FL, USA
McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX 77030, United States · School of Public Health, The University of Texas Health Science Center at Houston, Houston, TX 77030, United States · School of Dentistry, The University of Texas Health Science Center at Houston, Houston, TX 77054, United States +1