GS-BrainText: A Multi-Site Brain Imaging Report Dataset from Generation Scotland for Clinical Natural Language Processing Development and Validation
Authors: Beatrice Alex, Claire Grover, Arlene Casey, Richard Tobin, Heather Whalley, William Whiteley
Organizations: Advanced Care Research Centre, University of Edinburgh, Edinburgh, UK · School of Literatures, Languages and Cultures, University of Edinburgh, UK · Edinburgh Futures Institute, University of Edinburgh, UK · School of Mathematical and Computer Sciences, Heriot-Watt University, UK · School of Informatics, University of Edinburgh, UK · Usher Institute, University of Edinburgh, UK · Institute for Neuroscience and Cardiovascular Research, University of Edinburgh, UK · Generation Scotland, Institute of Genetics and Cancer, University of Edinburgh, UK · NHS Lothian, Edinburgh, UK
We present GS-BrainText, a curated dataset of 8,511 brain radiology reports from the Generation Scotland cohort, of which 2,431 are annotated for 24 brain disease phenotypes. This multi-site dataset spans five Scottish NHS health boards and includes broad age representation (mean age 58, median age 53), making it uniquely valuable for developing and evaluating generalisable clinical natural language processing (NLP) algorithms and tools. Expert annotations were performed by a multidisciplinary clinical team using an annotation schema, with 10-100% double annotation per NHS health board and rigorous quality assurance. Benchmark evaluation using EdIE-R, an existing rule-based NLP system developed in conjunction with the annotation schema, revealed some performance variation across health boards (F1: 86.13-98.13), phenotypes (F1: 22.22-100) and age groups (F1: 87.01-98.13), highlighting critical challenges in generalisation of NLP tools. The GS-BrainText dataset addresses a significant gap in available UK clinical text resources and provides a valuable resource for the study of linguistic variation, diagnostic uncertainty expression and the impact of data characteristics on NLP system performance.
Figures & tables
Figure 1: A synthetic brain imaging report annotated with text- and document-level annotations.
Annotator pairs
Fife
GGC
Grampian
Lothian
Tayside
A1-A2
88.15 (108)
-
-
-
-
A1-A3
87.50 (108)
-
-
-
-
A1-A4
89.90 (108)
-
-
-
-
A2-A3
83.58 (108)
91.18 (53)
-
86.51 (143)
90.91 (35)
A2-A4
90.82 (108)
-
95.65 (26)
-
-
A3-A4
86.17 (108)
86.49 (53)
91.67 (24)
-
87.50 (30)
Table 1: Pairwise inter-annotator agreement (IAA) reported in F1 (and number of reports in parentheses) for phenotype annotations in GS-BrainText across five Scottish NHS health boards (GGC = Greater Glasgow and Clyde). All annotators annotated all the reports in Fife and two annotated all the reports in Lothian. IAA for other boards was computed on smaller subsets with two pairwise comparisons each.
Phenotype
Fife
GGC
Grampian
Lothian
Tayside
Total
Ischaemic_stroke_deep_recent
0
9
9
3
0
21
Ischaemic_stroke_deep_old
6
78
55
15
25
179
Ischaemic_stroke_cortical_recent
6
22
9
5
5
47
Ischaemic_stroke_cortical_old
6
45
24
5
8
88
Ischaemic_stroke_underspecified
4
17
11
0
14
46
Haemorrhagic_stroke_deep_recent
0
0
1
1
0
2
Table 2: Distribution of phenotype prevalence across five Scottish NHS health boards in the annotated subset of GS-BrainText (GGC = Greater Glasgow and Clyde).
Phenotype
Precision
Recall
F1
Ischaemic_stroke_deep_recent
72.73
76.19
74.42
Ischaemic_stroke_deep_old
84.58
94.97
89.47
Ischaemic_stroke_cortical_recent
93.33
89.36
91.30
Ischaemic_stroke_cortical_old
86.02
90.91
88.40
Ischaemic_stroke_underspecified
51.56
71.74
60.00
Haemorrhagic_stroke_deep_recent
66.67
100.00
80.00
Table 3: Precision, recall and F1 scores for the EdIE-R NLP system on GS-BrainText for individual phenotypes as well as micro- and macro-averaged performance scores.
Health Board
Precision
Recall
F1
Fife
98.13
98.13
98.13
GGC
82.18
92.45
87.01
Grampian
87.12
91.27
89.15
Lothian
94.44
98.35
96.36
Tayside
80.09
93.16
86.13
Table 4: EdIE-R performance in precision, recall and micro-average F1-scores for GS-BrainText per NHS health board.
Age Group
Precision
Recall
F1
<50
69.78
87.39
77.60
50-69
81.24
92.76
86.62
70+
89.78
93.70
91.70
Table 5: EdIE-R performance in precision, recall and micro-average F1-scores by age group.
AI4LIFE, Hanoi University of Science and Technology, Vietnam · SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris, France · Concordia International School Hanoi, Vietnam +1
Department of Radiology & Nuclear Medicine, Erasmus MC, Rotterdam, the Netherlands · Department of Epidemiology, Erasmus MC, Rotterdam, the Netherlands · Department of Electrical and Electronics Engineering, ENSEEIHT, Toulouse, France +3