cs.CLMar 24, 2026
SaveRadTimeline: Timeline Summarization for Longitudinal Radiological Lung Findings
Organizations: University of Washington
Abstract
Tracking findings in longitudinal radiology reports is crucial for accurately identifying disease progression, and the time-consuming process would benefit from automatic summarization. This work introduces a structured summarization task, where we frame longitudinal report summarization as a timeline generation task, with dated findings organized in columns and temporally related findings grouped in rows. This structured summarization format enables straightforward comparison of findings across time and facilitates fact-checking against the associated reports. The timeline is generated using a 3-step LLM process of extracting findings, generating group names, and using the names to group the findings. To evaluate such systems, we create RadTimeline, a timeline dataset focused on tracking lung-related radiologic findings in chest-related imaging reports. Experiments on RadTimeline show tradeoffs of different-sized LLMs and prompting strategies. Our results highlight that group name generation as an intermediate step is critical for effective finding grouping. The best configuration has some irrelevant findings but very good recall, and grouping performance is comparable to human annotators.
Figures & tables
Figure 1: The timeline task and a three-step LLM approach. Each column corresponds to a time-stamped radiology exam (e.g., YYYY-05_chest-ct). Each cell in a column is a piece of lung finding fact (e.g., "stable subsolid pulmonary nodule in the right upper lobe") from that report including clinically important details (e.g., "stable", "subsolid"). Each row groups temporally related findings, with a row header describing the distinguishing characteristics of that group (e.g., "right upper lobe ground glass nodule").
| Wrong | Partial | Correct | Irrelevant | |
|---|---|---|---|---|
| Wrong | 0 | 0 | 2 | 0 |
| Partial | 0 | 12 | 3 | 0 |
| Correct | 2 | 9 | 106 | 2 |
| Irrelevant | 1 | 0 | 2 | 16 |
Table 1: Confusion - matrix of factuality annotation between two annotators. Annotators agree on factuality ratings for 86% of 155 generated findings.
| Statistic | Ann-1 | Ann-2 |
|---|---|---|
| # Groups | 65 | 62 |
| Avg. / Max size | 2.1 / 8 | 2.2 / 7 |
| % Non-singleton | 52 | 50 |
| # Findings (Total # 136 ) | ||
| % Linked (Any / Cross) | 77 / 70 | 77 / 71 |
| # Findings ( Merg.) | 116 | 115 |
Table 2: Group annotation statistics from two annotators (Ann-1 and Ann-2) on findings in RadTimeline. The group size is based on the 136 findings. "Findings (Merg.)" are findings that are merged with same-report links for finding extraction evaluation. The % Linked refer to the percentage of findings that have links to any other finding (Any) vs. only cross-note findings (Cross) among the total 136 findings.
| Model | RL (Rpt.) | RL (Fdg.) | Miss. % | Unnec. % |
|---|---|---|---|---|
| Llama-3.1 | 67 | 62 | 3 | 40 |
| GPT-4o | 68 | 67 | 5 | 25 |
| Llama-3.1* | - | 70 | 3 | 32 |
| GPT-4o* | - | 75 | 6 | 15 |
Table 3: Finding extraction evaluation against gold findings. ROUGE-L (RL) scores are at the report (Rpt.) and finding (Fdg.) level. Missing (Miss.) represents unmatched-gold percentage, and unnecessary (Unnec.) represents unmatched prediction percentage. The first two rows use the weaker absent finding filter; * indicates to the more aggressive filter.
| Method | RL | Miss. % | Dup. % | Unnec. % |
|---|---|---|---|---|
| Llama 3.1 (Full) | 38 | 25 | 32 | 60 |
| Llama 3.1 (Fdg.) | 42 | 27 | 42 | 58 |
| GPT-4o (Full) | 48 | 30 | 28 | 37 |
| GPT-4o (Fdg.) | 52 | 13 | 31 | 46 |
| Oracle | 87 | 9 | 11 | 9 |
Table 4: Evaluation results for group name generation. The groups are generated using full or finding-only (Fdg.) contexts.
| Group names | Single ZS Llama | Single FS Llama | Multi. ZS Llama | Multi. FS Llama | Single ZS GPT | Single FS GPT | Multi. ZS GPT | Multi. FS GPT | Embed. Gen. | Embed. Spec. |
| Llama-3.1 | 66 | 74 | 77 | 78 | 77 | 73 | 78 | 77 | 68 | 70 |
| GPT-4o | 75 | 79 | 77 | 81 | 82 | 80 | 84 | 82 | 65 | 69 |
| Oracle | 73 | 78 | 86 | 84 | 80 | 84 | 85 | 83 | 78 | 83 |
| No groups | n/a | n/a | 72 | 68 | n/a | n/a | 70 | 72 | n/a | n/a |
Table 5: Group assignment performance of prompting and embedding-based methods with no context in CoNLL F1 scores. We use LLM models (Llama 3.1, GPT-4o) and an embedder model (e5-mistral-7b-instruct). "Multi." is short for Multiple prompting. "Embed. Gen." and "Embed. Spec." represent embedding-based methods using the general or task-specific instructions.
| Context | Single ZS Llama | Single FS Llama | Multi. ZS Llama | Multi. FS Llama | Single ZS GPT | Multi. ZS GPT |
|---|---|---|---|---|---|---|
| Ann1 | ||||||
| No Ctx. | 67 | 74 | 79 | 78 | 81 | 82 |
| Full Ctx. | 56 | 70 | 76 | 78 | 77 | 81 |
| Ann2 | ||||||
| No Ctx. | 65 | 75 | 75 | 77 | 82 | 85 |
| Full Ctx. | 60 | 76 | 82 | 78 | 86 | 90 |
Table 6: Impact of full contexts in group assignment prompting methods. Evaluated in CoNLL-F1 against gold group assignments from two annotators separately. Each configuration uses either Llama 3.1 for all steps or uses GPT-4o for all. All experiments use full contexts by default in group name generation. "Ctx." is short for "Context".
| Group names | Single ZS | Single FS | Multi. ZS | Multi. FS |
|---|---|---|---|---|
| Llama (Full) | 66 | 74 | 77 | 78 |
| Llama (Fdg.) | 66 | 70 | 71 | 68 |
| GPT (Full) | 82 | 80 | 84 | 82 |
| GPT (Fdg.) | 73 | 74 | 74 | 73 |
Table 7: Impact of input contexts for group name generation on the final group assignment performance in CoNLL-F1. All experiments use the same LLM in all steps and no contexts in group assignment. Group names are generated using full or finding-only ("Fdg.") contexts.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Task 1: Cell-level Factuality Rating |
| • For all cells in the column, rate the factuality level [0 bad, 1 partially correct, 2 good] for each cell based on the consistency with the column-associated report. • Annotators should focus only on finding-related facts written in the specific report being evaluated. Do not refer to facts from other reports. • Correctness of those facts should NOT affect the rating: – Suggestive conditions * e.g., “considering neoplasm given the persistent nodules” * e.g., “likely due to infection” – Facts that are not lung-related or not radiological findings * e.g., “Lung RAD rating is 3A” * e.g., “Coronary Artery Calcifications” • Skip ‘not-mentioned’ cells. Blank cells are equivalent to ‘not mentioned’. Instead of annotating them, add them in Task 2 for missing findings if actually mentioned. • Label the cell as not containing lung-related findings as [-1: non-lung-findings] . • Add a note column next to the factuality cell. |
Table 8: Annotation guidelines for factuality rating
| Task 2: Group Assignments |
|---|
| Assign a group label for each cell. The group labels to choose are the headers in the finding_group column. You may add a new group label if no existing group labels can distinguish this cell from other groups. |
| What should be included in a group • The same finding in the notes, even if described differently, or • Findings from different notes that can change to one another. This may lead to trend descriptions such as: • new • increasing • resolving |
| Duplicate labels If two group labels are duplicated, choose one label and use it consistently for all cells belonging to that group. |
| Goal: |
| Ensure that the same findings or time-related findings are placed in the same group. |
| Skip condition: |
Table 9: Annotation guidelines for group assignments
| Task 3: Group Name Revision |
|---|
| Given a group of dated findings, add the group name for this group. The goal is to make the group name clearly distinguishable from other groups while remaining concise and containing critical information that a radiologist might use when referring to the finding. |
| Strategy: |
| The group name should: 1. Include the facts shared by all members in the group. When paraphrasing occurs, choose the description used more frequently. 2. Omit details that can change in the future (e.g., size, change trend, severity). 3. Use extra details if needed to distinguish similar groups. |
Table 10: Annotation guidelines for group name revision given formed groups.
Figure 2: A group assignment example provided to annotators.
| Prompt for step 1 finding extraction |
|---|
| Q: [ Single_Radiology_Report_Text , e.g. EXAMINATION: CT of the chest without contrast - low dose lung cancer screening….. ] : |
| Please list all lung-related findings (including pleura-ralated) from the exam report. Please use a python list to include the strings of those lung desription. Each string should include all details related to higher or lower risk, using the same description in the note , especially finding’s location, size, trend of change, and shape, density, or other characteristics. |
| Good example: [“stable 2 mm left upper lobe calcified nodule”, ’left lower lobe opacities’] |
| Now please anwswer: |
| lung_finding_list = |
| A: Here is the list of lung-related findings from the exam report: “‘python lung_finding_list = [ "2.6 mm noncalcified nodule in the anterior segment of the left upper lobe", "alveolar groundglass opacity in the posterior segment of the right upper lobe with a small central nodular component measuring 10.2 mm", … |
Table 11: Step 1 finding extraction input. The prompt includes the single report input, an one-shot example, and instructions requiring a python list output format.
| Prompt for step 2 group name generation |
|---|
| Q: [ Longitudinal_Textual_Reports , e.g. date,note_tag,QOClassificationName,ReportText YYYY-MM-DD,chest_ct,CT Lung Low Dose Screening,"EXAMINATION: CT of the chest without contrast - low dose lung cancer screening … ] : |
| Please group the lung-related findings recorded over time (including pleura-related findings) and list the group names. |
| Each group should contain findings that are essentially the same, even if described differently, as well as findings that evolve into one another over time. |
| Different findings or findings will not evolve into one another should be in separate groups |
| Group names should be specific that separate unrelated findings, avoid generic names. |
| Good group name: right upper lobe ground glass nodule |
Table 12: Step 2 group name generation prompt that requires list group names at the end of response. The prompt includes a good and a bad example for individual group names, but no examples for the list format.
| Prompt for step 3 group assignment for single finding |
|---|
| Q: [ Longitudinal_Textual_Reports , e.g. date,note_tag,QOClassificationName,ReportText YYYY-MM-DD,chest_ct,CT Lung Low Dose Screening,"EXAMINATION: CT of the chest without contrast - low dose lung cancer screening … ] |
| Please find the group best describing this finding given the radiology exam history and group labels. There are groups presented to track the lung-related findings recorded over time. |
| Each group should contain findings that are essentially the same, even if described differently, as well as findings that evolve into one another over time. |
| Different findings or findings will not evolve into one another should be in separate groups |
| [Zero-Shot Content] |
| The group labels are: |
Table 13: Step 3 Group Assignment for Single Finding. The prompts include underlined texts only when using full-contexts. We have both zero-shot and few-shot instructions, few-shot version adds two examples and further emphasize on the format requirements.
| Prompt for step 3 group assignment (multi-finding tag format, no group names given) |
|---|
| Q: Task: Your task is to group radiology findings evolving over time from multiple reports of the same patient. |
| Please replace every <#> with a group name tag in angle-bracket format, such as <10_mm_right_lower_lobe_nodule>. |
| You should invent your own group tags so that each tag is representative for its associated group. |
| Use the same group name tag across all reports for all mentions related to the same underlying entity. |
| Your Task: |
| Radiology Findings: |
Table 14: Step 3 Group Assignment for Multi-Finding when no group names are given. The prompt includes an example of single tag, but no examples for the final output for all provided findings.
| Prompt for step 3 group assignment for multi finding |
|---|
| Q: [ Longitudinal_Textual_Reports , e.g. date,note_tag,QOClassificationName,ReportText YYYY-MM-DD,chest_ct,CT Lung Low Dose Screening,"EXAMINATION: CT of the chest without contrast - low dose lung cancer screening … ] |
| Task: Your task is to group radiology findings evolving over time from multiple reports of the same patient. Those reports are as given above. |
| Please replace every <#> with a group name tag in angle-bracket format, such as <10_mm_right_lower_lobe_nodule>. |
| You can use either an existing group tag as provided, or invent your own. |
| Use the same group name tag across all reports for all mentions related to the same underlying entity. |
| [Few-Shot Content] |
Table 15: Step 3 Group Assignment for Multi-Finding when group names are given. The prompts include underlined texts only when using full-contexts. We have both zero-shot and few-shot instructions, few-shot version adds two examples.
| Gold | GPT-4o | Llama3.1 |
| Moderate centrilobular emphysema | Moderate centrilobular emphysema | Moderate centrilobular emphysema |
| stable lung nodules since [DATE], therefore benign. No new or enlarging pulmonary nodule | Stable lung nodules since [DATE], therefore benign | stable lung nodules since [DATE], therefore benign. No new or enlarging pulmonary nodule |
| No new or enlarging pulmonary nodule | Normal supraclavicular region | |
| Normal mediastinum | ||
| Normal pleura | ||
| Mildly dilated main pulmonary artery measuring 3.3 cm in caliber, also unchanged |
Table 16: Step 1 finding extraction examples from GPT-4o and Llama 3.1 8B Instruct. Llama 3.1 tends to generate more findings, and to include content irrelevant to lung findings. Human references are provided as the gold labels. The predicted findings are presented in the same order as they appear in the LLM responses.
| Gold (group assignments from Ann1) | Gold (group assignments from Ann2) | GPT-4o | Llama3.1 |
|---|---|---|---|
| Hyperinflated lungs | Centrilobular emphysema in upper lobe predominant distribution | Left upper lobe spiculated nodule | Centrilobular emphysema in upper lobe predominant distribution |
| Flattening of domes of diaphragm | Focal area of scarring | Centrilobular emphysema | Spiculated nodule in left upper lobe (concerning for primary lung malignancy) |
| Centrilobular emphysema in upper lobe predominant distribution | Spiculated nodule in left upper lobe (concerning for primary lung malignancy) | Pleura and pleural spaces | Unchanged spiculated left upper lobe nodule (6 x 9 mm) |
| Left anterior lingula scarring | Right upper lobe nodule | Pericardial effusion | Unchanged right upper lobe nodule (2 mm) |
| Spiculated nodule in left upper lobe (concerning for primary lung malignancy) | Ascending aorta abnormalities | No new pulmonary nodules | |
| Right upper lobe nodule | Pleural effusions or pneumothorax (no findings) |
Table 17: Step 2 group name generation examples from GPT-4o and Llama 3.1 8B Instruct when using full longitudinal reports as the context. Gold group names are provided in two versions, each reflecting the grouping assignments of Ann1 and Ann2 respectively, with names refined by a single third annotator across both versions. Similarly to step 1 results, Llama 3.1 tends to generate more group names and to include content irrelevant to lung findings. Two versions of human references are provided as the gold labels. The predicted group names are presented in the same order as they appear in the LLM responses.
| GPT-4o (Full) | GPT-4o (List) | Llama3.1 (Full) | Llama3.1 (List) |
|---|---|---|---|
| Left upper lobe spiculated nodule | Hyperinflated lungs | Centrilobular emphysema in upper lobe predominant distribution | Hyperinflated lungs and diaphragm flattening |
| Centrilobular emphysema | Flattening of the domes of the diaphragm | Spiculated nodule in left upper lobe (concerning for primary lung malignancy) | Centrilobular emphysema |
| Pleura and pleural spaces | Centrilobular emphysema | Unchanged spiculated left upper lobe nodule (6 x 9 mm) | Spiculated nodule in left upper lobe (primary lung malignancy concern) |
| Pericardial effusion | Spiculated nodule in the left upper lobe | Unchanged right upper lobe nodule (2 mm) | Small focal area of scarring in lingula |
| Ascending aorta abnormalities | Right upper lobe pulmonary nodule | No new pulmonary nodules | Clear trachea and central airways |
| Small focal area of scarring within the anterior aspect of the lingula | Pleural effusions or pneumothorax (no findings) | Pleural effusions or pneumothorax (absence of) |
Table 18: Comparison of step 2 group name generation examples using full longitudinal reports versus predicted finding lists as context. Examples are from GPT-4o and Llama 3.1 8B Instruct. The predicted group names are presented in the same order as they appear in the LLM responses.
More in this topic
Text Summarization
Text summarization develops methods for condensing documents while preserving the information, perspective, and evidence readers need.
See all 142 papers