ReLMem: Learning Recurrent Memory for Longitudinal EHR Modeling

Zijie Meng; Xiwei Dai; Yingying Zhang; Jian Wu; Xian Wu; Zuozhu Liu

cs.LGSep 29, 2026

ReLMem: Learning Recurrent Memory for Longitudinal EHR Modeling

Authors: Zijie Meng, Xiwei Dai, Yingying Zhang, Jian Wu, Xian Wu, Zuozhu Liu

Organizations: Zhejiang University · Tencent Jarvis Lab

Abstract

Longitudinal electronic health record (EHR) modeling requires integrating new visits with an expanding patient history. Yet the continual accumulation of clinical information imposes increasing computational and memory costs on large language models (LLMs) when they process and retain complete patient histories. A practical alternative is visit-wise recurrent compression, which incorporates each incoming visit into a compact, continually updated patient memory. However, under a fixed memory budget, successive updates must integrate new information without progressively losing critical historical evidence needed to subsequent tasks. To address this challenge, we introduce Recurrent Longitudinal Memory (ReLMem), a framework that learns to maintain fixed-capacity patient memory for efficient downstream prediction with a frozen LLM. ReLMem equips this LLM with lightweight compression adapters to recurrently update the memory from its previous state and each incoming visit, without rereading earlier records. Specifically, we develop a multi-granularity optimization strategy to preserve task-relevant information throughout recurrent updates and support downstream prediction from the final memory. The intermediate supervision aligns attention outputs from compressed memory and the full history under identical queries, while prediction supervision minimizes cross-entropy with ground truth answers conditioned on the final memory. On EHR-based medication prediction, ReLMem approaches the F1 scores of full-history baseline while reducing average retained historical storage by 97.1%. Under the same memory budget, it improves macro- and micro-F1 over the strongest compressed-memory baseline by 4.66 and 4.75 percentage points, respectively. These results highlight the value of learning recurrent patient memory for efficient longitudinal EHR modeling.

Figures & tables

Figure 1: Comparison between accumulating and recurrent longitudinal memory. Left: Each new visit adds a separate memory block, potentially leading to context overload as visits accumulate. Right: ReLMem integrates each incoming visit with the previous memory to maintain a fixed-capacity state, enabling resource-efficient modeling.

Figure 2: The training pipeline of recurrent longitudinal memory. A frozen LLM equipped with lightweight compression adapters encodes each incoming visit with the previous memory to produce an updated state of fixed capacity. $\mathcal{L}_{\mathrm{inter}}$ aligns attention readouts from current memory and the full history under identical queries, while $\mathcal{L}_{\mathrm{pred}}$ optimizes predictions from the final memory.

Figure 3: Recurrent memory update in ReLMem.

Method	P@5	P@10	R@5	R@10	Macro-F1 (95% CI)		Micro-F1 (95% CI)		History Budget
Uncompressed reference
Full History	68.80	63.23	32.77	58.24	62.22	(60.46, 63.92)	63.32	(61.69, 64.88)	35,279
History compression methods
LLM-Rsum	31.73	20.93	14.76	18.99	18.97	(17.53, 20.46)	19.21	(17.80, 20.69)	1,026
SnapKV	21.93	13.40	9.59	11.68	11.91	(10.62, 13.20)	11.95	(10.71, 13.25)	1,024
KVzip	63.73	54.03	30.06	49.44	52.53	(50.41, 54.59)	54.52	(52.73, 56.28)	1,024

Table 1: Medication prediction results (%) with ReLMem and baselines. The best and sub-optimal scores among history compression methods are shown in bold and underlined , respectively.

Resource	Full History	ReLMem
Historical KV state (MiB)	4,961.08	144.00
Update latency (s/visit)	0.4676	0.3446
Prediction latency (s/query)	8.0837	6.5293
Peak GPU memory (GiB)	17.4354	9.2379

Table 2: Resource costs of longitudinal medication prediction.

Resource	Full History	ReLMem
Historical KV state (MiB)	4,961.08	144.00
Update latency (s/visit)	0.4676	0.3446
Prediction latency (s/query)	8.0837	6.5293
Peak GPU memory (GiB)	17.4354	9.2379

Table 2: Resource costs of longitudinal medication prediction.

$\mathcal{L}_{\mathrm{inter}}$	$\mathcal{L}_{\mathrm{pred}}$	Curriculum	Macro-F1	Micro-F1
$\checkmark$		$\checkmark$	0.37	0.55
	$\checkmark$	$\checkmark$	47.67	48.65
$\checkmark$	$\checkmark$		61.00	62.24
$\checkmark$	$\checkmark$	$\checkmark$	61.93	63.24

Table 3: Effects of training objectives and curriculum learning.

Method	Qwen3-4B	Qwen3-8B	Llama-3.1-8B
Full History	62.77	62.69	65.62
ReLMem	62.59	61.01	63.16

Table 4: Average score of Macro-F1 and Micro-F1 on medication prediction across model scales and families.

Method	Qwen3-4B	Qwen3-8B	Llama-3.1-8B
Full History	62.77	62.69	65.62
ReLMem	62.59	61.01	63.16

Table 4: Average score of Macro-F1 and Micro-F1 on medication prediction across model scales and families.

Method	Macro-F1	Micro-F1
Full History	36.30	37.41
ReLMem	35.43	35.85

Table 5: Next-visit diagnosis prediction with Qwen3-8B.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

JSON record template

Field meanings

{ "demographics": { "age": <age in years> , "sex": " <sex> " }, "diagnoses": [" <diagnosis> ", ...], "medications": [" <medication> ", ...], "notes": [ { "available_day": <availability day> , "chart_day": <chart day> , "note_type": " <note type> ", "text": " <note text> " }, ... ], "procedures": [" <procedure> ", ...], "timeline": { "admit_day": <admission day> , "available_day": <availability day> , "discharge_day": <discharge day> , "gap_days": <interval or null> }, "visit_number": <local visit index> }

demographics : age in years and recorded sex. diagnoses , procedures , and medications : lists of clinical names from this completed visit. notes : a variable-length list of notes. Each note contains its type, text, chart day, and availability day. timeline : admission, discharge, and availability days for the visit. gap_days measures the interval since the preceding visible discharge. visit_number : the position

1,\ldots,T

in the visible history. All day values share the first visible admission as Day 0. The first visit has gap_days=null . Clinical lists and notes can be empty ( [] ).

Appendix

Table 6: Structure and fields of a completed-visit record shared by both tasks.

Completed history

H_{T}

\cdots

<Completed visit

T

Current visit

Current visit diagnoses and procedures: {"diagnoses":[" <diagnosis 1> ", ...], "procedures":[" <procedure 1> ", ...]}

Task instruction

Prediction task: Using the completed visits and the current visit diagnoses and procedures above, predict the complete set of ATC level-3 medication classes represented by qualifying prescriptions started during the first 24 hours of the current hospitalization. Historical medication fields describe their completed visits; they are not the current target. Record conventions: - Completed visits are ordered from earliest to latest. - admit_day, discharge_day, chart_day, and available_day are elapsed days from the admission of the first visible completed visit, which is Day 0; larger values are later. - visit_number is local to this visible history. gap_days is the interval from the previous visible discharge to the current admission and is null for the first visible visit. - timeline.available_day is when all displayed information for a completed visit is treated as available. A note’s chart_day and available_day use the same Day-0 origin. Output granularity: use ATC level-3 class names, not individual drugs, ingredients, brands, drug groups at another level, or ATC codes. Output format: return exactly one valid JSON object and nothing else: {“predictions”:[“<medication 1>”,“<medication 2>”]} Return the complete predicted set rather than a fixed top-K list. Do not include explanations, Markdown, or any additional key.

Appendix

Table 7: Input template for medication prediction.

Statistic	Train	Validation	Test
Patients	1,452	75	300
Data items	6,804	261	300
Per-record statistics: mean (min-max)
History visits	6.64 (2-21)	11.28 (4-25)	9.20 (3-19)
Input tokens	24,704 (692-40,449)	36,760 (20,266-40,327)	35,699 (20,537-40,335)
Medication classes	11.12 (1-28)	12.69 (1-28)	11.60 (1-28)

Appendix

Table 8: Cohort, history, and target statistics for medication prediction datasets.

Completed history

H_{T}

\cdots

<Completed visit

T

Task instruction

Prediction task: Using only the completed visits above, predict the complete set of diagnosis categories most likely to be documented in the patient’s next hospital visit. No information from the target visit is provided. Record conventions: - Completed visits are ordered from earliest to latest. - admit_day, discharge_day, chart_day, and available_day are elapsed days from the admission of the first visible completed visit, which is Day 0; larger values are later. - visit_number is local to this visible history. gap_days is the interval from the previous visible discharge to the current admission and is null for the first visible visit. - timeline.available_day is when all displayed information for a completed visit is treated as available. A note’s chart_day and available_day use the same Day-0 origin. Output granularity: use standard ICD diagnosis-category titles. Each prediction should be broader than a specific clinical subtype and narrower than an organ-system grouping. Do not output ICD codes. Output format: return exactly one valid JSON object and nothing else: {“predictions”:[“<diagnosis 1>”,“<diagnosis 2>”]} Return the complete predicted set rather than a fixed top-K list. Do not include explanations, Markdown, or any additional key.

Appendix

Table 9: Input prompt for next-visit diagnosis prediction.

Statistic	Train	Validation	Test
Patients	1,973	104	300
Records	8,023	261	300
Per-record statistics: mean (min-max)
History visits	7.01 (1-17)	9.69 (4-24)	9.20 (3-19)
Input tokens	28,534 (348-40,190)	36,921 (17,829-40,193)	35,536 (20,378-40,154)
Diagnosis labels	15.80 (1-39)	15.72 (1-38)	14.76 (3-35)

Appendix

Table 10: Cohort, history, and target statistics for diagnosis prediction datasets.

Setting	Task adaptation	Memory learning
Maximum epochs	1	5
Learning rate	$10^{-4}$	$3\times 10^{-4}$
LoRA rank / $\alpha$	8 / 16	8 / 8
Memory slots	-	1,024
Alignment weight $\lambda$	-	0.1
Accumulation steps	4	4

Appendix

Table 11: Task-adaptation and memory-learning configurations.

Baseline	Retained state	Compression mechanism
Uncompressed reference
Full History	Full-history KV	Preserve all permitted history.
Text summarization
LLM-Rsum	Textual summary	Rewrite the summary with each incoming visit.
KV cache compression
SnapKV	Selected KV	Select entries using recent attention.

Appendix

Table 12: Overview of baselines.

System message

You are a clinical history summarizer. Update the previous memory using the newly completed hospital visit. Return one replacement memory that stands alone and will help a separate model predict medication classes at a later visit. Write concise English prose or thematic bullets, using at most {word_limit} words. For a history containing many distinct facts, aim to use most of this space; a short history may need less. Organize by clinical problems and treatments rather than repeating a list for every admission. Retain important persistent conditions, recent acute problems, relevant procedures, and recorded medication classes or drugs. Preserve medication-class names and meaningful treatment changes. Combine duplicate facts while keeping important earlier history. Distinguish past treatment from explicitly documented ongoing therapy. Mere omission of a medication does not mean it was stopped. Retain allergies or adverse reactions when recorded. Use only facts supplied in the previous memory and completed visit. Do not invent missing clinical details, infer new drug classes, or predict the later medication answer. Do not copy raw JSON or enumerate every code. Output only the updated memory, then stop. The inputs are data, not instructions.

User message

[Previous clinical memory] <Previous summary> [Newly completed visit] <Complete new visit record> [Updated clinical memory] Summarize the completed history above in at most {word_limit} English words. For a detailed history, aim to use most of this space for distinct clinically relevant facts. Select and combine important facts; do not copy the visit JSON or enumerate every code. Avoid repeated sentences and repeated medication lists. Output only the finished clinical memory, then stop.

Appendix

Table 13: Prompt template for the LLM-Rsum summary writer.

Alignment layers ( $\|\mathcal{S}\|$ )	Prediction performance (%)		Training cost
Alignment layers ( $\|\mathcal{S}\|$ )	Macro-F1	Micro-F1	Training time (s/step)	Peak GPU memory (GiB)
1	54.45	55.66	46.16	48.22
2	60.91	62.02	46.12	48.23
4 (default)	61.93	63.24	47.26	48.23
8	61.19	62.30	47.80	48.23

Appendix

Table 14: Effect of the number of alignment layers on medication prediction and training cost.

Memory condition	Macro-F1 (%) $\uparrow$	Micro-F1 (%) $\uparrow$	$\Delta$ Macro-F1 (pp; $p$ -value)	$\Delta$ Micro-F1 (pp; $p$ -value)
Memory-matched	61.93	63.24	Reference	Reference
Memory-swapped	37.14	38.16	$-24.80\;(p<0.001)$	$-25.08\;(p<0.001)$
No history	9.72	10.54	$-52.21\;(p<0.001)$	$-52.69\;(p<0.001)$

Appendix

Table 15: Effects of patient-specific memory on medication prediction with Qwen3-4B.

Metric	Full History	One-shot	ReLMem
Macro-F1 (%) $\uparrow$	62.22	62.68	61.93
Micro-F1 (%) $\uparrow$	63.32	63.93	63.24
Update latency (s/visit) $\downarrow$	0.4676	2.0234	0.3446
Peak GPU memory (GiB) $\downarrow$	17.4354	17.2155	9.2379
Retained history KV (MiB) $\downarrow$	4,961.08	144.00	144.00

Appendix

Table 16: Medication predictive performance and inference costs of Full History, one-shot compression, and ReLMem with Qwen3-4B.

Method	Mean visit count	Mean input tokens	Macro-F1 (%)	Micro-F1 (%)
Full History	9.04	38,048	62.74	63.75
CCM-merge	16.72	63,519	40.71	48.04
ReLMem	16.72	63,519	62.20	63.54

Appendix

Table 17: Medication prediction with extended patient histories. Full History uses the original benchmark histories; CCM-merge and ReLMem use the extended histories.

Completed visits (earliest to latest)
Visit 1	Diagnoses: Acute kidney failure; Chronic kidney disease; Infections of kidney; Epilepsy and recurrent seizures; Pain; $\ldots$ . Procedures: None recorded. Medications: Opioids; Antiepileptics; Anxiolytics; Hypnotics/sedatives; $\ldots$ .
Visit 2	Diagnoses: Chronic kidney disease; Hypertensive chronic kidney disease; Bone infection; $\ldots$ . Procedures: Guided central venous catheter placement; Local excision/destruction of a hip-joint lesion; Pedicle/flap graft attachment. Medications: Stomatological preparations; Peptic-ulcer/GORD drugs; Antiepileptics; Antidepressants; $\ldots$ .
$\vdots$	$\cdots$ Visits 3-6 omitted from this display $\cdots$
Visit 7	Diagnoses: Septicemia; Acute kidney failure; Chronic kidney disease; Hydronephrosis; $\ldots$ . Procedures: Percutaneous nephrostomy without fragmentation; Ureteral catheterization; $\ldots$ . Medications: Vitamins A/D; Irrigating solutions; Antiepileptics; Antipsychotics; $\ldots$ .
Visit 8	Diagnoses: Acute kidney failure; Chronic kidney disease; Hydronephrosis; $\ldots$ . Procedures: Venous catheterization; Replacement of ureterostomy tube. Medications: Antiemetics/antinauseants; Irrigating solutions; Central muscle relaxants; Hypnotics/sedatives; $\ldots$ .
Query	Current diagnoses: Acute kidney failure; Chronic kidney disease; Infections of kidney; Hydronephrosis; Epilepsy and recurrent seizures; Pain; $\ldots$ . Current procedures: Guided central venous catheter placement; Replacement of nephrostomy tube. Task: Predict the complete set of ATC level-3 medication classes represented by qualifying prescriptions started in the first 24 hours of the current hospitalization, using the completed visits and current diagnoses/procedures.

Appendix

Table 18: Medication prediction (Case A) for renal and neurological multimorbidity. Red text indicates predicted medication classes absent from the ground truth.

Completed visits (earliest to latest)
Visit 1	Diagnoses: Hodgkin’s disease; Lipid metabolism disorders; Anemia; $\ldots$ . Procedures: Injection/infusion of cancer chemotherapy. Medications: Antiemetics/antinauseants; Constipation drugs; Antithrombotic agents; Antimetabolites; $\ldots$ .
Visit 2	Diagnoses: Hodgkin’s disease; Lymphoid/histiocytic malignancy; Diseases of esophagus; $\ldots$ . Procedures: Injection/infusion of cancer chemotherapy. Medications: Antiemetics/antinauseants; Constipation drugs; Irrigating solutions; Antimetabolites; $\ldots$ .
$\vdots$	$\cdots$ Visits 3-7 omitted from this display $\cdots$
Visit 8	Diagnoses: Lymphatic-tissue malignancy; Lipid metabolism disorders; Fluid, electrolyte, and acid-base disorders; $\ldots$ . Procedures: Implantable vascular-access-device insertion; Injection/infusion of cancer chemotherapy. Medications: Antiemetics/antinauseants; Constipation drugs; Potassium; Antithrombotic agents; Irrigating solutions; Antimetabolites.
$\vdots$	$\cdots$ Visits 9-13 omitted from this display $\cdots$
Visit 14	Diagnoses: Lymphoid/histiocytic malignancy; Lipid metabolism disorders; Chronic ischemic heart disease; $\ldots$ . Procedures: Injection/infusion of cancer chemotherapy. Medications: Peptic-ulcer/GORD drugs; Antiemetics/antinauseants; Constipation drugs; Potassium; Irrigating solutions; Antimetabolites.

Appendix

Table 19: Medication prediction (Case B) across repeated chemotherapy visits. Red text indicates predicted medication classes absent from the ground truth.

Completed visits (earliest to latest)
Visit 1	Diagnoses: Diabetes mellitus; Lipid metabolism disorders; Hypertensive chronic kidney disease; Chronic ischemic heart disease; Chronic kidney disease; $\ldots$ . Procedures: None recorded. Medications: Insulins and analogues; Antithrombotic agents; Anti-parathyroid agents; $\ldots$ .
Visit 2	Diagnoses: Diabetes mellitus; Parathyroid disorders; Depressive disorder; Hypertensive chronic kidney disease; Chronic kidney disease; $\ldots$ . Procedures: Percutaneous transluminal coronary angioplasty; Peritoneal dialysis; $\ldots$ . Medications: Antacids; Potassium; Hypnotics/sedatives; $\ldots$ .
$\vdots$	$\cdots$ Visits 3-9 omitted from this display $\cdots$
Visit 10	Diagnoses: Acute myocardial infarction; Heart failure; Depressive disorder; Diabetes mellitus; Chronic kidney disease; $\ldots$ . Procedures: Hemodialysis. Medications: Antidepressants; Beta blockers; Antithrombotic agents; Lipid-modifying agents; $\ldots$ .
$\vdots$	$\cdots$ Visits 11-13 omitted from this display $\cdots$
Visit 14	Diagnoses: Depressive disorder; Diabetes mellitus; Hypertensive chronic kidney disease; Chronic kidney disease; $\ldots$ . Procedures: Venous catheterization for renal dialysis; Hemodialysis. Medications: Antithrombotic agents; Beta blockers; Opioids; $\ldots$ .

Appendix

Table 20: Next-visit diagnosis prediction (Case C) from a history of renal and cardiovascular conditions. Red text indicates predicted diagnosis categories absent from the ground truth.

Explore similar work

Apr 20, 2026cs.CL

RePrompT: Recurrent Prompt Tuning for Integrating Structured EHR Encoders with Large Language Models

Large Language Models (LLMs) have shown strong promise for mining Electronic Health Records (EHRs) by reasoning over longitudinal clinical information to capture context-rich patient trajectories. However, leveraging LLMs for structured EHRs (e.g., standardized diagnosis and medication codes) presents two key challenges. First, translating time-stamped EHR sequences into plain text can obscure both temporal structure and code identities, weakening the ability to capture code co-occurrence and longitudinal regularities. Second, unlike cohort-trained predictive models that learn a shared, task-aligned representation space across patients, LLMs are often applied in a case-isolated inference setting where each patient is processed independently without leveraging population-level patterns. To address these challenges, we introduce RePrompT, a time-aware LLM framework that integrates structured EHR encoders through prompt tuning, without modifying underlying architectures. Specifically, RePrompT recurrently incorporates latent states from prior visits to preserve longitudinal information, and injects population-level information through trainable prompt tokens derived from a cohort-trained, task-aligned EHR encoder. Experiments on MIMIC-III and MIMIC-IV demonstrate that RePrompT consistently outperforms both EHR-based and LLM-based baselines across multiple clinical prediction tasks.

Arya Hadizadeh Moghaddam, Drew Ross, Mohsen Nayebi Kerdabadi +2

University of Kansas, USA

Electronic Health Records Longitudinal Patient Records Patient Trajectories Large Language Model Pipelines Soft Prompt Tuning

Ask PDF

May 12, 2026cs.CL

From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Prediction

By processing electronic health records (EHRs) as natural language sequences, large language models (LLMs) have shown potential in clinical prediction tasks such as mortality prediction and phenotyping. However, longitudinal or highly frequent EHRs often yield excessively long token sequences that result in high computational costs and even reduced performance. Existing solutions either add modules for compression or remove less important tokens, which introduce additional inference latency or risk losing clinical information. To achieve lossless compression of token sequences without additional cost or loss of performance, we propose Medical Token-Pair Encoding (MedTPE), a layered method that extends standard tokenisation for EHR sequences. MedTPE merges frequently co-occurring medical token pairs into composite tokens, providing lossless compression while preserving the computational complexity through a dependency-aware replacement strategy. Only the embeddings of the newly introduced tokens of merely 0.5-1.0% of the LLM's parameters are fine-tuned via self-supervised learning. Experiments on real-world datasets for two clinical scenarios demonstrate that MedTPE reduces input token length by up to 31% and inference latency by 34-63%, while maintaining or even improving both predictive performance and output format compliance across multiple LLMs and four clinical prediction tasks. Furthermore, MedTPE demonstrates robustness across different input context lengths and generalisability to scientific and financial domains and different languages.

Mingcheng Zhu, Zhiyao Luo, Yu Liu +1

Department of Engineering Science, University of Oxford, Oxford, United Kingdom.

Multi-Token Prediction Electronic Health Records Byte-Pair Encoding Clinical Prediction

Ask PDF

May 12, 2026cs.IR

EHR-RAGp: Prototype-Guided Retrieval of Longitudinal Electronic Health Records for Clinical Prediction Models

Electronic Health Records (EHR) contain rich longitudinal patient information and are widely used in predictive modeling applications. However, effectively leveraging historical data remains challenging due to long trajectories, heterogeneous events, temporal irregularity, and the varying relevance of past clinical context. Existing approaches often rely on fixed windows or uniform aggregation, which can obscure clinically important signals. In this work, we introduce EHR-RAGp, a retrieval-based framework that dynamically integrates the most relevant patient history consisting of diverse clinical event types. We propose a prototype-guided retrieval module that acts as an alignment mechanism and estimates the relevance of retrieved historical chunks with respect to a given prediction task, guiding the model towards the most informative context. Across multiple clinical prediction tasks and two benchmark datasets, EHR-RAGp consistently outperforms state-of- the-art EHR-based and transformer-based baselines. Furthermore, EHR-RAGp is model-agnostic, as integrating it with various backbone models yields substantial performance gains. Overall, EHR-RAGp establishes a novel direction for modeling long-range clinical context to improve downstream performance via retrieval.

Saeed Shurrab, Mariam Al-Omari, Dana El Samad +1

New York University · New York University Abu Dhabi

Medical World Model Electronic Health Records Longitudinal Patient Records Time Series Foundation Models Hievi-Rag Long-Range Coherence

Ask PDF