ESL-Bench: An Event-Driven Synthetic Longitudinal Benchmark for Health Agents

Chao Li; Cailiang Liu; Ang Gao; Kexin Deng; Shu Zhang; Langping Xu; Xiaotong Shi; Xionghao Ding

cs.AIApr 3, 2026

ESL-Bench: An Event-Driven Synthetic Longitudinal Benchmark for Health Agents

Authors: Chao Li, Cailiang Liu, Ang Gao, Kexin Deng, Shu Zhang, Langping Xu, Xiaotong Shi, Xionghao Ding,

Organizations: Shanda Group

Abstract

Longitudinal health agents must reason across multi-source trajectories that combine continuous device streams, sparse clinical exams, and episodic life events - yet evaluating them is hard: real-world data cannot be released at scale, and temporally grounded attribution questions seldom admit definitive answers without structured ground truth. We present ESL-Bench, an event-driven synthesis framework and benchmark providing 100 synthetic users, each with a 1-5 year trajectory comprising a health profile, a multi-phase narrative plan, daily device measurements, periodic exam records, and an event log with explicit per-indicator impact parameters. Each indicator follows a baseline stochastic process driven by discrete events with sigmoid-onset, exponential-decay kernels under saturation and projection constraints; a hybrid pipeline delegates sparse semantic artifacts to LLM-based planning and dense indicator dynamics to algorithmic simulation with hard physiological bounds. Users are each paired with 100 evaluation queries across five dimensions - Lookup, Trend, Comparison, Anomaly, Explanation - stratified into Easy, Medium, and Hard tiers, with all ground-truth answers programmatically computable from the recorded event-indicator relationships. Evaluating 13 methods spanning LLMs with tools, DB-native agents, and memory-augmented RAG, we find that DB agents (48-58%) substantially outperform memory RAG baselines (30-38%), with the gap concentrated on Comparison and Explanation queries where multi-hop reasoning and evidence attribution are required.

Figures & tables

Figure 1: Structure and evaluation design of the event-driven synthetic longitudinal benchmark.

Figure 2: Internal probabilistic modeling architecture. Blue nodes denote LLM-driven conditional sampling; green nodes denote algorithmic simulation governed by equations 5 , 7 and 10 ; orange nodes denote validation mechanisms. Each LLM call conditions on a narrow context (short history window, single event type) to estimate a low-dimensional conditional distribution, while the simulator computes dense daily dynamics deterministically. The three-level validation layer covers expert review of event-impact templates, population-level marginal calibration against published norms, and per-indicator conformance auditing.

Dimension	Easy	Medium	Hard
Lookup	Profile attribute, device value on date, exam indicator status	Event-to-indicator mapping, yearly event/exam counts, indicator-to-event listing	Most indicator-rich event, multi-constraint chain queries, most impacted indicator
Trend	Best/worst month, quarter means, seasonal pattern detection	Monthly improvement, recovery detection, EWMA vs. SMA divergence	Piecewise trend breakpoints, rolling z-score outlier count, regime change detection
Comparison	Two-event comparison, post-event change, relative change ratio	Event severity ranking, indicator event-vs-baseline ratio, pre/post event comparison	Shared indicators between events, symmetric difference, events affecting both indicators
Anomaly	Ever-abnormal check, abnormal streak, threshold exceedance days	Abnormal trend acceleration, cross-exam persistence, deterioration rate	Causal chain detection, abnormal periodicity, multi-indicator abnormal cluster
Explanation	Active events at anomaly, event contribution ranking on date, net indicator effect	Impact ranking by magnitude, treatment effectiveness, recovery timeline	Counterfactual attribution, dominant event identification, multi-event net attribution

Table 1: Five evaluation dimensions stratified by three difficulty tiers. Each cell summarizes the core operations tested at that dimension–tier combination.

Dimension	Tier	Example Query
Lookup	Easy	“What was this user’s resting heart rate on 2024-03-15?”
Lookup	Med.	“Which indicators were affected by the event started jogging routine ?”
Lookup	Hard	“Which event affected the most indicators?”
Trend	Easy	“In which month was the average step count highest?”
Trend	Med.	“In which month did resting heart rate show the largest month-over-month change?”
Trend	Hard	“At which date did resting heart rate exhibit a regime change?”

Table 2: Representative query examples across dimensions and difficulty tiers.

Figure 3: Hybrid generation pipeline. LLM modules handle sparse semantic decisions (profiles, trajectory plan, event narratives, exam metadata), while algorithmic simulation produces daily device indicators under explicit dynamics and deterministic constraints. (1) Initialization with Profile Generation and Indicator Selection; (2) Trajectory Planning that produces a multi-phase narrative arc; (3) Daily Loop with Event Decision (LLM + trajectory context + sparsity gate), Device Indicator Simulator (algorithmic), and Exam Generation (LLM + deterministic anchoring); (4) Export producing structured artifacts.

Metric	Value
Total users	100
Age mean (std)	43 (16)
Age range	18–80+
Male	53%
Female	47%
Users with $\geq 1$ chronic condition	98%

Table 3: Cohort summary statistics.

Age Group	Primary Conditions	Users
18–44	Obesity/Metabolic, Pre-diabetes, Depression/Sleep, Healthy	33
45–64	T2DM, Hypertension, Metabolic, Depression, Comorbid	44
65–80+	HTN+CVD, T2DM+CKD, Osteoarthritis	23

Table 4: Age group and condition distribution (target mixture).

Metric	Value
Time span per user (mean)	1,196 days
Time span per user (median)	1,189 days
Time span per user (range)	388–1,813 days
Total observation days $\sum_{i}T_{i}$	119,600
Exam visits per user (mean)	7
Exam visit density (mean)	2 visits/year

Table 5: Longitudinal coverage statistics.

Metric	Value
Total events	18,800+
Events per user (mean)	188
Event categories	4
Mean duration (days)	88
Median duration (days)	4
Concurrent events per day (mean)	2–3 short-term; $\sim$ 9 total

Table 6: Event statistics.

Type	Count	Examples / Frequency
Device indicators	159 (60/user)	Daily (sleep, cardio, metabolic, activity, weight, SpO 2 )
Exam indicators	322 (200/user)	Per visit (CBC, metabolic, lipid, liver, renal, inflammatory)
Cross-source overlap	15	BP, glucose, SpO 2 , body composition, pulmonary

Table 7: Indicator coverage.

Metric	Mean
Indicator key presence rate	100%
Unit (UCUM) presence rate	100%
Range violation rate (pre-projection)	0%
Slope violation rate (pre-projection)	0%
Clipping rate (post-projection)	0%
Exam–device consistency (overlap indicators)	100%

Table 8: Aggregated audit metrics (means across users).

Figure 4: Four-month trajectory excerpt for one synthetic user showing four device indicators (Daily Stress Score, Resting Heart Rate, Total Sleep Time, Daily Step Count) with nine labeled life events. Shaded regions mark active event periods; blue indicates a beneficial effect on that indicator, red an adverse effect, and gray no effect—so the same event may appear in different colors across panels (e.g., “Indoor VR fitness routines” is blue for Stress/HR but red for Sleep/Steps). The black line is the 7-day rolling mean; the orange dotted line marks each indicator’s personalized baseline; the green dashed line marks an exam visit. Only the most prominent events are labeled; additional short-term events (e.g., acute anxiety episodes, OTC medication use) also contribute to the observed fluctuations.

Paradigm	Method	Base LLM	Description
LLM w/ tools	GPT-5.2	–	Each model uses its native tool-use capability with the same tool suite (lookup, query, read, search) over user artifacts. No agent framework; the LLM decides which tools to call.
	GPT-5.4	–
	Gemini 3 Flash	–
	Sonnet 4.6	–
	MiniMax M2.5	–
	GLM-5	–

Table 9: Baseline methods across three paradigms.

Method	Paradigm	Lookup	Trend	Comp.	Anom.	Expl.	Total
Gemini 3 Flash Preview	LLM w/ tools	73.5	94.8	74.4	64.3	25.3	62.9
Theta General (GPT-5.4)	DB agent	73.8	70.0	67.0	66.1	26.4	57.9
Theta Expert (GPT-5.4)	DB agent	75.6	49.5	68.1	71.2	30.9	56.4
Sonnet 4.6	LLM w/ tools	50.8	87.0	42.7	72.6	25.6	53.7
MiniMax M2.5	LLM w/ tools	43.1	87.0	43.7	73.4	17.5	49.6
GPT-5.4	LLM w/ tools	52.3	76.5	32.0	75.7	17.9	49.2

Table 10: Main results by dimension: accuracy (%). Methods are grouped by paradigm: LLM w/ tools (tool-use), DB agent (structured API), and Memory RAG. The base LLM for DB agent and Memory RAG methods is GPT-5.4. Total is the per-query average across all queries; dimension sub-scores are computed over varying query counts due to the sampling distribution.

Method	Easy	Med.	Hard	Total
Gemini 3 Flash Preview	75.0	70.6	55.0	62.9
Theta General	76.2	61.3	50.5	57.9
Theta Expert	73.8	55.2	51.9	56.4
Sonnet 4.6	73.1	71.6	37.9	53.7
MiniMax M2.5	56.2	67.0	38.0	49.6
GPT-5.4	73.8	60.7	35.6	49.2

Table 11: Main results by difficulty tier: accuracy (%). Total is the per-query average; tier sub-scores reflect the actual query distribution, which may deviate slightly from the 20/30/50 target split.

Synthetic Clinical Data Generation

See all 22 papers

Jan 20, 2026

LifeAgentBench: Benchmarking LLMs for Long-Horizon, Cross-Dimensional Lifestyle Health Reasoning

Personalized lifestyle health analysis requires long-horizon, multi-dimensional reasoning over heterogeneous lifestyle signals, and recent advances in mobile sensing and large language models (LLMs) make such support increasingly feasible. However, the capabilities of current LLMs in this setting remain insufficiently understood due to the lack of systematic benchmarks. In this paper, we introduce LifeAgentBench, a large-scale QA benchmark for long-horizon, cross-dimensional, and multi-user lifestyle health reasoning, containing 22,573 questions spanning from basic retrieval to complex reasoning. We release an extensible benchmark construction pipeline and a standardized evaluation protocol, deriving verifiable answers through executable queries and programs to support reliable assessment. We then systematically evaluate 13 representative LLMs on LifeAgentBench and identify key bottlenecks in long-horizon aggregation and cross-dimensional reasoning. Motivated by these findings, we propose LifeAgent, a tool-augmented reasoning baseline that decomposes complex queries, performs multi-step evidence retrieval, and invokes tools for deterministic aggregation. LifeAgent substantially enhances LLMs' capabilities on challenging reasoning tasks, achieving clear improvements over widely used baselines and showing potential for health reasoning in everyday scenarios. The benchmark is publicly available.

Ye Tian, Zihao Wang, Onat Gungor +2

University of California San Diego, La Jolla, CA, USA · Google Research, Mountain View, CA, USA

Healthcare Long-Horizon Agent Evaluation AI Agent Benchmarks LLM Tool Use

Ask PDF

Aug 27, 2026

BALMS: Benchmarking Agentic LLMs for Longitudinal Mental Health Sensing

Mental health assessment relies on episodic self-report scales, which convert subjective states such as stress into numerical scores but provide only sparse snapshots of wellbeing. Wearable devices offer longitudinal behavioral and physiological signals for continuous, low-burden monitoring. Recent LLM-driven personal-health agents enable natural language queries over wearable signals, but mainly handle short-term, retrieval-based lookups (e.g., highest step count over a week). They do not evaluate whether agents can reason over long-term signals to predict wellbeing scores paired with evidence-grounded rationales. To address this gap, we introduce BALMS, the first systematic benchmark of LLM-based agentic systems for longitudinal mental health sensing. BALMS spans 3 real-world longitudinal datasets, 2 task families (closed-form wellbeing-score prediction and rationale generation auto-graded by an LLM-as-Judge), 3 agentic paradigms evaluated across 5 open- and closed-source LLM backbones. We find that zero-shot agents rarely outperform a simple mean baseline, except with stronger backbones or compact, semantically meaningful features. Chain-of-thought prompting improves reasoning-oriented backbones, but does not guarantee temporal grounding or numerical correctness. Together with more analysis on efficiency and temporal scaling, BALMS highlights the need for longitudinal mental health agents that selectively retrieve history, ground temporal evidence, and reason over interpretable behavioral features.

Yu Yvonne Wu, Arvind Pillai, Yuliang Chen +7

Dartmouth College · University of Cambridge

AI Agent Benchmarks Mental Health Long-Horizon Agent Evaluation Wearable Sensing

Ask PDF

May 19, 2026

Synthesis and Evaluation of Long-term History-aware Medical Dialogue

An effective healthcare agent must be able to recall and reason over a patient's longitudinal medical history. However, the absence of datasets with realistic long-term dialogue timelines limits systematic evaluation. Real clinical text is constrained by privacy and ethics, while existing benchmarks focus on isolated interactions, failing to capture cross-session reasoning. We introduce a framework for synthesizing high-quality, long-term medical dialogues with LLMs. Our approach entails a knowledge-guided decomposition into three stages: constructing synthetic patient profiles with diverse disease and complication trajectories, generating multi-turn dialogues per encounter, and integrating them into a coherent longitudinal history dataset, MediLongChat. We establish three benchmark tasks-In-dialogue Reasoning, Cross-dialogue Reasoning, and Synthesis Reasoning-to evaluate the memory capabilities of healthcare agents. To assess data quality, we introduce a multi-dimensional evaluation framework combining vector-based metrics with LLM-as-a-judge assessments. Specifically, we define automatic measures-Faithfulness, Coherence, and Diversity-together with two LLM-based evaluations: Correctness and Realism. Benchmark experiments show that even state-of-the-art LLMs struggle with MediLongChat. These findings highlight the benchmark's applicability and underscore the need for tailored methods to advance healthcare agents.

Hebin Hu, Renke Dai, Ah-Hwee Tan +1

South-Central Minzu University Wuhan, China · Singapore Management University Singapore, Singapore

Healthcare AI Agent Benchmarks Long-Term Conversational Memory Synthetic Data Generation for Language Models Medical LLMs Text Generation

Ask PDF

Abstract

Synthetic Clinical Data Generation

Explore similar work

LifeAgentBench: Benchmarking LLMs for Long-Horizon, Cross-Dimensional Lifestyle Health Reasoning

BALMS: Benchmarking Agentic LLMs for Longitudinal Mental Health Sensing

Synthesis and Evaluation of Long-term History-aware Medical Dialogue