cs.AIApr 3, 2026
SaveESL-Bench: An Event-Driven Synthetic Longitudinal Benchmark for Health Agents
Organizations: Shanda Group
Abstract
Longitudinal health agents must reason across multi-source trajectories that combine continuous device streams, sparse clinical exams, and episodic life events - yet evaluating them is hard: real-world data cannot be released at scale, and temporally grounded attribution questions seldom admit definitive answers without structured ground truth. We present ESL-Bench, an event-driven synthesis framework and benchmark providing 100 synthetic users, each with a 1-5 year trajectory comprising a health profile, a multi-phase narrative plan, daily device measurements, periodic exam records, and an event log with explicit per-indicator impact parameters. Each indicator follows a baseline stochastic process driven by discrete events with sigmoid-onset, exponential-decay kernels under saturation and projection constraints; a hybrid pipeline delegates sparse semantic artifacts to LLM-based planning and dense indicator dynamics to algorithmic simulation with hard physiological bounds. Users are each paired with 100 evaluation queries across five dimensions - Lookup, Trend, Comparison, Anomaly, Explanation - stratified into Easy, Medium, and Hard tiers, with all ground-truth answers programmatically computable from the recorded event-indicator relationships. Evaluating 13 methods spanning LLMs with tools, DB-native agents, and memory-augmented RAG, we find that DB agents (48-58%) substantially outperform memory RAG baselines (30-38%), with the gap concentrated on Comparison and Explanation queries where multi-hop reasoning and evidence attribution are required.
Figures & tables
Figure 1: Structure and evaluation design of the event-driven synthetic longitudinal benchmark.
Figure 2: Internal probabilistic modeling architecture. Blue nodes denote LLM-driven conditional sampling; green nodes denote algorithmic simulation governed by equations 5 , 7 and 10 ; orange nodes denote validation mechanisms. Each LLM call conditions on a narrow context (short history window, single event type) to estimate a low-dimensional conditional distribution, while the simulator computes dense daily dynamics deterministically. The three-level validation layer covers expert review of event-impact templates, population-level marginal calibration against published norms, and per-indicator conformance auditing.
| Dimension | Easy | Medium | Hard |
|---|---|---|---|
| Lookup | Profile attribute, device value on date, exam indicator status | Event-to-indicator mapping, yearly event/exam counts, indicator-to-event listing | Most indicator-rich event, multi-constraint chain queries, most impacted indicator |
| Trend | Best/worst month, quarter means, seasonal pattern detection | Monthly improvement, recovery detection, EWMA vs. SMA divergence | Piecewise trend breakpoints, rolling z-score outlier count, regime change detection |
| Comparison | Two-event comparison, post-event change, relative change ratio | Event severity ranking, indicator event-vs-baseline ratio, pre/post event comparison | Shared indicators between events, symmetric difference, events affecting both indicators |
| Anomaly | Ever-abnormal check, abnormal streak, threshold exceedance days | Abnormal trend acceleration, cross-exam persistence, deterioration rate | Causal chain detection, abnormal periodicity, multi-indicator abnormal cluster |
| Explanation | Active events at anomaly, event contribution ranking on date, net indicator effect | Impact ranking by magnitude, treatment effectiveness, recovery timeline | Counterfactual attribution, dominant event identification, multi-event net attribution |
Table 1: Five evaluation dimensions stratified by three difficulty tiers. Each cell summarizes the core operations tested at that dimension–tier combination.
| Dimension | Tier | Example Query |
|---|---|---|
| Lookup | Easy | “What was this user’s resting heart rate on 2024-03-15?” |
| Lookup | Med. | “Which indicators were affected by the event started jogging routine ?” |
| Lookup | Hard | “Which event affected the most indicators?” |
| Trend | Easy | “In which month was the average step count highest?” |
| Trend | Med. | “In which month did resting heart rate show the largest month-over-month change?” |
| Trend | Hard | “At which date did resting heart rate exhibit a regime change?” |
Table 2: Representative query examples across dimensions and difficulty tiers.
Figure 3: Hybrid generation pipeline. LLM modules handle sparse semantic decisions (profiles, trajectory plan, event narratives, exam metadata), while algorithmic simulation produces daily device indicators under explicit dynamics and deterministic constraints. (1) Initialization with Profile Generation and Indicator Selection; (2) Trajectory Planning that produces a multi-phase narrative arc; (3) Daily Loop with Event Decision (LLM + trajectory context + sparsity gate), Device Indicator Simulator (algorithmic), and Exam Generation (LLM + deterministic anchoring); (4) Export producing structured artifacts.
| Metric | Value |
|---|---|
| Total users | 100 |
| Age mean (std) | 43 (16) |
| Age range | 18–80+ |
| Male | 53% |
| Female | 47% |
| Users with chronic condition | 98% |
Table 3: Cohort summary statistics.
| Age Group | Primary Conditions | Users |
|---|---|---|
| 18–44 | Obesity/Metabolic, Pre-diabetes, Depression/Sleep, Healthy | 33 |
| 45–64 | T2DM, Hypertension, Metabolic, Depression, Comorbid | 44 |
| 65–80+ | HTN+CVD, T2DM+CKD, Osteoarthritis | 23 |
Table 4: Age group and condition distribution (target mixture).
| Metric | Value |
|---|---|
| Time span per user (mean) | 1,196 days |
| Time span per user (median) | 1,189 days |
| Time span per user (range) | 388–1,813 days |
| Total observation days | 119,600 |
| Exam visits per user (mean) | 7 |
| Exam visit density (mean) | 2 visits/year |
Table 5: Longitudinal coverage statistics.
| Metric | Value |
|---|---|
| Total events | 18,800+ |
| Events per user (mean) | 188 |
| Event categories | 4 |
| Mean duration (days) | 88 |
| Median duration (days) | 4 |
| Concurrent events per day (mean) | 2–3 short-term; 9 total |
Table 6: Event statistics.
| Type | Count | Examples / Frequency |
|---|---|---|
| Device indicators | 159 (60/user) | Daily (sleep, cardio, metabolic, activity, weight, SpO 2 ) |
| Exam indicators | 322 (200/user) | Per visit (CBC, metabolic, lipid, liver, renal, inflammatory) |
| Cross-source overlap | 15 | BP, glucose, SpO 2 , body composition, pulmonary |
Table 7: Indicator coverage.
| Metric | Mean |
|---|---|
| Indicator key presence rate | 100% |
| Unit (UCUM) presence rate | 100% |
| Range violation rate (pre-projection) | 0% |
| Slope violation rate (pre-projection) | 0% |
| Clipping rate (post-projection) | 0% |
| Exam–device consistency (overlap indicators) | 100% |
Table 8: Aggregated audit metrics (means across users).
Figure 4: Four-month trajectory excerpt for one synthetic user showing four device indicators (Daily Stress Score, Resting Heart Rate, Total Sleep Time, Daily Step Count) with nine labeled life events. Shaded regions mark active event periods; blue indicates a beneficial effect on that indicator, red an adverse effect, and gray no effect—so the same event may appear in different colors across panels (e.g., “Indoor VR fitness routines” is blue for Stress/HR but red for Sleep/Steps). The black line is the 7-day rolling mean; the orange dotted line marks each indicator’s personalized baseline; the green dashed line marks an exam visit. Only the most prominent events are labeled; additional short-term events (e.g., acute anxiety episodes, OTC medication use) also contribute to the observed fluctuations.
| Paradigm | Method | Base LLM | Description |
| LLM w/ tools | GPT-5.2 | – | Each model uses its native tool-use capability with the same tool suite (lookup, query, read, search) over user artifacts. No agent framework; the LLM decides which tools to call. |
| GPT-5.4 | – | ||
| Gemini 3 Flash | – | ||
| Sonnet 4.6 | – | ||
| MiniMax M2.5 | – | ||
| GLM-5 | – |
Table 9: Baseline methods across three paradigms.
| Method | Paradigm | Lookup | Trend | Comp. | Anom. | Expl. | Total |
|---|---|---|---|---|---|---|---|
| Gemini 3 Flash Preview | LLM w/ tools | 73.5 | 94.8 | 74.4 | 64.3 | 25.3 | 62.9 |
| Theta General (GPT-5.4) | DB agent | 73.8 | 70.0 | 67.0 | 66.1 | 26.4 | 57.9 |
| Theta Expert (GPT-5.4) | DB agent | 75.6 | 49.5 | 68.1 | 71.2 | 30.9 | 56.4 |
| Sonnet 4.6 | LLM w/ tools | 50.8 | 87.0 | 42.7 | 72.6 | 25.6 | 53.7 |
| MiniMax M2.5 | LLM w/ tools | 43.1 | 87.0 | 43.7 | 73.4 | 17.5 | 49.6 |
| GPT-5.4 | LLM w/ tools | 52.3 | 76.5 | 32.0 | 75.7 | 17.9 | 49.2 |
Table 10: Main results by dimension: accuracy (%). Methods are grouped by paradigm: LLM w/ tools (tool-use), DB agent (structured API), and Memory RAG. The base LLM for DB agent and Memory RAG methods is GPT-5.4. Total is the per-query average across all queries; dimension sub-scores are computed over varying query counts due to the sampling distribution.
| Method | Easy | Med. | Hard | Total |
|---|---|---|---|---|
| Gemini 3 Flash Preview | 75.0 | 70.6 | 55.0 | 62.9 |
| Theta General | 76.2 | 61.3 | 50.5 | 57.9 |
| Theta Expert | 73.8 | 55.2 | 51.9 | 56.4 |
| Sonnet 4.6 | 73.1 | 71.6 | 37.9 | 53.7 |
| MiniMax M2.5 | 56.2 | 67.0 | 38.0 | 49.6 |
| GPT-5.4 | 73.8 | 60.7 | 35.6 | 49.2 |
Table 11: Main results by difficulty tier: accuracy (%). Total is the per-query average; tier sub-scores reflect the actual query distribution, which may deviate slightly from the 20/30/50 target split.