TRACER: A Multimodal Harness for Patient Care in Hospitals

Zane Durante; Paul Tang; Keshigeyan Chandrasegaran; Amit Kaushal; Shrinidhi Kowshika Lakshmikanth; Feolu Kolawole; Yu Jun Zheng; Yifan Geng

cs.CVOct 15, 2026

TRACER: A Multimodal Harness for Patient Care in Hospitals

Authors: Zane Durante, Paul Tang, Keshigeyan Chandrasegaran, Amit Kaushal, Shrinidhi Kowshika Lakshmikanth, Feolu Kolawole, Yu Jun Zheng, Yifan Geng,

Organizations: Stanford University

Abstract

Continuous video and clinical records provide complementary information for understanding how patient behavior evolves over a hospital stay. A central challenge is how to represent behavior across the entire stay while preserving access to fine-grained visual and clinical evidence. We introduce TRACER, a training-free multimodal harness that constructs minute-level action records from video and combines them with clinical records into hourly reports. The harness equips an LLM agent with tools to query this hierarchy, inspect relevant video, and retrieve clinical records. To evaluate TRACER, we construct two benchmarks. The first contains 397 annotated one-minute clips spanning 15 atomic actions; the second poses 144 clinician-informed multiple-choice questions, grounded in 264 hours of video and clinical records from two ICU patients. TRACER achieves up to 84.8% accuracy for action detection and 61.0% QA accuracy versus 49.3% for the strongest baseline using the same backbone. Across 280 patients spanning approximately 60,000 camera-hours, TRACER identifies significantly less purposeful activity in patients with a positive delirium assessment during their stay than in those without (p < 0.0001). Our results show that combining continuous video with clinical records can support understanding of patient behavior over time and across patient cohorts without training on protected health data.

Figures & tables

Figure 1: Overview of TRACER. TRACER equips an agent model with hierarchical behavioral representations and tools for understanding multi-day ICU patient trajectories. Vision–language models analyze one-minute bedside video segments to detect atomic actions defined by a clinically informed taxonomy and generate descriptive captions. These outputs are combined with concurrent medication records to generate structured hourly behavioral reports. Through report, timeline, video, and EHR tools, the agent iteratively retrieves evidence and examines relevant video segments to answer retrospective clinical questions. Our evaluation suite assesses atomic action detection on 397 annotated one-minute clips and longitudinal clinical reasoning on 144 multiple-choice questions spanning 264 hours of continuous video.

No video
	Video	EHR	Cross-modal	Avg.
Random	20.1	20.0	20.0	20.1
Text-only LLM, Gemini 3.1 Flash Lite	32.3	26.9	25.0	30.9
Long-context VLMs
Qwen3-VL 8B, 475 frames	31.6	10.0	47.5	28.6
Qwen3-VL 8B, 950 frames	36.3	13.8	50.0	33.0

Table 1: MCQ performance on TRAJECTORY-144. QA accuracy (%) is shown by question type and is averaged over 5 gold-answer placements. TRACER rows share an identical harness (prompts, taxonomy, and temporal hierarchy) and differ only in the agent model. Best per column is in bold, and second best is underlined. The video tool is served by the agent model, or by Gemini 3.1 Flash Lite where marked † .

Figure 2: Per-Category Breakdown of Atomic Action Performance. (a) We plot the per-category F1 scores of the atomic actions used by TRACER. (b) We show the relative frequency of predicted atomic actions with respect to delirium status across a cohort of 280 patients spanning $\sim 60,000$ hours of video data. Delirium is defined as any positive CAM-ICU assessment during the stay ( $n=53$ delirium, $n=227$ non-delirium). Other torso movements and clinician restraining contained too few positive samples to be evaluated accurately and are therefore excluded from both panels.

Baselines
Model	Accuracy	F1	Precision	Recall
Random (Match distr.)	69.9 $\pm$ 0.4	44.8 $\pm$ 0.8	44.8 $\pm$ 0.8	44.7 $\pm$ 1.0
Majority Class	77.7	45.2	68.9	33.6
Open-source
Qwen3-VL (8B-Instruct)	78.0 $\pm$ 0.1	37.6 $\pm$ 0.4	84.8 $\pm$ 0.3	24.2 $\pm$ 0.3
Qwen3.5 (4B)	78.2 $\pm$ 0.3	41.0 $\pm$ 0.8	78.3 $\pm$ 1.7	27.7 $\pm$ 0.6

Table 2: Average atomic action detection performance. Micro-averaged across all 15 atomic actions on the 397-clip test set. Model rows are mean

\pm

std across 5 runs. Sampling settings for every model are listed in Appendix A.8 . Best result per column in bold .

	Video			EHR	Cross-modal	Avg.
Agent tools	$<$ 5 min	5–30 min	$>$ 30 min	Record only	Video + EHR
Single toolset performance
Timeline only	27.5	43.3	60.7	30.3	25.0	43.7
Reports only	33.7	52.5	47.4	33.3	43.3	42.1
EHR only	32.9	31.7	41.2	96.4	60.0	49.3
Video only	39.8	50.8	38.3	25.1	27.5	37.0

Table 3: Harness ablation. Accuracy (%) averaged over three agent models: Gemini 3.5 Flash, Gemini 3.1 Flash Lite, and Qwen3.5 9B. Single-toolset rows give the agent one tool family; cumulative rows add one family at a time up to the full harness, with gains shown in green. Best per column and block in bold, second best underlined. Additional results in Appendix A.12 and A.13 .

Proprietary
	Video			EHR	Cross-modal	Avg.	Calls
Agent model	$<$ 5 min	5–30 min	$>$ 30 min	Record only	Video + EHR
Gemini 3.5 Flash	35.8	57.5	80.3	98.5	77.5	70.7	9.5
Gemini 3.1 Flash Lite	36.4	52.5	63.7	96.2	45.0	61.0	8.9
Open-weight
gpt-oss-20b	32.1	41.2	62.5	93.1	40.0	57.4	3.7

Table 4: Comparison across agent models with fixed perception capabilities. Accuracy (%) on TRAJECTORY-144 with Gemini 3.1 Flash Lite used for all video tool calls, timeline, and report construction. Best per column in bold, second best underlined.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Scaffold (every configuration)
Tool	Parameters	Returns
get_patient_metadata	pat_id ?	demographics, ICU admission time, observation window, encounter history
convert_utc_local	dt_str , to_timezone	the datetime converted between UTC and local time
Timeline (minute-level records)
get_atomic_actions	hour_index	per-minute active action labels for one hour from admission
query_actions	labels ?, start_hour ?, end_hour ?	minutes active per label over the range; per-hour counts; per-label per-hour minutes

Appendix

Table 5: The agent’s tools. Exact names, parameters, and return values of the eleven tools, grouped by family; the row labels of Table 3 name the families each configuration adds to the scaffold pair. A question mark marks an optional parameter.

Time reference in the question	Questions	Accuracy
Clock time, span under 5 minutes	33	30.9
Clock time, span 5–30 minutes	15	57.3
Clock time, span over 30 minutes	12	50.0
Hour	13	50.8
Part of day (e.g., morning, afternoon)	16	66.2
Day (e.g., November 23)	5	56.0

Appendix

Table 6: TRAJECTORY-144 question breakdown by time reference. Accuracy (%) for Gemini 3.1 Flash Lite using TRACER, from an independent run of the Table 1 configuration (achieves 61.1% overall).

Figure 3: Patient cohort selection and characteristics. Left , CONSORT-style flow diagram of patient cohort selection. Sequential exclusion criteria were applied to identify patients with sufficient ICU stay duration, sedative exposure, non-neurological admission, and adequate bilateral video coverage. Right , Characteristics of the final study cohort (

N=280

Input	Gemini 3 Family (tokens)	Qwen3-VL 8B (tokens)
One minute, one camera view	19,823	8,640
One minute, both views	39,646	17,280
One hour, both views	2.38M	1.04M
One day, both views	57.1M	24.9M
Median stay (3.3 days)	188M	82M
Longest stay in cohort (14 days)	799M	349M

Appendix

Table 7: Token cost of raw video surpasses context lengths. Context windows: 1M tokens for the Gemini models, 256K for Qwen3-VL 8B. Gemini uses the original 1920

\times

1080 clip at 5 fps. Qwen3-VL uses 512

\times

288 resolution at 2 fps.

Inputs	Coverage	Tokens	$<$ 5 min	5–30 min	$>$ 30 min	EHR	Cross	All
Left camera captions	2.8k / 3.4k	1.0M	34.5	32.5	39.2	16.2	47.5	33.7
Left camera captions + EHR	2.8k / 1.7k	1.0M / 0.9M	30.3	32.5	43.3	76.2	42.5	45.0
Both cameras’ captions + EHR	1.4k / 0.9k	0.7M / 0.7M	32.7	31.2	35.9	76.2	50.0	42.7
950 frames	950 / 950	1.1M	35.8	32.5	38.0	14.6	25.0	31.9
475 frames	475 / 475	0.5M	26.7	35.0	36.6	17.7	22.5	29.9
475 frames + EHR	475 / 475	0.7M / 1.0M	34.5	30.0	48.9	83.8	40.0	49.3

Appendix

Table 8: Baselines with clinical records using Gemini 3.1 Flash Lite. Accuracy (%) averaged over gold placements. “Coverage” is the number of captioned minutes or frames that survive subsampling given the fixed 1M context for stay A / stay B; “Tokens” is the prompt length per trial. TRACER is shown on the last row for comparison.

Answering model	Minutes	$<$ 5 min	5–30 min	$>$ 30 min	EHR	Cross	All	TRACER
Gemini 3.1 FL (1M)	2.8k / 3.4k	34.5	32.5	39.2	16.2	47.5	33.7	61.0
gpt-oss-20b (131K)	379 / 391	33.3	26.3	22.7	14.6	20.0	23.9	57.4
Gemma-4 31B (131K)	379 / 377	33.9	25.0	31.4	23.1	22.5	29.3	67.9
Qwen3.5 9B (65K)	183 / 185	30.3	28.8	30.8	18.5	10.0	27.1	61.6

Appendix

Table 9: Socratic baselines with Gemini 3.1 Flash Lite captions, by open-weight answering model. Accuracy (%) averaged over gold placements; “minutes” is the number of captioned minutes that survive subsampling for stay A / stay B; the last column is the same model as the TRACER agent with Gemini 3.1 Flash Lite serving the video tool. Served context length shown in parentheses.

Figure 4: Perception accuracy versus cost. Left: Gemini models, list-price cost per 1,000 camera-hours of one-minute clips (log scale); the star marks the model used for the cohort run. Right: open-weight models, device-hours per 1,000 camera-hours on one MI250X graphics compute die.

	Movement	Interaction	Moments	Medication	Alignment	Summary	All	%
	Report section
Verifiable claims	41	36	28	16	12	20	153
Supported	31	32	25	13	9	19	129	84.3
Errors	10	4	3	3	3	1	24	15.7
Only partial timing match	7	1	3	0	0	1	12	7.8
Movement summary misleading	3	0	0	0	0	0	3	2.0

Appendix

Table 10: Claim-level audit of ten hourly reports. Counts are verifiable claims. Partial timing match: under 80% of cited minutes show the action. Carry over: a false medication absence repeated in the alignment section.

Agent model	All	$<$ 5 min	5–30 min	$>$ 30 min	EHR	Cross	Avg. Calls
Gemini 2.5 Flash Lite	49.2	32.7	35.0	53.9	69.2	45.0	3.7
Gemini 2.5 Flash	60.3	33.9	56.2	64.0	90.8	50.0	4.7
Gemini 3 Flash	70.8	44.8	66.3	77.0	91.5	72.5	12.3
Gemini 3.1 Flash Lite	61.0	36.4	52.5	63.7	96.2	45.0	8.9
Gemini 3.5 Flash Lite	61.1	40.0	52.5	59.8	99.2	52.5	6.8
Gemini 3.5 Flash	76.9	54.5	75.0	80.7	99.2	72.5	9.0

Appendix

Table 11: TRACER applied across twelve agent configurations. Full TRACER with the agent model as the only variable; the background records and reports are fixed. Tool calls are means per question. † Video tool served by Gemini 3.1 Flash Lite rather than the agent’s own model.

Gemini 3.5 Flash
	Video			EHR	Cross-modal	Stay		All
Agent tools	$<$ 5 min	5–30 min	$>$ 30 min	Record only	Video + EHR	A	B
Text-only	37.0	32.5	51.4	21.5	27.5	35.9	41.6	39.3
+ timeline	37.6	57.5	67.5	42.3	20.0	62.0	45.6	52.4
+ reports	40.6	72.5	70.5	38.5	30.0	70.2	45.9	55.8
+ EHR	39.4	76.2	71.1	100.0	75.0	84.7	59.5	69.9

Appendix

Table 12: Per-model cumulative ablation, by question type and by stay. Cumulative ablation: each row adds one tool family to the row above (Table 5 ). Paired McNemar tests on identical trials for Gemini 3.5 Flash show that every step is significant (

p<0.001

) except adding reports (

+3.5

\chi^{2}=3.6

p=0.06

Gemini 3.5 Flash
Agent tools	All	$<$ 5 min	5–30 min	$>$ 30 min	EHR	Cross
Reports only	51.0	44.2	72.5	55.4	36.9	47.5
EHR only	59.6	49.1	47.5	49.5	100.0	72.5
Video only	46.4	54.5	76.2	43.9	30.0	27.5
All tools	76.9	54.5	75.0	80.7	99.2	72.5
Gemini 3.1 Flash Lite

Appendix

Table 13: Single tool families on TRAJECTORY-144. Each row gives the agent one tool family only. Accuracy (%) averaged over gold placements; category columns are question means.

Action	Med. del.	Med. non-del.	MWU $p$	Holm	BH	Perm. $p$ (mw / ew)
Using non-medical objects	0.016	0.080	$1.8\times 10^{-10}$	$2.3\times 10^{-9}$	$2.3\times 10^{-9}$	$5\times 10^{-7}$ / $5\times 10^{-7}$
Object exchange	0.004	0.010	$8.9\times 10^{-10}$	$1.1\times 10^{-8}$	$5.8\times 10^{-9}$	$5\times 10^{-7}$ / $5\times 10^{-7}$
Leg ADL movement	0.002	0.007	$7.3\times 10^{-7}$	$8.0\times 10^{-6}$	$3.2\times 10^{-6}$	$4\times 10^{-6}$ / $5\times 10^{-7}$
Clinician–patient contact	0.423	0.349	$1.4\times 10^{-5}$	$1.4\times 10^{-4}$	$4.7\times 10^{-5}$	$1.1\times 10^{-3}$ / $1.0\times 10^{-4}$
Other arm movements	0.165	0.243	$2.9\times 10^{-5}$	$2.6\times 10^{-4}$	$7.4\times 10^{-5}$	$6.5\times 10^{-6}$ / $2\times 10^{-6}$
Gesticulating	0.030	0.053	$1.9\times 10^{-4}$	$1.5\times 10^{-3}$	$4.1\times 10^{-4}$	$1.4\times 10^{-4}$ / $3.5\times 10^{-4}$

Appendix

Table 14: Significance tests for delirium frequency differences. Per-patient rates (median) by group with two-sided Mann–Whitney

U

p

and Holm and Benjamini–Hochberg adjustment over the 13 evaluated actions. Permutation

p

for the minute-weighted and equal-weighted

\log_{2}

frequency ratio over 2,000,000 label permutations (floor

5\times 10^{-7}

Characteristic	Delirium ( $n=53$ )	Non-delirium ( $n=227$ )
ICU LOS, days, median [IQR]	4.62 [3.01, 8.40]	3.09 [1.89, 4.84]
Observed minutes, median [IQR]	7,560 [4,680, 12,840]	4,920 [2,580, 7,440]
Invasive mechanical ventilation, $n$ (%)	31 (58.5)	126 (55.5)
Per-patient median RASS, median [IQR]	0.00 [ $-1.00$ , 0.00]	0.00 [ $-0.50$ , 0.00]
RASS assessments $\leq-3$ , %	5.0 (of 6,979)	2.0 (of 18,402)

Appendix

Table 15: Cohort characteristics by delirium status. Delirium is defined as any positive CAM-ICU assessment during the ICU stay. RASS percentages are computed over all recorded assessments in each group rather than over patients. Observed minutes run from arrival in the monitored room and can exceed ICU LOS, which counts only time at the critical-care level of care.

	Minute-weighted		Equal patient weight
Action	FR	$\log_{2}$ FR [95% CI]	FR	$\log_{2}$ FR [95% CI]
Using non-medical objects	0.41	$-1.28$ $[-1.92,-0.75]$	0.39	$-1.35$ $[-1.94,-0.87]$
Leg ADL movement	0.46	$-1.13$ $[-1.69,-0.67]$	0.47	$-1.07$ $[-1.59,-0.64]$
Object exchange	0.48	$-1.05$ $[-1.57,-0.61]$	0.47	$-1.08$ $[-1.55,-0.69]$
Gesticulating	0.60	$-0.74$ $[-1.17,-0.35]$	0.68	$-0.56$ $[-0.97,-0.21]$
Other arm movements	0.62	$-0.70$ $[-1.00,-0.42]$	0.69	$-0.53$ $[-0.79,-0.29]$

Appendix

Table 16: Delirium frequency ratios under weighting schemes with bootstrap intervals. Minute-weighted ratios follow Eq. 1 and reproduce Figure 2 (b); equal-weighted ratios average per-patient rates within each group. Intervals are 95% percentile bootstrap intervals over 10,000 patient-level resamples. Values below

1.00

(

\log_{2}

below zero) indicate actions detected less frequently among patients with documented delirium. The final two actions are excluded from the 13-action evaluation set and from Figure 2 (b) for lack of sufficient positive instances in the annotated benchmark.

Video Understanding

Video understanding develops methods for recognizing actions, tracking events, grounding evidence in time, and reasoning across visual and audio streams.

See all 116 papers

TRACE: Tracking Multimodal Clinical Trajectories for Inpatient Risk Prediction
Oct 19, 2026Minjun Kim, Jong Hak MoonClinical Outcome Prediction Clinical Risk Prediction
Yeji X
Logi-PAR: Logic-Infused Patient Activity Recognition via Differentiable Rule
Mar 5, 2026Muhammad Zarar, MingZheng Zhang, Xiaowang Zhang +3Human Activity Recognition Neuro-Symbolic Reasoning
School of Computer Software, Tianjin University, China
MedTRACE: Tool-Augmented Multimodal Clinical Reasoning Agents for Evidence-Grounded Decision-Making
Sep 13, 2026Ji Lu, Lifei Liu, Haoran Yu +5Medical Diagnosis Clinical Decision-Making
Vanderbilt University, United States · Wichita State University, United States · University of Florida, United States +4

Oct 19, 2026J/K move · Enter open · S save

TRACE: Tracking Multimodal Clinical Trajectories for Inpatient Risk Prediction

Minjun Kim, Jong Hak Moon

Yeji X

Clinical Outcome Prediction Clinical Risk Prediction Longitudinal EHR Modeling Multimodal Trajectory Prediction

Open paper AskPDF arXiv

Accurate inpatient risk prediction requires modeling heterogeneous clinical evidence that accumulates irregularly over time. This challenge spans in-hospital mortality, prolonged length of stay, and readmission, each with task-specific observation windows. Existing multimodal approaches combine structured health record data, chest x-rays, and clinical text, but primarily focus on modality fusion or missing-modality compensation, with limited attention to temporal trajectories over an admission. We propose TRACE (TRAjectory modeling of multimodal Clinical Events), a daily-scale framework organizing clinical events into day-level trajectories, separating day-invariant global context from day-specific local events drawn from EHR data, chest x-rays, radiology reports, and discharge summaries. The trajectory is encoded using temporal attention, allowing unavailable modalities to emit no tokens. A phenotype-guided contrastive objective further organizes admissions under heterogeneous disease distributions. Across cohorts from MIMIC-IV, eICU, and MIMIC-III comprising nearly 180,000 total stays, we evaluate TRACE on mortality, length of stay, and readmission prediction. TRACE achieves the highest AUROC across all nine dataset-task settings and the highest ACC in eight, outperforming baselines. Ablation studies validate our attention masks and contrastive learning, while trajectory-level attribution localizes risk-associated hospital days and modalities.

Abstract

Video Understanding

Explore similar work

TRACE: Tracking Multimodal Clinical Trajectories for Inpatient Risk Prediction

Logi-PAR: Logic-Infused Patient Activity Recognition via Differentiable Rule

MedTRACE: Tool-Augmented Multimodal Clinical Reasoning Agents for Evidence-Grounded Decision-Making