TimeCap: A Scalable Open-Source Pipeline for Captioning Wearable Health Time-Series Data

Max Rosenblattl; Milana Gurbanova; Karl Aurel Deck; Narayan Schuetz; Thomas Kaar; Manuel Milling; Nicolas Zumarraga; Ning Wang

cs.LGOct 15, 2026

TimeCap: A Scalable Open-Source Pipeline for Captioning Wearable Health Time-Series Data

Authors: Max Rosenblattl, Milana Gurbanova, Karl Aurel Deck, Narayan Schuetz, Thomas Kaar, Manuel Milling, Nicolas Zumarraga, Ning Wang,

Organizations: Stanford University, USA · Aionic Labs, Switzerland · Agentic Systems Lab, ETH Zurich, Switzerland · CHI, Technical University of Munich University Hospital, Germany ·

Abstract

Time-series data generated continuously across industrial, environmental, and especially health domains produce multimodal signals whose scale and heterogeneity make their processing and interpretation particularly challenging. Time-Series Language Models (TSLMs) promise scalable interpretation of these signals, yet remain bottlenecked by a fundamental lack of text-annotated time series pairs that are signal-grounded, reproducible, and extensible across modalities and schemas. Existing paired corpora are either closed or limited in scale and modality coverage. Annotation with large language models could scale them up, but its outputs cannot be regenerated exactly, which prevents others from re-deriving, extending, or auditing the resulting corpus. To address this, we release TimeCap, an open deterministic captioning pipeline with three reusable extractor tiers and a cross-channel synthesis layer that operates directly on the signal with no language model in the loop, extensible to new schemas by writing channel definitions alone. This positions TimeCap as a reproducible mechanism for future dataset generation rather than a one-off release. Applying TimeCap to OpenMHC, the MyHeart Counts wearable dataset, yields MHCap, the largest open caption layer for multimodal wearable time-series data. MHCap spans 19 Apple HealthKit channels and 7,005,763 captions over 908,944 person-days. To showcase the impact of the corpus, we train OpenTSLM, a time-series language model, on MHCap and benchmark on health-outcome prediction. Caption-augmented training (OpenTSLM-Cap) outperforms a label-only baseline on 29 of 32 per-user binary evaluations across 8 tasks.

Figures & tables

Corpus	Scale	Modalities	Annotation	Open	Det.	Ext.
SensorLM a	59.7M h, 103,731 ppl	PPG, ACC, EDA, TEMP, ALT	3-tier	✗	✓	✗
Sensor-QA b	5,648 QA, 60 ppl	IMU, audio, phone state	human-written	✓	✗	✗
TSR-Suite c	23,605 (2,339 h.)	multi-domain TS	LLM + human-in-loop	✗	✗	✗
ChatTS d	134K samples (synthetic only)	univariate/multivariate TS	LLM + synthetic	✓	✗	✗
SleepLM e	>100K h, >10K ppl	PSG, (EEG/EOG, ECG, resp., EMG)	multi-level captioning	✗	✓	✗
MHCap	7.0M cap, 909k days, 2,081 ppl	19 HealthKit channels	3-tier + cross-channel	✓ f	✓	✓

Table 1: Paired sensor and language corpora at a glance. MHCap combines an open caption corpus with deterministic generation, schema extensibility, and an open-source pipeline. Open : corpus and pipeline code released. Det. : caption generation is deterministic. Ext. : captioning pipeline extends to new sensor schemas via configuration rather than redesign. Numbers are reported as in the cited papers.

Figure 1: TimeCap pipeline overview. Raw multivariate sensor streams are mapped into a unified Recording representation, then captioned by four composable extractor tiers (statistical, structural, semantic, cross-channel) that emit a shared annotation schema. Cross-channel synthesis composes per-channel evidence into multi-channel narratives.

Corpus			Captions by tier (count, share)
Dataset	Units a	Sequences b	Statistical	Structural	Semantic	Cross-channel
OpenMHC -weekly	908,944	144,412	1,002,819 (14.3%)	3,191,600 (45.6%)	867,948 (12.4%)	1,943,396 (27.7%)
WESAD	15	15	240 (16.4%)	1,071 (73.3%)	75 (5.1%)	75 (5.1%)

Table 2: TimeCap corpus scale on OpenMHC -weekly and WESAD. For OpenMHC -weekly, 7,005,763 captions are generated with a mean of 48.5 captions per user-week for a total of 2,081 participants. For WESAD, 1,461 captions are generated with a mean of 97.4 captions per session for a total of 15 participants.

	Majority		1B				3B
Task			OpenTSLM-Cap		OpenTSLM		OpenTSLM-Cap		OpenTSLM
Binary (AUROC / AUPRC)	AUROC	AUPRC	AUROC	AUPRC	AUROC	AUPRC	AUROC	AUPRC	AUROC	AUPRC
Biological Sex	50.00	25.38	79.24	51.72	78.90	49.99	77.86	51.24	75.47	49.80
Cardiovascular Disease †	50.00	15.30	65.15	29.99	62.09	26.79	67.19	28.94	65.28	26.25
Coronary Artery Disease †	50.00	6.90	65.88	15.76	63.38	16.62	62.46	12.57	59.94	9.69
Atrial Fibrillation †	50.00	3.23	68.36	7.24	59.82	4.91	64.08	7.33	61.72	5.40

Table 3: User-level results. OpenTSLM-Cap vs. OpenTSLM at the 1B and 3B parameter scales on the test split. Binary tasks report AUROC and AUPRC computed over per-user mean predictions. Ordinal tasks report Spearman

\rho

over the per-user mean predicted class, and QWK against the rounded per-user mean. The Majority column reports a class-prior baseline (always predict the majority class) as a no-skill floor and is excluded from best-of-row bolding. Best per task within each parameter scale in bold. † Held-out clinical task, not seen during fine-tuning.

	Majority		1B				3B
Task			OpenTSLM-Cap		OpenTSLM		OpenTSLM-Cap		OpenTSLM
Binary (F1 / Acc)	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc
Biological Sex	43.47	76.91	64.59	78.71	65.20	78.12	64.87	78.29	61.24	78.40
Cardiovascular Disease †	43.49	76.94	61.65	72.62	61.13	76.37	62.00	70.22	58.62	67.13
Coronary Artery Disease †	47.10	89.02	58.83	76.02	61.11	81.91	55.60	70.78	52.96	66.82
Atrial Fibrillation †	49.24	97.02	43.68	67.76	44.99	71.41	44.39	67.97	41.79	62.20

Table 4: Per-week results. Same model lineup at the (user, week) granularity, computed across all weekly predictions in the test split. Binary tasks (top) report macro-F1 (mean of class-0 and class-1 F1, label-symmetric; reported as F1) and accuracy; ordinal tasks (bottom) report Spearman

\rho

and QWK. The Majority column reports a class-prior baseline (always predict the majority class) as a no-skill floor and is excluded from best-of-row bolding. Best per task within each parameter scale in bold. † Held-out clinical task, not seen during fine-tuning. User-level metrics are given in Table 3 .

Binary (AUROC)
		OpenTSLM-Cap , caption family
Task	OpenTSLM	Structural	Semantic	Cross-channel	All
Biological Sex	78.90	78.87	78.42	78.56	79.10
Employment	64.74	69.09	66.61	66.30	66.33
Hypertension	67.96	68.77	71.70	70.38	69.74
Sleep Diagnosis	58.86	60.44	59.25	61.26	59.20

Table 5: Caption-family ablation (1B, user level). Per-user AUROC (binary) and Spearman

\rho

(ordinal) as in Table 3 , whose 1B OpenTSLM-Cap is re-evaluated here as All . One seed per arm; best per row in bold. † Held-out task.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Task	n_users	n_pos	prev. (%)
Biological Sex (Female=pos)	587	149	25.4
Hypertension	279	90	32.3
Employment (False=pos)	531	127	23.9
Sleep Diagnosis	530	101	19.1
Diabetes †	279	23	8.2
Cardiovascular Disease †	464	71	15.3

Appendix

Table 6: Per-task test-set composition: binary tasks. Captioned test partition: 604 unique users; per-task counts vary due to per-label NaN filtering. “n_pos” is the minority-class user count (the convention used for AUPRC reporting); the minority label is given in parentheses where it differs from the codebase default. † Held-out clinical task, not seen during fine-tuning.

Task	n_users	Level distribution (n, %)
BMI (5 levels)	557	Underweight 25 (4.5%); Normal weight 175 (31.4%); Overweight 206 (37.0%); Obesity 121 (21.7%); Morbid Obesity 30 (5.4%)
Happiness (4 levels)	328	Low 28 (8.5%); Medium 53 (16.2%); High 133 (40.5%); Very High 114 (34.8%)

Appendix

Table 7: Per-task test-set composition: ordinal tasks. User counts per ordinal level. Spearman

\rho

and QWK reliability scales with the spread of both predicted and ground-truth scores, so heavily concentrated tasks (e.g., Happiness, with 75% of users in High/Very High) have lower effective signal than tasks where the distribution is more even (e.g., BMI).

Channels	Aggregator	Extractor output
IPSteps, IPDist, IPFlights, WSteps, WDist, WEnergy	MetricAggregator (default)	Mean, max, min, std
WHR	NonZeroAggregator	Non-zero mean, max, min, std

Appendix

Table 8: OpenMHC statistical aggregation configuration (current implementation).

Channels	filter_zeros	min_height	min_prominence	min_distance	Unit
IPSteps, WSteps	False	100.0	0.0	10	steps/min
IPDist, WDist	False	72.0	0.0	10	m/min
IPFlights	False	1.0	0.0	10	count/min
WHR	True	100.0	5.0	20	bpm
WEnergy	False	3000.0	0.0	15	cal/min

Appendix

Table 9: OpenMHC spike-detector configuration. All spike detectors use min_width=1 , top_k=4 .

Channels	filter_zeros	min_coverage	min_effect	min_r2	Notes
IPSteps, IPDist, IPFlights, WSteps, WDist, WEnergy	False	0.75	1.25	0.50	Default
WHR	True	0.75	1.25	0.50	Zero-filtered

Appendix

Table 10: OpenMHC trend-detector configuration. Unless overridden, trend detectors use window_fracs=(0.1, 0.25, 1.0) , min_window=12 , and stride_frac=0.25 .

Synthesizer	Source channels	Supporting channels	Output labels
Sleep	SleepAsleep, SleepInBed	WHR, WDist, WSteps	sleep_asleep , sleep_inbed , in_bed_not_sleeping
Walking	Walk	WHR, WDist, WSteps, WEnergy	walking
Cardio	Run, Cycle	WHR, WDist, WEnergy; WSteps for Run	cardio_running , cardio_cycling
EnduranceActivity	Elliptical, MixedCardio	WHR, WDist, WEnergy	endurance_elliptical , endurance_mixed_cardio
StationaryActivity	HIIT, Strength, Functional	WHR, WEnergy, WDist, WSteps	stationary_hiit , stationary_strength , stationary_functional
MindBodyActivity	Yoga	WHR, WEnergy, WDist, WSteps	mind_body_yoga

Appendix

Table 11: OpenMHC cross-channel synthesizer definitions. All listed synthesizers are instantiated with min_duration=5 at minute resolution.

Channels	Aggregator	Extractor output
wesad:e4:hr	NonZeroAggregator	Non-zero mean, max, min, std
All other channels	MetricAggregator (default)	Mean, max, min, std

Appendix

Table 12: WESAD statistical aggregation configuration (current implementation).

Channels	filter_zeros	min/max_window	min_effect	min_r2	Notes
Chest/Wrist ACC axes and ACC magnitude	False	40 / 150	1.8	0.40	min_span=30 , merge_gap=15
Chest EMG	False	20 / 240	0.6	0.15	min_span=14 , merge_gap=15
Wrist HR ( wesad:e4:hr )	True	defaults	1.25	0.50	defaults: min_span=10 , merge_gap=5
Chest/Wrist EDA	False	30 / 240	0.10	0.60	min_span=40 , merge_gap=15
Chest/Wrist temperature	False	90 / 600	0.18	0.70	min_span=60 , merge_gap=30
Chest respiration	False	24 / 180	0.75	0.20	min_span=16 , merge_gap=15

Appendix

Table 13: WESAD trend-detector configuration by channel group.

Channels	filter_zeros	min_height	min_prominence	Unit
Chest ACC magnitude	False	$-$	0.05	g
Wrist ACC magnitude	False	$-$	0.10	g
Wrist HR	True	100.0	5.0	bpm
Chest EDA	False	$-$	0.25	$\mu$ S
Wrist EDA	False	$-$	0.50	$\mu$ S
Chest/Wrist temperature	False	$-$	0.25	∘ C

Appendix

Table 14: WESAD spike-detector configuration. All spike detectors use min_width=1 , top_k=4 , and min_distance=60 .

Synthesizer	Source channel	Supporting channels	Output label
StressResponseSynthesizer	wesad:label:stress	HR, chest/wrist EDA, chest respiration, chest/wrist ACC magnitude	stress_response
RecoveryCalmSynthesizer (baseline)	wesad:label:baseline	HR, chest/wrist EDA, chest respiration, chest/wrist ACC magnitude	calm_baseline
RecoveryCalmSynthesizer (meditation)	wesad:label:meditation	HR, chest/wrist EDA, chest respiration, chest/wrist ACC magnitude	calm_meditation
AmusementActivationSynthesizer	wesad:label:amusement	HR, chest/wrist EDA, chest/wrist ACC magnitude	amusement_activation

Appendix

Table 15: WESAD cross-channel synthesizer definitions. All listed synthesizers are instantiated with min_duration=30 at second resolution.

Figure 3: Example with sleep caption 1

Figure 4: Example with sleep caption 2

Figure 5: Example with sleep caption 3

Figure 6: Example with cycling

Figure 7: Example with elliptical

Figure 8: Example with running

Figure 9: Example with HIIT

Figure 10: Example with walking 1

Figure 11: Example with walking 2

Figure 12: Example with yoga

Figure 13: WESADCap subject 1.

Figure 14: WESADCap subject 2.

Figure 15: WESADCap subject 3.

Figure 16: WESADCap subject 4.

Figure 17: WESADCap subject 5.

Figure 18: WESADCap subject 6.

Figure 19: WESADCap subject 7.

Figure 20: WESADCap subject 8.

Figure 21: WESADCap subject 9.

Figure 22: WESADCap subject 10.

Figure 23: WESADCap subject 11.

Figure 24: WESADCap subject 12.

Figure 25: WESADCap subject 13.

Figure 26: WESADCap subject 14.

Figure 27: WESADCap subject 15.

Time Series Language Modeling

See all 14 papers

Decoupling Perception from Description: Computation-Grounded Representation Alignment between Multivariate Time Series and Language
Aug 5, 2026Xinran Feng, Yi Xie, Chao Zhang +4Cross-Modal Representation Learning Multimodal Pretraining
Thrust of Financial Technology, The Hong Kong University of Science and Technology (Guangzhou) · Technical University of Munich · Heilbronn Data Science Center, Munich Data Science Institute +1
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
May 18, 2026Paul Quinlan, Jeremy Levasseur, Qingguo Li +1Cross-Modal Learning Multivariate Time Series Forecasting
1InertialAI · Department of Electrical and Computer Engineering, Queen’s University · Department of Mechanical and Materials Engineering, Queen’s University
Learning Transferable Sensor Models via Language-Informed Pretraining
Mar 12, 2026Yuliang Chen, Arvind Pillai, Yu Yvonne Wu +6Multimodal Pretraining Language Modeling
Dartmouth College

Aug 5, 2026J/K move · Enter open · S save

Decoupling Perception from Description: Computation-Grounded Representation Alignment between Multivariate Time Series and Language

Xinran Feng, Yi Xie, Chao Zhang, Ruikun Li+3

Thrust of Financial Technology, The Hong Kong University of Science and Technology (Guangzhou) · Technical University of Munich · Heilbronn Data Science Center, Munich Data Science Institute+1

Cross-Modal Representation Learning Multimodal Pretraining Time Series Language Modeling

Open paper AskPDF arXiv

Training multimodal models to align time series with language runs into a self-supervision trap. The usual recipe asks an LLM to read a series and write a description, so label quality is capped by the perceptual skill the model is supposed to learn. The data can never teach more than the labeler already knows. A second gap makes this worse: most datasets use a single variable, but the patterns that matter (cross-channel correlation, lead-lag structure, co-occurring anomalies) appear only with several variables, right where the labeling LLM's limits are most exposed. These two problems create a trilemma: existing methods are reliable, realistic, or scalable, but none achieves all three. We resolve this by decoupling perception from description. Deterministic code computes a set of statistics from real, open-source multivariate series; the LLM verbalizes those precomputed facts. Perception, which LLMs do poorly, is handled by computation, while the LLM handles expression. This produces CGTime, our 4B-parameter computation-grounded time-series-language model. CGTime outperforms far larger general-purpose models on multivariate understanding tasks: it attains the best multivariate fact score on our held-out benchmark (0.283 vs. 0.173 for GPT-4o-mini and 0.203 for GPT-5.4-nano), a gap that survives Holm-corrected paired significance tests against every baseline. It also states verifiable numerical facts in generated captions more accurately and covers a broader range of statistical properties.

Abstract

Time Series Language Modeling

Explore similar work

Decoupling Perception from Description: Computation-Grounded Representation Alignment between Multivariate Time Series and Language

Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding

Learning Transferable Sensor Models via Language-Informed Pretraining