Learning Transferable Sensor Models via Language-Informed Pretraining

Yuliang Chen; Arvind Pillai; Yu Yvonne Wu; Sudarshan Regmi; Tess Z. Griffin; Lisa Marsch; Michael V. Heinz; Nicholas C. Jacobson

cs.AIMar 12, 2026

Learning Transferable Sensor Models via Language-Informed Pretraining

Authors: Yuliang Chen, Arvind Pillai, Yu Yvonne Wu, Sudarshan Regmi, Tess Z. Griffin, Lisa Marsch, Michael V. Heinz, Nicholas C. Jacobson,

Organizations: Dartmouth College

Abstract

Multimodal language models have demonstrated strong semantic understanding and reasoning over physiological and behavioral signals captured from diverse healthcare sensors. However, existing sensor-language models can only process a fixed number of sensor channels at a fixed sampling rate, making them difficult to adapt to new or different sensor configurations. This inflexibility prevents models from generalizing across heterogeneous health sensing ecosystems, where data spans high-frequency wearable streams to sparse, day-scale clinical signals. To bridge this gap, we introduce SLIP (Sensor Language-Informed Pretraining), an open-source framework for learning language-aligned representations that generalize across diverse sensor setups. SLIP integrates contrastive alignment with sensor-conditioned captioning, facilitating both discriminative understanding and generative reasoning. By repurposing a pretrained decoder-only language model via cross-attention and introducing a flexible patch embedder, SLIP transfers to new sensor configurations at inference without retraining, regardless of their native sampling rate or input length. Across 11 datasets, SLIP demonstrates superior performance in retrieval, signal captioning, and question answering. It achieves a 77.14% average linear-probing accuracy, a 3.56% improvement over the strongest baseline (SensorLM). Beyond classification, SLIP supports open-vocabulary sensor captioning and question answering without any architectural modifications. Our work offers broad implications for the development of sensor-language models, utilizing cross-modal supervision to ensure more robust generalizability in health sensing applications.

Figures & tables

Figure 2: SLIP architecture. Left: SLIP aligns sensor signals and text with a contrastive captioner: a unified sensor encoder handles varying sensor configurations, and a pretrained LM serves as both text encoder and multimodal decoder. Right: Minimal FlexMLP pseudocode. MLP weights learned at base_patch $=16$ are resized to the runtime patch_size , so a single encoder handles variable-length inputs. Changes to existing code are highlighted.

Figure 3: Overview of the pretrained SLIP and its three downstream tasks: (a) pretraining of SLIP via joint sensor-text contrastive matching and caption generation, (b) classification via linear probing on frozen sensor encoder features, (c) zero-shot sensor-text retrieval via cosine similarity between frozen sensor and text encoder embeddings, and (d) instruction following (sensor captioning and question answering) after supervised finetuning.

			Activity Recognition			Clinical Diagnosis						Stress Prediction			Urban Sensing
Category	Model	# Parameters	Average	WISDM	UCI-HAR	Average	Stroke	Diabetes	Hypertension	Sleep Stag.	Heart Cond.	Average	WESAD	Studentlife	Average	Obstacles	Beijing AQI
ML	PatchTST (Supervised)	0.9M	85.08	85.66	84.49	76.52	90.91	82.96	48.48	84.29	75.94	62.20	75.78	48.62	80.51	84.91	76.11
	Statistical ML	−	78.34	72.43	84.24	70.87	90.15	82.22	37.88	78.95	65.15	55.70	62.78	48.62	74.97	81.33	68.60
	SimMTM	0.5M	58.35	44.93	71.75	62.55	89.39	82.22	38.64	46.34	56.18	62.20	75.78	48.62	71.85	76.47	67.23
	TF-C	12M	59.60	55.39	63.81	63.66	90.91	82.96	39.39	46.97	58.06	49.58	56.95	42.20	47.98	49.87	46.08
TSFM	Sundial	128M	70.07	81.25	58.89	69.04	90.91	82.22	42.42	68.69	60.97	53.94	56.50	51.38	67.68	68.80	66.55

Table 2: Evaluation of the SLIP on 11 downstream sensor tasks across 4 domains compared against multiple baselines with linear-probing. We report the top-1 accuracy on the test set. Best results are in bold and second best are underlined .

	Activity Classification			Clinical Diagnosis						Stress Prediction			Urban Sensing
Model	Average	WISDM	UCI-HAR	Average	Stroke	Diabetes	Hypertension	Sleep Stag.	Heart Cond.	Average	WESAD	Studentlife	Average	Obstacles	Beijing AQI
NormWear	8.45	3.91	12.99	52.52	89.39	82.22	36.36	39.09	15.52	29.86	16.59	43.12	13.35	24.30	2.39
SensorLM	17.86	9.66	26.05	41.07	45.00	67.11	28.64	15.96	48.64	28.60	38.12	19.08	19.22	25.53	12.90
SLIP	17.54	7.45	27.62	49.11	86.40	74.07	34.85	34.83	15.39	49.12	56.95	41.28	27.08	29.92	24.23

Table 3: Retrieval evaluation on 11 downstream tasks. We compare SLIP with NormWear and SensorLM under the same sensor-text retrieval setting. All results report top-1 accuracy (R@1) on the test set. Best results are in bold and second best are underlined .

Model	Sensor Captioning (M4)					TSQA	HAR-CoT	Sleep-CoT	ECG-QA-CoT
Model	BLEU@4	METEOR	ROUGE-L	SBERTSim	BERTScore	Accuracy	Accuracy	Accuracy	Accuracy
OpenTSLM-SP	0.0026	0.0456	0.0255	0.1551	0.7250	26.79	0.55	5.91	1.11
OpenTSLM-Flamingo	0.1141	0.3210	0.2894	0.7990	0.8858	78.12	63.43	68.49	35.50
SensorLM	0.2008	0.4151	0.3595	0.9014	0.9024	95.73	60.80	73.12	30.29
SLIP	0.2046	0.4027	0.3658	0.8977	0.9077	99.75	64.35	74.19	37.18

Table 4: Sensor captioning and question answering results. All models share the same Gemma-3-270M language backbone for a controlled comparison. Captioning is evaluated on the M4 test set with BLEU@4, METEOR, ROUGE-L, SBERTSim, and BERTScore. TSQA, HAR-CoT, Sleep-CoT, and ECG-QA-CoT share the same QA evaluation protocol, reporting top-1 accuracy. Higher is better for all metrics. Best results are in bold and second best are underlined .

	Sensor Perception (Accuracy)		Retrieval (Accuracy)		QA (Accuracy)
(a) Training Objectives
SLIP pretraining	77.14		39.36		68.87
Caption-only	74.30	(-2.84)	25.12	(-14.24)	62.85	(-6.02)
Contrastive-only	74.77	(-2.37)	36.90	(-2.46)	53.06	(-15.81)
(b) FlexMLP
w/ FlexMLP	77.14		39.36		68.87

Table 5: Ablation study results . The default setting adopted by SLIP is marked in blue . We calculate

\pm

\Delta

within each group of ablations in comparison with the default setting.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

	Health	Web	Synthetic	Nature	Energy	IoT	Environment	Transport	Total
# of Samples	285054	145063	105085	69518	5846	5508	2368	66	618508
Percent (%)	46.09	23.45	16.99	11.24	0.95	0.89	0.38	0.01	100.00
Sources	( Liu et al., 2024 ) ( Luo et al., 2025 ) ( Chan et al., 2024 )	( Liu et al., 2024 )	( Xie et al., 2024 )	( Liu et al., 2024 )	( Liu et al., 2024 )	( Liu et al., 2024 )	( Liu et al., 2024 )	( Liu et al., 2024 )

Appendix

Table 6: Category distribution.

Dataset	Sensor	# Samples (Train / Test)	Freq.	# Cls	Label Names
WISDM ( Weiss, 2019 )	Accelerometer X, Y, Z	22396 / 5600	30Hz	18	Catch, Chips, Clap, Dribble, Drink, Fold, Jog, Kick, Pasta, Sandwich, Sit, Soup, Stair, Stand, Teeth, Type, Walk, Write
UCI-HAR ( Reyes-Ortiz et al., 2013 )	Accelerometer X, Y, Z Gyroscope X, Y, Z	1847 / 793	50Hz	7	Lay, Sit, Stand, Walk, Walking Upstairs, Walking Downstairs, Transition
PPG-CVA (Stroke) ( Liang et al., 2018 )	PPG	525 / 132	65Hz	2	Normal, Stroke
PPG-DM (Diabetes) ( Liang et al., 2018 )	PPG	522 / 135	65Hz	2	Normal, Diabetes
PPG-HTN (Hypertension) ( Liang et al., 2018 )	PPG	525 / 132	65Hz	4	Normal, Pre-hypertension, Stage-1, Stage-2
Sleep Stage (Sleep stage) ( Kemp et al., 2000 )	EEG-Fpz-Cz, EEG-Pz-Oz	33599 / 8709	100Hz	5	Light spindle, Light, Deep, REM, Wake

Appendix

Table 7: Evaluation dataset details.

Model	Cls. Avg	$\Delta$	Retrieval	$\Delta$	QA	$\Delta$
Ours (1.7 B)	77.14		39.36		68.87
0.8 B	75.23	(-1.91)	27.09	(-12.27)	60.18	(-8.69)
0.3 B	74.26	(-2.88)	21.64	(-17.72)	62.95	(-5.92)

Appendix

Table 8: Effect of Pretraining Dataset Size

Hyperparameter	TSQA	HAR-CoT	Sleep-CoT	ECG-QA-CoT	M4 Caption
Optimizer	AdamW ( Loshchilov and Hutter, 2019 )
Gradient clip	1.0
LR decay schedule	Cosine Schedule Decaying to $1e^{-7}$
Train Epoch	4	4	10	4	4
Train batch size	64	64	32	32	64
Warm up epochs	1	1	1	1	1

Appendix

Table 9: Hyperparameters used in the supervised finetuning experiment.

Eval	Model	Avg. rank	$p$	sig
LP	SLIP (ours)	1.64	−	−
	SensorLM	4.64	0.0098	**
	NormWear	4.68	0.0020	**
	Chronos	5.18	0.0049	**
	Chronos2	5.55	0.0020	**
	MOMENT	5.82	0.0049	**

Appendix

Table 10: Wilcoxon signed-rank tests of SLIP against each baseline. Average ranks are over the 11 datasets, lower is better.

{}^{*}p<0.1

{}^{**}p<0.05

, n.s. = not significant.

Table 11: Enlarging the patch versus downsampling at equal sequence length. (a) EEG rhythms retained at each sampling rate, on a 30s Sleep Stage window, with bands from the AASM manual ( Berry et al., 2020 ) . (b) Sleep Stage linear probing accuracy at equal tokens per channel. Reference settings are marked in blue .

Eval	Model	WISDM	UCIHAR	Stroke	Diabetes	Hypertension	Sleep Stage	Heart Cond.	WESAD	StudentLife	Obstacles	Beijing AQI	Average
LP	Stat Feat	72.03	84.18	47.41	45.12	26.85	66.41	39.12	47.81	42.46	80.98	36.86	53.57
	SimMTM	42.93	69.13	47.20	31.35	45.12	28.68	17.17	54.56	35.86	75.85	37.32	44.11
	TFC	54.60	64.20	47.62	45.12	23.86	33.08	24.50	23.60	25.11	47.40	23.44	37.50
	NormWear	70.60	76.88	52.30	44.90	32.99	72.25	47.52	63.02	32.24	82.82	48.58	56.74
	Chronos2	74.43	81.41	47.41	43.28	34.96	73.83	35.76	57.26	26.35	78.83	25.28	52.62
	Chronos	80.06	81.73	47.35	57.70	35.54	66.07	48.89	51.72	28.17	74.61	61.17	57.47

Appendix

Table 12: Additional F1 scores. Classification (LP) and retrieval (ZS), mirroring Table 2 .

Eval	Model	WISDM	UCIHAR	Stroke	Diabetes	Hypertension	Sleep Stage	Heart Cond.	WESAD	StudentLife	Obstacles	Beijing AQI
LP	Stat Feat	77.14 $\pm$ 0.25	85.62 $\pm$ 0.38	90.15 $\pm$ 0.00	82.22 $\pm$ 0.00	37.73 $\pm$ 1.30	77.33 $\pm$ 0.02	60.99 $\pm$ 0.33	68.52 $\pm$ 0.77	48.26 $\pm$ 1.37	81.69 $\pm$ 0.80	68.81 $\pm$ 0.51
	SimMTM	33.76 $\pm$ 0.08	70.52 $\pm$ 0.24	89.39 $\pm$ 0.00	82.22 $\pm$ 0.00	37.88 $\pm$ 1.07	41.76 $\pm$ 0.07	55.90 $\pm$ 0.08	62.51 $\pm$ 0.46	50.64 $\pm$ 0.69	36.37 $\pm$ 0.98	71.26 $\pm$ 0.26
	TFC	44.11 $\pm$ 0.11	59.95 $\pm$ 0.13	89.85 $\pm$ 0.61	80.74 $\pm$ 0.47	40.00 $\pm$ 0.57	44.88 $\pm$ 0.05	58.11 $\pm$ 0.25	55.43 $\pm$ 0.46	43.67 $\pm$ 1.70	33.66 $\pm$ 0.62	45.60 $\pm$ 0.46
	NormWear	70.82 $\pm$ 0.03	79.37 $\pm$ 0.32	90.45 $\pm$ 0.61	82.37 $\pm$ 0.30	40.45 $\pm$ 1.13	82.92 $\pm$ 0.18	69.53 $\pm$ 0.07	80.27 $\pm$ 0.49	49.17 $\pm$ 1.24	83.38 $\pm$ 0.32	74.40 $\pm$ 0.38
	Chronos2	75.18 $\pm$ 0.07	75.38 $\pm$ 0.51	87.12 $\pm$ 0.96	73.33 $\pm$ 3.18	38.79 $\pm$ 0.88	83.74 $\pm$ 0.14	61.20 $\pm$ 0.19	67.26 $\pm$ 0.75	47.89 $\pm$ 0.69	84.14 $\pm$ 0.23	51.74 $\pm$ 0.63
	Chronos	81.19 $\pm$ 0.13	84.04 $\pm$ 0.30	88.48 $\pm$ 2.00	80.59 $\pm$ 1.36	45.30 $\pm$ 0.88	76.69 $\pm$ 0.14	73.27 $\pm$ 0.18	66.46 $\pm$ 0.18	49.54 $\pm$ 1.00	75.45 $\pm$ 0.28	80.48 $\pm$ 0.14

Appendix

Table 13: Additional supplementary results. Classification (LP) and retrieval (ZS), mean

\pm

std over 5 random seeds.

Model	WISDM	UCIHAR	Stroke	Diabetes	Hypertension	Sleep Stage	Heart Cond.	WESAD	StudentLife	Obstacles	Beijing AQI	All Avg
SLIP	82.36	86.25	91.67	84.44	50.00	82.04	70.79	82.96	54.13	86.45	77.47	77.14
Contrastive-Only	79.14	85.37	90.91	84.44	40.91	81.10	68.12	82.96	50.46	84.65	74.40	74.77
Caption-Only	82.11	84.36	90.91	83.70	37.12	81.96	69.70	76.68	46.79	85.42	78.50	74.30
Random Paired	38.16	65.45	90.91	82.96	41.67	59.62	55.39	52.47	52.47	74.93	69.62	62.15
w/o FlexMLP ( $ps=16$ )	76.80	89.16	87.88	82.22	47.73	82.55	70.73	72.20	46.79	84.91	74.74	74.16
w/ Group Attention	79.45	84.36	92.42	85.19	48.48	81.67	72.12	83.41	56.88	85.94	77.47	77.04

Appendix

Table 14: Ablation Studies Linear Probing Performance. Reported Top-1 accuracy on 11 datasets.

Model	WISDM	UCIHAR	Stroke	Diabetes	Hypertension	Sleep Stage	Heart Cond.	WESAD	StudentLife	Obstacles	Beijing AQI	All Avg
SLIP	7.45	27.62	86.40	74.07	34.85	34.83	15.39	56.95	41.28	29.92	24.23	39.36
Contrastive-Only	9.50	19.67	79.55	80.74	24.24	39.05	10.97	52.47	40.37	30.18	19.11	36.90
Caption-Only	5.82	6.56	78.03	17.78	38.64	20.87	7.21	21.52	42.20	29.16	8.53	25.12
Random Paired	5.09	4.04	81.06	17.78	9.09	22.20	3.39	16.59	24.77	18.16	44.03	22.38
w/ Group Attention	8.30	27.11	84.09	82.22	36.36	26.28	26.85	52.02	44.95	24.04	19.11	39.21
w/o FlexMLP ( $ps=16$ )	5.86	13.75	90.15	59.26	29.54	29.02	8.79	40.81	30.28	30.18	45.05	34.79

Appendix

Table 15: Ablation Studies Retrieval Performance. Top-1 accuracy on 11 datasets.

Model	TSQA	HAR-CoT	Sleep-CoT	ECG-QA-CoT	Avg
SLIP	99.75	64.35	74.19	37.18	68.87
Caption-Only	99.04	49.16	69.14	34.07	62.85
Contrastive-Only	95.88	48.57	29.57	38.21	53.06
Random Paired	74.40	27.93	21.51	33.16	39.25
w/ Group Attention	99.27	60.41	69.14	37.05	66.47
w/o FlexMLP ( $ps=16$ )	98.65	61.66	69.03	38.17	66.88

Appendix

Table 16: Ablation Studies SFT performance. Top-1 accuracy on four sensor QA benchmarks.

Model	Sensor Captioning (M4)					TSQA	HAR-CoT	Sleep-CoT	ECG-QA-CoT
Model	BLEU@4	METEOR	ROUGE-L	SBERTSim	BERTScore	Accuracy	Accuracy	Accuracy	Accuracy
ITFormer	0.1641	0.3708	0.3182	0.8748	0.9001	89.35	41.29	43.99	38.01
ChatTS	0.1252	0.3518	0.2950	0.8500	0.8906	96.25	41.16	55.22	34.81
OpenTSLM-SP	0.0026	0.0456	0.0255	0.1551	0.7250	26.79	0.55	5.91	1.11
OpenTSLM-Flamingo	0.1141	0.3210	0.2894	0.7990	0.8858	78.12	63.43	68.49	35.50
SensorLM	0.2008	0.4151	0.3595	0.9014	0.9024	95.73	60.80	73.12	30.29

Appendix

Table 17: Sensor captioning and question answering results with larger-LLM baselines. Extends Table 4 with ITFormer-0.5B and ChatTS-8B, which use larger LLMs than the Gemma-3-270M backbone shared by OpenTSLM, SensorLM, and SLIP; they are shown in gray and excluded from highlighting. Metrics and protocol follow Table 4 . Best results are in bold and second best are underlined .

Figure 5: A qualitative example of sensor captioning from the M4 dataset. Highlighted segments denote salient local trends referenced in the generated caption.

Figure 6: A qualitative example of sensor question answering from the HAR-CoT dataset.

Figure 7: A qualitative example of sensor question answering from the Sleep-CoT dataset.

Figure 8: A qualitative example of sensor question answering from the TSQA dataset.

Figure 9: A qualitative failure example of sensor question answering from the ECG-QA-CoT dataset.

Multimodal Pretraining

Multimodal pretraining learns reusable representations by training models on paired or mixed data from multiple modalities, such as text, images, audio, and signals.

See all 177 papers

Oct 6, 2026

Sensor-Language-Action Models

Sensors are useful not only for understanding the world but also for deciding what to do next. Existing sensor models however largely stop at perception: they recognize states or predict outcomes, leaving actions modeled separately through task-specific and often closed label spaces. We introduce Sensor-Language-Action (SLA) modeling, a framework that connects multimodal sensor observations, natural language, and actions within a unified model. SLA uses language as a semantic interface between sensing and acting, allowing heterogeneous actions to be represented, predicted, and explained while remaining grounded in the underlying sensor evidence. We build a large-scale SLA benchmark consisting of datasets that span more than 116,000 individuals, 79 sensor modalities, and 60 action groups, together with a multi-faceted captioning pipeline that aligns user context, sensor dynamics, and action evidence. Building on this framework, we present OpenSLA, a unified SLA model for hierarchical action prediction, state understanding, and action explanation. Extensive experiments on real-world tasks in clinical prediction, operating rooms, and metabolic health verify its superior performance over the state-of-the-art. OpenSLA also demonstrates intriguing capabilities including language-guided evidence grounding and zero-shot generalization to unseen actions and cohorts.

Yuekai Xu, Zitao Shuai, Yuzhe Yang

University of California, Los Angeles

Next-Action Prediction

Ask PDF

Oct 15, 2026

TimeCap: A Scalable Open-Source Pipeline for Captioning Wearable Health Time-Series Data

Time-series data generated continuously across industrial, environmental, and especially health domains produce multimodal signals whose scale and heterogeneity make their processing and interpretation particularly challenging. Time-Series Language Models (TSLMs) promise scalable interpretation of these signals, yet remain bottlenecked by a fundamental lack of text-annotated time series pairs that are signal-grounded, reproducible, and extensible across modalities and schemas. Existing paired corpora are either closed or limited in scale and modality coverage. Annotation with large language models could scale them up, but its outputs cannot be regenerated exactly, which prevents others from re-deriving, extending, or auditing the resulting corpus. To address this, we release TimeCap, an open deterministic captioning pipeline with three reusable extractor tiers and a cross-channel synthesis layer that operates directly on the signal with no language model in the loop, extensible to new schemas by writing channel definitions alone. This positions TimeCap as a reproducible mechanism for future dataset generation rather than a one-off release. Applying TimeCap to OpenMHC, the MyHeart Counts wearable dataset, yields MHCap, the largest open caption layer for multimodal wearable time-series data. MHCap spans 19 Apple HealthKit channels and 7,005,763 captions over 908,944 person-days. To showcase the impact of the corpus, we train OpenTSLM, a time-series language model, on MHCap and benchmark on health-outcome prediction. Caption-augmented training (OpenTSLM-Cap) outperforms a label-only baseline on 29 of 32 per-user binary evaluations across 8 tasks.

Max Rosenblattl, Milana Gurbanova, Karl Aurel Deck +16

Stanford University, USA · Aionic Labs, Switzerland · Agentic Systems Lab, ETH Zurich, Switzerland +7

Time Series Language Modeling Wearable Health Monitoring Multimodal Learning

Ask PDF

Apr 12, 2026

Sense Less, Infer More: Agentic Multimodal Transformers for Edge Medical Intelligence

Edge-based multimodal medical monitoring requires models that balance diagnostic accuracy with severe energy constraints. Continuous acquisition of ECG, PPG, EMG, and IMU streams rapidly drains wearable batteries, often limiting operation to under 10 hours, while existing systems overlook the high temporal redundancy present in physiological signals. We introduce Adaptive Multimodal Intelligence (AMI), an end-to-end framework that jointly learns when to sense and how to infer. AMI integrates three components: (1) a lightweight Agentic Modality Controller that uses differentiable Gumbel-Sigmoid gating to dynamically select active sensors based on model confidence and task relevance; (2) a Learned Sigma-Delta Sensing module that applies patch-wise Delta-Sigma operations with learnable thresholds to skip temporally redundant samples; and (3) a Foundation-backed Multimodal Prediction Model built on unimodal foundation encoders and a cross-modal transformer with temporal context, enabling robust fusion even under gated or missing inputs. These components are trained jointly via a multi-objective loss combining classification accuracy, sparsity regularization, cross-modal alignment, and predictive coding. AMI is hardware-aware, supporting dynamic computation graphs and masked operations, leading to real energy and latency savings. Across MHEALTH, HMC Sleep, and WESAD datasets, it reduces sensor usage by 48.8% while improving state-of-the-art accuracy by 1.9% on average.

Chengwei Zhou, Zhaoyan Jia, Haotian Yu +6

Case Western Reserve University, USA · University of Southern California, USA

Efficient Multimodal Inference Wearable Health Monitoring Edge ML Adaptive Sampling Multimodal Sensor Fusion

Ask PDF

Abstract

Multimodal Pretraining

Explore similar work

Sensor-Language-Action Models

TimeCap: A Scalable Open-Source Pipeline for Captioning Wearable Health Time-Series Data

Sense Less, Infer More: Agentic Multimodal Transformers for Edge Medical Intelligence