Learning Transferable Sensor Models via Language-Informed Pretraining

Yuliang Chen; Arvind Pillai; Yu Yvonne Wu; Sudarshan Regmi; Tess Z. Griffin; Lisa Marsch; Michael V. Heinz; Nicholas C. Jacobson

cs.AIMar 12, 2026

Learning Transferable Sensor Models via Language-Informed Pretraining

Authors: Yuliang Chen, Arvind Pillai, Yu Yvonne Wu, Sudarshan Regmi, Tess Z. Griffin, Lisa Marsch, Michael V. Heinz, Nicholas C. Jacobson,

Organizations: Dartmouth College

Abstract

Multimodal language models have demonstrated strong semantic understanding and reasoning over physiological and behavioral signals captured from diverse healthcare sensors. However, existing sensor-language models can only process a fixed number of sensor channels at a fixed sampling rate, making them difficult to adapt to new or different sensor configurations. This inflexibility prevents models from generalizing across heterogeneous health sensing ecosystems, where data spans high-frequency wearable streams to sparse, day-scale clinical signals. To bridge this gap, we introduce SLIP (Sensor Language-Informed Pretraining), an open-source framework for learning language-aligned representations that generalize across diverse sensor setups. SLIP integrates contrastive alignment with sensor-conditioned captioning, facilitating both discriminative understanding and generative reasoning. By repurposing a pretrained decoder-only language model via cross-attention and introducing a flexible patch embedder, SLIP transfers to new sensor configurations at inference without retraining, regardless of their native sampling rate or input length. Across 11 datasets, SLIP demonstrates superior performance in retrieval, signal captioning, and question answering. It achieves a 77.14% average linear-probing accuracy, a 3.56% improvement over the strongest baseline (SensorLM). Beyond classification, SLIP supports open-vocabulary sensor captioning and question answering without any architectural modifications. Our work offers broad implications for the development of sensor-language models, utilizing cross-modal supervision to ensure more robust generalizability in health sensing applications.

Figures & tables

Figure 2: SLIP architecture. Left: SLIP aligns sensor signals and text with a contrastive captioner: a unified sensor encoder handles varying sensor configurations, and a pretrained LM serves as both text encoder and multimodal decoder. Right: Minimal FlexMLP pseudocode. MLP weights learned at base_patch $=16$ are resized to the runtime patch_size , so a single encoder handles variable-length inputs. Changes to existing code are highlighted.

Figure 3: Overview of the pretrained SLIP and its three downstream tasks: (a) pretraining of SLIP via joint sensor-text contrastive matching and caption generation, (b) classification via linear probing on frozen sensor encoder features, (c) zero-shot sensor-text retrieval via cosine similarity between frozen sensor and text encoder embeddings, and (d) instruction following (sensor captioning and question answering) after supervised finetuning.

			Activity Recognition			Clinical Diagnosis						Stress Prediction			Urban Sensing
Category	Model	# Parameters	Average	WISDM	UCI-HAR	Average	Stroke	Diabetes	Hypertension	Sleep Stag.	Heart Cond.	Average	WESAD	Studentlife	Average	Obstacles	Beijing AQI
ML	PatchTST (Supervised)	0.9M	85.08	85.66	84.49	76.52	90.91	82.96	48.48	84.29	75.94	62.20	75.78	48.62	80.51	84.91	76.11
	Statistical ML	−	78.34	72.43	84.24	70.87	90.15	82.22	37.88	78.95	65.15	55.70	62.78	48.62	74.97	81.33	68.60
	SimMTM	0.5M	58.35	44.93	71.75	62.55	89.39	82.22	38.64	46.34	56.18	62.20	75.78	48.62	71.85	76.47	67.23
	TF-C	12M	59.60	55.39	63.81	63.66	90.91	82.96	39.39	46.97	58.06	49.58	56.95	42.20	47.98	49.87	46.08
TSFM	Sundial	128M	70.07	81.25	58.89	69.04	90.91	82.22	42.42	68.69	60.97	53.94	56.50	51.38	67.68	68.80	66.55

Table 2: Evaluation of the SLIP on 11 downstream sensor tasks across 4 domains compared against multiple baselines with linear-probing. We report the top-1 accuracy on the test set. Best results are in bold and second best are underlined .

	Activity Classification			Clinical Diagnosis						Stress Prediction			Urban Sensing
Model	Average	WISDM	UCI-HAR	Average	Stroke	Diabetes	Hypertension	Sleep Stag.	Heart Cond.	Average	WESAD	Studentlife	Average	Obstacles	Beijing AQI
NormWear	8.45	3.91	12.99	52.52	89.39	82.22	36.36	39.09	15.52	29.86	16.59	43.12	13.35	24.30	2.39
SensorLM	17.86	9.66	26.05	41.07	45.00	67.11	28.64	15.96	48.64	28.60	38.12	19.08	19.22	25.53	12.90
SLIP	17.54	7.45	27.62	49.11	86.40	74.07	34.85	34.83	15.39	49.12	56.95	41.28	27.08	29.92	24.23

Table 3: Retrieval evaluation on 11 downstream tasks. We compare SLIP with NormWear and SensorLM under the same sensor-text retrieval setting. All results report top-1 accuracy (R@1) on the test set. Best results are in bold and second best are underlined .

Model	Sensor Captioning (M4)					TSQA	HAR-CoT	Sleep-CoT	ECG-QA-CoT
Model	BLEU@4	METEOR	ROUGE-L	SBERTSim	BERTScore	Accuracy	Accuracy	Accuracy	Accuracy
OpenTSLM-SP	0.0026	0.0456	0.0255	0.1551	0.7250	26.79	0.55	5.91	1.11
OpenTSLM-Flamingo	0.1141	0.3210	0.2894	0.7990	0.8858	78.12	63.43	68.49	35.50
SensorLM	0.2008	0.4151	0.3595	0.9014	0.9024	95.73	60.80	73.12	30.29
SLIP	0.2046	0.4027	0.3658	0.8977	0.9077	99.75	64.35	74.19	37.18

Table 4: Sensor captioning and question answering results. All models share the same Gemma-3-270M language backbone for a controlled comparison. Captioning is evaluated on the M4 test set with BLEU@4, METEOR, ROUGE-L, SBERTSim, and BERTScore. TSQA, HAR-CoT, Sleep-CoT, and ECG-QA-CoT share the same QA evaluation protocol, reporting top-1 accuracy. Higher is better for all metrics. Best results are in bold and second best are underlined .

	Sensor Perception (Accuracy)		Retrieval (Accuracy)		QA (Accuracy)
(a) Training Objectives
SLIP pretraining	77.14		39.36		68.87
Caption-only	74.30	(-2.84)	25.12	(-14.24)	62.85	(-6.02)
Contrastive-only	74.77	(-2.37)	36.90	(-2.46)	53.06	(-15.81)
(b) FlexMLP
w/ FlexMLP	77.14		39.36		68.87

Table 5: Ablation study results . The default setting adopted by SLIP is marked in blue . We calculate

\pm

\Delta

within each group of ablations in comparison with the default setting.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

	Health	Web	Synthetic	Nature	Energy	IoT	Environment	Transport	Total
# of Samples	285054	145063	105085	69518	5846	5508	2368	66	618508
Percent (%)	46.09	23.45	16.99	11.24	0.95	0.89	0.38	0.01	100.00
Sources	( Liu et al., 2024 ) ( Luo et al., 2025 ) ( Chan et al., 2024 )	( Liu et al., 2024 )	( Xie et al., 2024 )	( Liu et al., 2024 )	( Liu et al., 2024 )	( Liu et al., 2024 )	( Liu et al., 2024 )	( Liu et al., 2024 )

Appendix

Table 6: Category distribution.

Dataset	Sensor	# Samples (Train / Test)	Freq.	# Cls	Label Names
WISDM ( Weiss, 2019 )	Accelerometer X, Y, Z	22396 / 5600	30Hz	18	Catch, Chips, Clap, Dribble, Drink, Fold, Jog, Kick, Pasta, Sandwich, Sit, Soup, Stair, Stand, Teeth, Type, Walk, Write
UCI-HAR ( Reyes-Ortiz et al., 2013 )	Accelerometer X, Y, Z Gyroscope X, Y, Z	1847 / 793	50Hz	7	Lay, Sit, Stand, Walk, Walking Upstairs, Walking Downstairs, Transition
PPG-CVA (Stroke) ( Liang et al., 2018 )	PPG	525 / 132	65Hz	2	Normal, Stroke
PPG-DM (Diabetes) ( Liang et al., 2018 )	PPG	522 / 135	65Hz	2	Normal, Diabetes
PPG-HTN (Hypertension) ( Liang et al., 2018 )	PPG	525 / 132	65Hz	4	Normal, Pre-hypertension, Stage-1, Stage-2
Sleep Stage (Sleep stage) ( Kemp et al., 2000 )	EEG-Fpz-Cz, EEG-Pz-Oz	33599 / 8709	100Hz	5	Light spindle, Light, Deep, REM, Wake

Appendix

Table 7: Evaluation dataset details.

Model	Cls. Avg	$\Delta$	Retrieval	$\Delta$	QA	$\Delta$
Ours (1.7 B)	77.14		39.36		68.87
0.8 B	75.23	(-1.91)	27.09	(-12.27)	60.18	(-8.69)
0.3 B	74.26	(-2.88)	21.64	(-17.72)	62.95	(-5.92)

Appendix

Table 8: Effect of Pretraining Dataset Size

Hyperparameter	TSQA	HAR-CoT	Sleep-CoT	ECG-QA-CoT	M4 Caption
Optimizer	AdamW ( Loshchilov and Hutter, 2019 )
Gradient clip	1.0
LR decay schedule	Cosine Schedule Decaying to $1e^{-7}$
Train Epoch	4	4	10	4	4
Train batch size	64	64	32	32	64
Warm up epochs	1	1	1	1	1

Appendix

Table 9: Hyperparameters used in the supervised finetuning experiment.

Eval	Model	Avg. rank	$p$	sig
LP	SLIP (ours)	1.64	−	−
	SensorLM	4.64	0.0098	**
	NormWear	4.68	0.0020	**
	Chronos	5.18	0.0049	**
	Chronos2	5.55	0.0020	**
	MOMENT	5.82	0.0049	**

Appendix

Table 10: Wilcoxon signed-rank tests of SLIP against each baseline. Average ranks are over the 11 datasets, lower is better.

{}^{*}p<0.1

{}^{**}p<0.05

, n.s. = not significant.

Table 11: Enlarging the patch versus downsampling at equal sequence length. (a) EEG rhythms retained at each sampling rate, on a 30s Sleep Stage window, with bands from the AASM manual ( Berry et al., 2020 ) . (b) Sleep Stage linear probing accuracy at equal tokens per channel. Reference settings are marked in blue .

Eval	Model	WISDM	UCIHAR	Stroke	Diabetes	Hypertension	Sleep Stage	Heart Cond.	WESAD	StudentLife	Obstacles	Beijing AQI	Average
LP	Stat Feat	72.03	84.18	47.41	45.12	26.85	66.41	39.12	47.81	42.46	80.98	36.86	53.57
	SimMTM	42.93	69.13	47.20	31.35	45.12	28.68	17.17	54.56	35.86	75.85	37.32	44.11
	TFC	54.60	64.20	47.62	45.12	23.86	33.08	24.50	23.60	25.11	47.40	23.44	37.50
	NormWear	70.60	76.88	52.30	44.90	32.99	72.25	47.52	63.02	32.24	82.82	48.58	56.74
	Chronos2	74.43	81.41	47.41	43.28	34.96	73.83	35.76	57.26	26.35	78.83	25.28	52.62
	Chronos	80.06	81.73	47.35	57.70	35.54	66.07	48.89	51.72	28.17	74.61	61.17	57.47

Appendix

Table 12: Additional F1 scores. Classification (LP) and retrieval (ZS), mirroring Table 2 .

Eval	Model	WISDM	UCIHAR	Stroke	Diabetes	Hypertension	Sleep Stage	Heart Cond.	WESAD	StudentLife	Obstacles	Beijing AQI
LP	Stat Feat	77.14 $\pm$ 0.25	85.62 $\pm$ 0.38	90.15 $\pm$ 0.00	82.22 $\pm$ 0.00	37.73 $\pm$ 1.30	77.33 $\pm$ 0.02	60.99 $\pm$ 0.33	68.52 $\pm$ 0.77	48.26 $\pm$ 1.37	81.69 $\pm$ 0.80	68.81 $\pm$ 0.51
	SimMTM	33.76 $\pm$ 0.08	70.52 $\pm$ 0.24	89.39 $\pm$ 0.00	82.22 $\pm$ 0.00	37.88 $\pm$ 1.07	41.76 $\pm$ 0.07	55.90 $\pm$ 0.08	62.51 $\pm$ 0.46	50.64 $\pm$ 0.69	36.37 $\pm$ 0.98	71.26 $\pm$ 0.26
	TFC	44.11 $\pm$ 0.11	59.95 $\pm$ 0.13	89.85 $\pm$ 0.61	80.74 $\pm$ 0.47	40.00 $\pm$ 0.57	44.88 $\pm$ 0.05	58.11 $\pm$ 0.25	55.43 $\pm$ 0.46	43.67 $\pm$ 1.70	33.66 $\pm$ 0.62	45.60 $\pm$ 0.46
	NormWear	70.82 $\pm$ 0.03	79.37 $\pm$ 0.32	90.45 $\pm$ 0.61	82.37 $\pm$ 0.30	40.45 $\pm$ 1.13	82.92 $\pm$ 0.18	69.53 $\pm$ 0.07	80.27 $\pm$ 0.49	49.17 $\pm$ 1.24	83.38 $\pm$ 0.32	74.40 $\pm$ 0.38
	Chronos2	75.18 $\pm$ 0.07	75.38 $\pm$ 0.51	87.12 $\pm$ 0.96	73.33 $\pm$ 3.18	38.79 $\pm$ 0.88	83.74 $\pm$ 0.14	61.20 $\pm$ 0.19	67.26 $\pm$ 0.75	47.89 $\pm$ 0.69	84.14 $\pm$ 0.23	51.74 $\pm$ 0.63
	Chronos	81.19 $\pm$ 0.13	84.04 $\pm$ 0.30	88.48 $\pm$ 2.00	80.59 $\pm$ 1.36	45.30 $\pm$ 0.88	76.69 $\pm$ 0.14	73.27 $\pm$ 0.18	66.46 $\pm$ 0.18	49.54 $\pm$ 1.00	75.45 $\pm$ 0.28	80.48 $\pm$ 0.14

Appendix

Table 13: Additional supplementary results. Classification (LP) and retrieval (ZS), mean

\pm

std over 5 random seeds.

Model	WISDM	UCIHAR	Stroke	Diabetes	Hypertension	Sleep Stage	Heart Cond.	WESAD	StudentLife	Obstacles	Beijing AQI	All Avg
SLIP	82.36	86.25	91.67	84.44	50.00	82.04	70.79	82.96	54.13	86.45	77.47	77.14
Contrastive-Only	79.14	85.37	90.91	84.44	40.91	81.10	68.12	82.96	50.46	84.65	74.40	74.77
Caption-Only	82.11	84.36	90.91	83.70	37.12	81.96	69.70	76.68	46.79	85.42	78.50	74.30
Random Paired	38.16	65.45	90.91	82.96	41.67	59.62	55.39	52.47	52.47	74.93	69.62	62.15
w/o FlexMLP ( $ps=16$ )	76.80	89.16	87.88	82.22	47.73	82.55	70.73	72.20	46.79	84.91	74.74	74.16
w/ Group Attention	79.45	84.36	92.42	85.19	48.48	81.67	72.12	83.41	56.88	85.94	77.47	77.04

Appendix

Table 14: Ablation Studies Linear Probing Performance. Reported Top-1 accuracy on 11 datasets.

Model	WISDM	UCIHAR	Stroke	Diabetes	Hypertension	Sleep Stage	Heart Cond.	WESAD	StudentLife	Obstacles	Beijing AQI	All Avg
SLIP	7.45	27.62	86.40	74.07	34.85	34.83	15.39	56.95	41.28	29.92	24.23	39.36
Contrastive-Only	9.50	19.67	79.55	80.74	24.24	39.05	10.97	52.47	40.37	30.18	19.11	36.90
Caption-Only	5.82	6.56	78.03	17.78	38.64	20.87	7.21	21.52	42.20	29.16	8.53	25.12
Random Paired	5.09	4.04	81.06	17.78	9.09	22.20	3.39	16.59	24.77	18.16	44.03	22.38
w/ Group Attention	8.30	27.11	84.09	82.22	36.36	26.28	26.85	52.02	44.95	24.04	19.11	39.21
w/o FlexMLP ( $ps=16$ )	5.86	13.75	90.15	59.26	29.54	29.02	8.79	40.81	30.28	30.18	45.05	34.79

Appendix

Table 15: Ablation Studies Retrieval Performance. Top-1 accuracy on 11 datasets.

Model	TSQA	HAR-CoT	Sleep-CoT	ECG-QA-CoT	Avg
SLIP	99.75	64.35	74.19	37.18	68.87
Caption-Only	99.04	49.16	69.14	34.07	62.85
Contrastive-Only	95.88	48.57	29.57	38.21	53.06
Random Paired	74.40	27.93	21.51	33.16	39.25
w/ Group Attention	99.27	60.41	69.14	37.05	66.47
w/o FlexMLP ( $ps=16$ )	98.65	61.66	69.03	38.17	66.88

Appendix

Table 16: Ablation Studies SFT performance. Top-1 accuracy on four sensor QA benchmarks.

Model	Sensor Captioning (M4)					TSQA	HAR-CoT	Sleep-CoT	ECG-QA-CoT
Model	BLEU@4	METEOR	ROUGE-L	SBERTSim	BERTScore	Accuracy	Accuracy	Accuracy	Accuracy
ITFormer	0.1641	0.3708	0.3182	0.8748	0.9001	89.35	41.29	43.99	38.01
ChatTS	0.1252	0.3518	0.2950	0.8500	0.8906	96.25	41.16	55.22	34.81
OpenTSLM-SP	0.0026	0.0456	0.0255	0.1551	0.7250	26.79	0.55	5.91	1.11
OpenTSLM-Flamingo	0.1141	0.3210	0.2894	0.7990	0.8858	78.12	63.43	68.49	35.50
SensorLM	0.2008	0.4151	0.3595	0.9014	0.9024	95.73	60.80	73.12	30.29

Appendix

Table 17: Sensor captioning and question answering results with larger-LLM baselines. Extends Table 4 with ITFormer-0.5B and ChatTS-8B, which use larger LLMs than the Gemma-3-270M backbone shared by OpenTSLM, SensorLM, and SLIP; they are shown in gray and excluded from highlighting. Metrics and protocol follow Table 4 . Best results are in bold and second best are underlined .

Figure 5: A qualitative example of sensor captioning from the M4 dataset. Highlighted segments denote salient local trends referenced in the generated caption.

Figure 6: A qualitative example of sensor question answering from the HAR-CoT dataset.

Figure 7: A qualitative example of sensor question answering from the Sleep-CoT dataset.

Figure 8: A qualitative example of sensor question answering from the TSQA dataset.

Figure 9: A qualitative failure example of sensor question answering from the ECG-QA-CoT dataset.

Multimodal Pretraining

Multimodal pretraining learns reusable representations by training models on paired or mixed data from multiple modalities, such as text, images, audio, and signals.

See all 177 papers

Sensor-Language-Action Models
Oct 6, 2026Yuekai Xu, Zitao Shuai, Yuzhe YangNext-Action Prediction
University of California, Los Angeles
TimeCap: A Scalable Open-Source Pipeline for Captioning Wearable Health Time-Series Data
Oct 15, 2026Max Rosenblattl, Milana Gurbanova, Karl Aurel Deck +16Time Series Language Modeling Wearable Health Monitoring
Stanford University, USA · Aionic Labs, Switzerland · Agentic Systems Lab, ETH Zurich, Switzerland +7
Sense Less, Infer More: Agentic Multimodal Transformers for Edge Medical Intelligence
Apr 12, 2026Chengwei Zhou, Zhaoyan Jia, Haotian Yu +6Efficient Multimodal Inference Wearable Health Monitoring
Case Western Reserve University, USA · University of Southern California, USA

Oct 6, 2026J/K move · Enter open · S save

Sensor-Language-Action Models

Yuekai Xu, Zitao Shuai, Yuzhe Yang

University of California, Los Angeles

Next-Action Prediction

Open paper AskPDF arXiv

Sensors are useful not only for understanding the world but also for deciding what to do next. Existing sensor models however largely stop at perception: they recognize states or predict outcomes, leaving actions modeled separately through task-specific and often closed label spaces. We introduce Sensor-Language-Action (SLA) modeling, a framework that connects multimodal sensor observations, natural language, and actions within a unified model. SLA uses language as a semantic interface between sensing and acting, allowing heterogeneous actions to be represented, predicted, and explained while remaining grounded in the underlying sensor evidence. We build a large-scale SLA benchmark consisting of datasets that span more than 116,000 individuals, 79 sensor modalities, and 60 action groups, together with a multi-faceted captioning pipeline that aligns user context, sensor dynamics, and action evidence. Building on this framework, we present OpenSLA, a unified SLA model for hierarchical action prediction, state understanding, and action explanation. Extensive experiments on real-world tasks in clinical prediction, operating rooms, and metabolic health verify its superior performance over the state-of-the-art. OpenSLA also demonstrates intriguing capabilities including language-guided evidence grounding and zero-shot generalization to unseen actions and cohorts.

Abstract

Multimodal Pretraining

Explore similar work

Sensor-Language-Action Models

TimeCap: A Scalable Open-Source Pipeline for Captioning Wearable Health Time-Series Data

Sense Less, Infer More: Agentic Multimodal Transformers for Edge Medical Intelligence