SPECTRA: An Efficient Spectral-Informed Neural Network for Sensor-Based Activity Recognition

Deepika Gurung; Lala Shakti Swarup Ray; Mengxi Liu; Bo Zhou; Paul Lukowicz

cs.LGMar 27, 2026

SPECTRA: An Efficient Spectral-Informed Neural Network for Sensor-Based Activity Recognition

Authors: Deepika Gurung, Lala Shakti Swarup Ray, Mengxi Liu, Bo Zhou, Paul Lukowicz

Organizations: RPTU & DFKI, Kaiserslautern, Germany

Abstract

Real time sensor based applications in pervasive computing require edge deployable models to ensure low latency privacy and efficient interaction. A prime example is sensor based human activity recognition where models must balance accuracy with stringent resource constraints. Yet many deep learning approaches treat temporal sensor signals as black box sequences overlooking spectral temporal structure while demanding excessive computation. We present SPECTRA a deployment first co designed spectral temporal architecture that integrates short time Fourier transform STFT feature extraction depthwise separable convolutions and channel wise self attention to capture spectral temporal dependencies under real edge runtime and memory constraints. A compact bidirectional GRU with attention pooling summarizes within window dynamics at low cost reducing downstream model burden while preserving accuracy. Across five public HAR datasets SPECTRA matches or approaches larger CNN LSTM and Transformer baselines while substantially reducing parameters latency and energy. Deployments on a Google Pixel 9 smartphone and an STM32L4 microcontroller further demonstrate end to end deployable realtime private and efficient HAR.

Figures & tables

Fig. 1 : SPECTRA architecture: signal-informed spectral–temporal modeling achieves competitive HAR accuracy with a compact, shallow pipeline. Legend: input window $X\in\mathrm{R}^{T\times C}$ ( $T$ samples/window, $C$ IMU channels/axes, dataset-dependent); STFT magnitudes $M\in\mathrm{R}^{L\times F\times C}$ ( $L$ STFT frames, $F=n_{\mathrm{fft}}/2+1$ freq bins); separable conv maps to $L\times F\times D$ features ( $D$ conv feature dim); channel self-attention operates over $C$ channels per frame; Bi-GRU with hidden size $H$ and attention pooling aggregates over $L$ to a $2H$ vector; classifier outputs $\hat{\mathbf{y}}$ over $K$ activities.

Dataset	Model	F1 $\pm$ std (%) ( $\uparrow$ )	K ( $\downarrow$ )	MACs ( $\downarrow$ )	ms ( $\downarrow$ )	samples/s ( $\uparrow$ )	Mem ( $\downarrow$ )	J ( $\downarrow$ )	J/sample ( $\downarrow$ )
WISDM	DeepConvLSTM	83.2 $\pm$ 0.3	227654	9940678	10.34	96.7	76.7	150.3	0.00587
	TinyHAR	84.1 $\pm$ 0.3	25028	1144056	4.70	212.8	33.9	113.4	0.00443
	TinierHAR	82.1 $\pm$ 0.6	4981	199223	2.48	403.2	39.8	46.7	0.00183
	SPECTRA (FP32)	83.6 $\pm$ 0.5	8892	28530	$0.52{+}0.96$	671.1	26.9	12.1	0.00047
	SPECTRA (FP16)	82.7 $\pm$ 0.4	8892	28530	$0.18{+}0.35$	1850.4	18.7	0.726	0.00007
	SPECTRA (INT8)	82.0 $\pm$ 0.6	8892	28530	$0.28{+}0.52$	1241.2	18.7	1.211	0.00012

TABLE I : Results on PC workstation across five datasets. Metrics reported: F1-score (

\uparrow

), parameters (K,

\downarrow

), multiply accumulate operations (MACs,

\downarrow

), inference latency (ms,

\downarrow

), throughput (samples/s,

\uparrow

), peak memory (MEM,

\downarrow

), total energy (J,

\downarrow

), and energy per sample (J/sample,

\downarrow

). For SPECTRA variants, latency is reported as STFT feature transformation time

+

neural backbone time (ms); other models report directly total latency. Best results are in bold , second best are underlined .

Dataset	Model	Latency (ms $\downarrow$ )	samples/s ( $\uparrow$ )	Mem (KB $\downarrow$ )	Energy (mJ $\downarrow$ )	mJ/sample ( $\downarrow$ )
WISDM	TinyHAR	5.2	192.3	121.5	13,838.0	14.8
	TinierHAR	2.6	384.6	54.0	5,797.0	6.2
	DConvLSTM	11.4	87.7	762.0	27,021.5	28.9
	SPECTRA (FP32)	1.7	588.2	27.0	3,833.5	4.1
	SPECTRA (FP16)	1.4	692.0	24.3	3,258.5	3.5
	SPECTRA (INT8)	1.2	840.3	21.6	2,683.4	2.9

TABLE II : Deployment results on Google Pixel 9 (ExecuTorch, float32, float16 and INT8). Metrics reported: parameters (K,

\downarrow

), multiply accumulate operations (MACs,

\downarrow

), inference latency (ms,

\downarrow

), throughput (samples/s,

\uparrow

), peak memory (KB,

\downarrow

), total energy (mJ,

\downarrow

), and energy per sample (mJ/sample,

\downarrow

). Best results are in bold , second best are underlined .

Dataset	Model	Latency (ms $\downarrow$ )	samples/s ( $\uparrow$ )	Flash (KB $\downarrow$ )	RAM (KB $\downarrow$ )	Energy (µJ $\downarrow$ )	µJ/sample ( $\downarrow$ )
WISDM	TinyHAR	4.463	224.0	115.112	80.32	105,433.614	112.8
	DeepConvLSTM	9.863	101.4	741.22	5.04	145,061.423	155.2
	SPECTRA (FP32)	1.071	933.5	7.98	3.29	184.293	0.197
	SPECTRA (INT8)	0.803	1245.6	2.79	3.29	110.576	0.118
USC-HAD	TinyHAR	6.299	158.8	112.32	80.32	141,145.933	239.2
	DeepConvLSTM	17.695	56.5	1,036.54	5.04	258,518.654	438.3

TABLE III : Deployment results on STM32L4S5VIT6 microcontroller (ONNX

\rightarrow

TFLM, float32 and INT8, float16 is not supported by STM32). Metrics reported: inference latency (ms), throughput (samples/s), Flash and RAM memory footprint (KB), total energy (µJ), and energy per sample (µJ/sample). Best results are in bold , second best are underlined .

Variant	WISDM	USC-HAD	UCI HAR	DSADS	PAMAP2
Standard SPECTRA	83.6 / 8892 / 28530	71.2 / 10050 / 51462	94.8 / 10812 / 113112	87.6 / 22761 / 481551	75.0 / 13890 / 235539
ine w/o STFT	78.8 / 4691 / 438152	61.6 / 5113 / 670454	91.1 / 5651 / 1366438	80.8 / 11632 / 6093494	70.1 / 7033 / 3241008
w/ FFT	80.0 / 5831 / 8895	65.3 / 6989 / 14253	93.2 / 7751 / 19215	84.9 / 19700 / 81564	74.6 / 10829 / 34893
w/ CWT	84.8 / 5047 / 507498	65.2 / 6781 / 661296	92.2 / 8119 / 1042758	85.3 / 26980 / 3322752	80.5 / 12925 / 2142900
w/ MFCC	70.6 / 4511 / 46083	32.6 / 5669 / 74481	88.4 / 6431 / 143397	81.4 / 18380 / 529449	63.0 / 9509 / 336789
w/o separable conv	82.4 / 8003 / 18544	68.5 / 9045 / 33450	94.6 / 9731 / 73523	86.0 / 20485 / 313008	73.9 / 12501 / 153100

TABLE IV : Ablation study of SPECTRA across five datasets. Each cell reports F1 / K / MACs (F1 in %, K parameters, MACs in M). New ablations added: separable vs. standard convolution and attention-type variants.

Dataset	Frac.	SPECTRA	T.HAR	Tr.HAR	DC.LSTM
WISDM	25%	81.7	80.2	80.2	80.1
	50%	82.1	82.1	82.1	81.6
	75%	82.7	82.5	82.5	83.0
	90%	83.2	82.8	82.8	82.7
	100%	83.6	84.1	82.1	83.2
USC-HAD	25%	68.8	73.5	67.8	69.8

TABLE V : Macro F1-score (in %) across training fractions for different models and datasets. Best results are in bold , second best are underlined .

Explore similar work

EdgeHAR: An Edge-Native Compact Sensor Foundation Model for Human Activity Recognition
Sep 13, 2026He Zhang, Siyu Yuan, Siyu Liu +2Human Activity Recognition Wearable Sensing
Northwestern Polytechnical University Xi’an, Shaanxi, China · RPTU University Kaiserslautern-Landau Kaiserslautern, Rheinland Pfalz, Germany · Xi’an University of Technology Xi’an, Shaanxi, China
LITEWAY: LIghtweight HAR via Temporal Efficient highWAY
Aug 10, 2026Dominique Nshimyimana, Vitor Fortes Rey, Mengxi Liu +2Efficient Neural Network Inference Human Activity Recognition
RPTU, DFKI Kaiserslautern, Germany
Towards Green Wearable Computing: A Physics-Aware Spiking Neural Network for Energy-Efficient IMU-based Human Activity Recognition
Apr 12, 2026Naichuan Zheng, Hailun Xia, Zepeng Sun +2Spiking Neural Networks Human Activity Recognition
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, China · International School, Beijing University of Posts and Telecommunications, Beijing, China

Sep 13, 2026J/K move · Enter open · S save

EdgeHAR: An Edge-Native Compact Sensor Foundation Model for Human Activity Recognition

He Zhang, Siyu Yuan, Siyu Liu, Sizhen Bian+1

Northwestern Polytechnical University Xi’an, Shaanxi, China · RPTU University Kaiserslautern-Landau Kaiserslautern, Rheinland Pfalz, Germany · Xi’an University of Technology Xi’an, Shaanxi, China

Human Activity Recognition Wearable Sensing Edge ML Domain Generalization Disentangled Representation Learning

Open paper AskPDF arXiv

Sensor-based human activity recognition (HAR) is fundamental to ubiquitous and wearable computing, yet existing foundation models are largely designed for cloud-scale deployment and struggle with real-world sensing shifts, including unseen users, devices, sampling rates, and sensor placements. We present EdgeHAR, an edge-native compact sensor foundation model designed for wearable intelligence. Unlike conventional models that entangle activity knowledge with acquisition variations, EdgeHAR learns transferable representations by factorizing sensor signals into three latent codes: an (i)Activity-Semantic Code capturing reusable activity knowledge, a (ii)Motion-Dynamics Code modeling temporal patterns, and an (iii)Acquisition-Context Code representing sensor-specific variations. This disentangled design enables efficient adaptation to new users, devices, placements, and activity classes with limited target-domain data. By incorporating lightweight adaptation modules, EdgeHAR achieves foundation-model-level transferability while satisfying edge constraints in computation, memory, latency, and privacy. Experiments across heterogeneous HAR datasets demonstrate that EdgeHAR maintains competitive recognition performance under distribution shifts with substantially reduced deployment cost. EdgeHAR establishes a practical paradigm for compact, edge-first foundation models for ubiquitous sensing systems.

Abstract

Explore similar work

EdgeHAR: An Edge-Native Compact Sensor Foundation Model for Human Activity Recognition

LITEWAY: LIghtweight HAR via Temporal Efficient highWAY

Towards Green Wearable Computing: A Physics-Aware Spiking Neural Network for Energy-Efficient IMU-based Human Activity Recognition