Timing-Aware and Interpretable Self-Attention for Event-Based Wearable Task Transition Detection

Anda Ouyang; Siyuan Chen; Julien Epps

cs.LGOct 20, 2026

Timing-Aware and Interpretable Self-Attention for Event-Based Wearable Task Transition Detection

Authors: Anda Ouyang, Siyuan Chen, Julien Epps

Organizations: School of Electrical Engineering and Telecommunications, University of New South Wales, Kensington, NSW 2033, Australia

Abstract

Task transition detection aims to identify when a person shifts from one task to another, providing a foundation for continuous health monitoring and timely context-aware interventions. While statistical and deep feature representations have demonstrated strong classification performance, they often rely on aggregated or latent features that are difficult to interpret in behavioural terms. Event-based features provide a compact and interpretable alternative by encoding behavioural and physiological patterns as discrete tokens with explicit semantic meaning, making them well suited for behavioural analysis in health informatics. Self-attention offers a mechanism for modelling interactions among such tokens; however, conventional self-attention classifiers primarily rely on token order and do not explicitly incorporate timing or duration information. As a result, informative temporal cues in event representations may not be fully exploited for task transition detection. This study proposes an event-based time-aware self-attention framework that represents behavioural signals as sequences of discrete events augmented with onset timestamps and durations. Event timing is encoded using a parameter-efficient kernel-based representation, while a novel type-aware duration embedding modulates information propagation according to event persistence. Experiments on a semi-naturalistic wearable dataset show that the proposed model outperforms existing event-based approaches, achieves performance beyond or comparable with statistical and deep-learning models with a more compact input representation, and provides improved interpretability through integrated-gradient-based analysis. These results indicate that explicit modelling of event timing and duration enhances both classification performance and interpretability, supporting reliable deployment in biomedical and health informatics applications.

Figures & tables

Fig. 1: Overview of the proposed EvtTiSA system that combines events, event timestamps, and event duration information in attention. The structure inside the light grey box corresponds to a conventional self-attention classifier (with the skip connection from $X$ to $H$ removed to enable integrated-gradient-based attribution analysis). EvtTiSA (1) replaces order positional embeddings with timing positional embeddings to capture temporal interactions between events, (2) incorporates event duration information $D$ into the value matrix $V$ , and (3) enables post-hoc IG-based Attribution Analysis for attention mechanism interpretability.

Event Type	Abbrev.	Event Type	Abbrev.
$head\_central$	$h0$	$pupil\_size\_central$	$p0$
$head\_increase$	$h+$	$pupil\_size\_increase$	$p+$
$head\_decrease$	$h-$	$pupil\_size\_decrease$	$p-$
$is\_blink$	$b+$	$is\_saccade$	$s+$
$not\_blink$	$b-$	$not\_saccade$	$s-$

TABLE I: Event Token Abbreviations

Fig. 2: Converting continuous-valued sensor-derived feature streams into temporal events. (a) Pupil center position time series (pixels). (b) Head angular velocity time series (deg/s). (c) Uniform-time binary-valued event indicator sequences ( $0/1$ ; false/true for each event type at each time sample), extracted from the continuous signals in (a) and (b); contiguous runs of ones correspond to event instances. (d) Temporal event sequence obtained by grouping the indicator sequences in (c), serving as the input to the proposed EvtTiSA. Each event instance is represented as a tuple $(e_{i},t_{i},d_{i})$ , where $e_{i}$ is the event type, $t_{i}$ is the onset timestamp, and $d_{i}$ is the duration; the model input is the ordered sequence $\{(e_{i},t_{i},d_{i})\}_{i=1}^{N}$ in ascending temporal order of $t_{i}$ .

Fig. 3: AUC performance comparison of Mercer and sinusoidal encoding for timing and order information, and no position embedding baseline, with the variation of embedding dimensions. The solid lines are the average performances across 15 different random seeds, and the light colour regions are $\pm$ standard deviation. All models, except for the ”event indicator seq” (uniform sampled event indicator sequence), use only event onsets occurring at irregular (non-uniform) timestamps. In contrast, the uniform timestep model uses the full event sequence sampled at a fixed rate of 60 Hz.

Fig. 4: Attribution insights, where a) is the integrated-gradient-based attribution heatmap of a non-transition sliding window, b) is the duration of each event onset of the same sliding window, and c) is the timestamps of each event onset of the same sliding window. d) e) f) are those of a transition sliding window

Event Type	Attribution	Event Type	Attribution
$h0$	$-0.348$	$p0$	$-0.051$
$h+$	$~~0.286$	$p+$	$~~0.050$
$h-$	$~~0.157$	$p-$	$~~0.024$
$s-$	$-0.112$	$b-$	$~~0.003$
$b+$	$~~0.102$	$s+$	$~~0.001$

TABLE II: Average attribution per sliding window (

y=0.424+\Delta y\in[0,1]

), aggregated by event type.

Fig. 5: Effect of attribution-guided feature pruning on AUC performance. Models were trained using either the most contributive event types (high-attribution features) or the least contributive event types (low-attribution features), as determined by Table II . High-attribution features achieve strong performance with few event types, whereas low-attribution features require substantially more event types to approach comparable performance.

Feature Type	Model	AUC	Accuracy	F1-Score	Wilcoxon test
Feature Type	Model	AUC	Accuracy	F1-Score	$p$ -value vs. EvtTiSA
Statistical	SVM [ 10 ]	$0.911(0.035)$	$82.8\%(5.3\%)$	$72.1\%(7.1\%)$	$5.20\times 10^{-1}$
Deep	TST [ 43 ]	$\textbf{0.918}(\textbf{0.032})$	$\textbf{83.1\%}(\textbf{4.8\%})$	$\textbf{74.6\%}(\textbf{5.9\%})$	$8.30\times 10^{-2}$
Deep	LSTM	$0.915(0.032)$	$82.8\%(5.1\%)$	$73.9\%(4.9\%)$	$6.74\times 10^{-2}$
Event + Timing	SVM [ 10 ]	$0.888(0.052)$	$\textbf{82.8\%}~(\textbf{5.5\%})$	$72.1\%(4.7\%)$	$1.86\times 10^{-2}$
Event + Order	Transformer Encoder [ 15 ]	$0.884(0.039)$	$79.0\%(5.6\%)$	$68.1\%(3.4\%)$	$9.77\times 10^{-4}$

TABLE III: Performance comparison of different models on the task transition detection dataset. Values in parentheses indicate the standard deviation over 11 leave-one-participant-out cross validation folds. Among the cited models, only the SVM models were evaluated on this dataset in prior work [ 10 ] ; all other models were freshly implemented and trained under the same experimental protocol for fair comparison. Wilcoxon

p

-values were obtained from paired signed-rank tests comparing each model’s AUC against EvtTiSA across the corresponding cross-validation folds.

Temporal Attention

Temporal attention models how information across time should be selected and combined in video, sequences, and time-series data.

See all 64 papers

HealthCAT: An Interpretable Encoder-only Transformer Framework for Health Indicator Prediction and Temporal Interpretation of Wearable Sensor Data
Jul 30, 2026Xiaotong Yu, Joshua Y. Kim, HaeJin Lee +1Transformer Interpretability Explainable Time Series Forecasting
Temporal Structure Matters for Efficient Test-Time Adaptation in Wearable Human Activity Recognition
May 6, 2026Zishu Zhou, Zaipeng Xie, Xuanyao JieTest-Time Adaptation Human Activity Recognition
College of Computer Science and Software Engineering, Hohai University
Explainable AI Using Inherently Interpretable Components for Wearable-based Health Monitoring
Mar 13, 2026Maurice Kuschel, Solveig Vieluf, Claus Reinsberger +2Explainable Time Series Forecasting Explainable Artificial Intelligence
Signal and System Theory Group, Paderborn University, Paderborn, Germany · Department of Medicine I, Ludwig Maximilian University, Munich, Germany · Konrad Zuse School of Excellence in Reliable AI, Munich, Germany +3

Jul 30, 2026J/K move · Enter open · S save

HealthCAT: An Interpretable Encoder-only Transformer Framework for Health Indicator Prediction and Temporal Interpretation of Wearable Sensor Data

Xiaotong Yu, Joshua Y. Kim, HaeJin Lee, Kalina Yacef

Transformer Interpretability Explainable Time Series Forecasting Explainable Artificial Intelligence Time Series Classification Wearable Health Monitoring Time Series Transformers

Open paper AskPDF arXiv

Wearable sensors continuously capture fine-grained multivariate time-series data, providing opportunities to model behavioural patterns associated with health outcomes. However, existing deep learning methods prioritise predictive accuracy over interpretability, limiting their application in health research. In this study, we present HealthCAT, a flexible framework that integrates an Encoder-only Transformer with an Attentive Class Activation Token (AttentiveCAT) to generate class-specific, time-step-level interpretations. These interpretations can be mapped back onto behavioural cycles that are relevant to the domain (e.g., time-of-day), supporting individual-level analysis of wearable sensor data. We evaluated HealthCAT using two real-world wearable sensor datasets (306 participants in total). HealthCAT outperformed deep learning baselines by up to 17% in F1-score and 12% in accuracy on both datasets (

p<0.05

). In masking experiments, the time steps identified by HealthCAT carried significantly more predictive value than random selection across all masking conditions (

p<0.05

), indicating that the identified time steps are predictively informative. By coupling predictive performance with validated time-step-level interpretability, HealthCAT moves wearable sensor analysis beyond aggregated metrics towards temporal patterns that support health monitoring, behavioural pattern analysis, and intervention design in health research. The significance of this work is that it enables accurate prediction of health indicators from wearable sensor data while providing insights into when and how physical activity patterns occur, rather than relying solely on aggregated summary measures.

Abstract

Temporal Attention

Explore similar work

HealthCAT: An Interpretable Encoder-only Transformer Framework for Health Indicator Prediction and Temporal Interpretation of Wearable Sensor Data

Temporal Structure Matters for Efficient Test-Time Adaptation in Wearable Human Activity Recognition

Explainable AI Using Inherently Interpretable Components for Wearable-based Health Monitoring