Timing-Aware and Interpretable Self-Attention for Event-Based Wearable Task Transition Detection

Anda Ouyang; Siyuan Chen; Julien Epps

cs.LGOct 20, 2026

Timing-Aware and Interpretable Self-Attention for Event-Based Wearable Task Transition Detection

Authors: Anda Ouyang, Siyuan Chen, Julien Epps

Organizations: School of Electrical Engineering and Telecommunications, University of New South Wales, Kensington, NSW 2033, Australia

Abstract

Task transition detection aims to identify when a person shifts from one task to another, providing a foundation for continuous health monitoring and timely context-aware interventions. While statistical and deep feature representations have demonstrated strong classification performance, they often rely on aggregated or latent features that are difficult to interpret in behavioural terms. Event-based features provide a compact and interpretable alternative by encoding behavioural and physiological patterns as discrete tokens with explicit semantic meaning, making them well suited for behavioural analysis in health informatics. Self-attention offers a mechanism for modelling interactions among such tokens; however, conventional self-attention classifiers primarily rely on token order and do not explicitly incorporate timing or duration information. As a result, informative temporal cues in event representations may not be fully exploited for task transition detection. This study proposes an event-based time-aware self-attention framework that represents behavioural signals as sequences of discrete events augmented with onset timestamps and durations. Event timing is encoded using a parameter-efficient kernel-based representation, while a novel type-aware duration embedding modulates information propagation according to event persistence. Experiments on a semi-naturalistic wearable dataset show that the proposed model outperforms existing event-based approaches, achieves performance beyond or comparable with statistical and deep-learning models with a more compact input representation, and provides improved interpretability through integrated-gradient-based analysis. These results indicate that explicit modelling of event timing and duration enhances both classification performance and interpretability, supporting reliable deployment in biomedical and health informatics applications.

Figures & tables

Fig. 1: Overview of the proposed EvtTiSA system that combines events, event timestamps, and event duration information in attention. The structure inside the light grey box corresponds to a conventional self-attention classifier (with the skip connection from $X$ to $H$ removed to enable integrated-gradient-based attribution analysis). EvtTiSA (1) replaces order positional embeddings with timing positional embeddings to capture temporal interactions between events, (2) incorporates event duration information $D$ into the value matrix $V$ , and (3) enables post-hoc IG-based Attribution Analysis for attention mechanism interpretability.

Event Type	Abbrev.	Event Type	Abbrev.
$head\_central$	$h0$	$pupil\_size\_central$	$p0$
$head\_increase$	$h+$	$pupil\_size\_increase$	$p+$
$head\_decrease$	$h-$	$pupil\_size\_decrease$	$p-$
$is\_blink$	$b+$	$is\_saccade$	$s+$
$not\_blink$	$b-$	$not\_saccade$	$s-$

TABLE I: Event Token Abbreviations

Fig. 2: Converting continuous-valued sensor-derived feature streams into temporal events. (a) Pupil center position time series (pixels). (b) Head angular velocity time series (deg/s). (c) Uniform-time binary-valued event indicator sequences ( $0/1$ ; false/true for each event type at each time sample), extracted from the continuous signals in (a) and (b); contiguous runs of ones correspond to event instances. (d) Temporal event sequence obtained by grouping the indicator sequences in (c), serving as the input to the proposed EvtTiSA. Each event instance is represented as a tuple $(e_{i},t_{i},d_{i})$ , where $e_{i}$ is the event type, $t_{i}$ is the onset timestamp, and $d_{i}$ is the duration; the model input is the ordered sequence $\{(e_{i},t_{i},d_{i})\}_{i=1}^{N}$ in ascending temporal order of $t_{i}$ .

Fig. 3: AUC performance comparison of Mercer and sinusoidal encoding for timing and order information, and no position embedding baseline, with the variation of embedding dimensions. The solid lines are the average performances across 15 different random seeds, and the light colour regions are $\pm$ standard deviation. All models, except for the ”event indicator seq” (uniform sampled event indicator sequence), use only event onsets occurring at irregular (non-uniform) timestamps. In contrast, the uniform timestep model uses the full event sequence sampled at a fixed rate of 60 Hz.

Fig. 4: Attribution insights, where a) is the integrated-gradient-based attribution heatmap of a non-transition sliding window, b) is the duration of each event onset of the same sliding window, and c) is the timestamps of each event onset of the same sliding window. d) e) f) are those of a transition sliding window

Event Type	Attribution	Event Type	Attribution
$h0$	$-0.348$	$p0$	$-0.051$
$h+$	$~~0.286$	$p+$	$~~0.050$
$h-$	$~~0.157$	$p-$	$~~0.024$
$s-$	$-0.112$	$b-$	$~~0.003$
$b+$	$~~0.102$	$s+$	$~~0.001$

TABLE II: Average attribution per sliding window (

y=0.424+\Delta y\in[0,1]

), aggregated by event type.

Fig. 5: Effect of attribution-guided feature pruning on AUC performance. Models were trained using either the most contributive event types (high-attribution features) or the least contributive event types (low-attribution features), as determined by Table II . High-attribution features achieve strong performance with few event types, whereas low-attribution features require substantially more event types to approach comparable performance.

Feature Type	Model	AUC	Accuracy	F1-Score	Wilcoxon test
Feature Type	Model	AUC	Accuracy	F1-Score	$p$ -value vs. EvtTiSA
Statistical	SVM [ 10 ]	$0.911(0.035)$	$82.8\%(5.3\%)$	$72.1\%(7.1\%)$	$5.20\times 10^{-1}$
Deep	TST [ 43 ]	$\textbf{0.918}(\textbf{0.032})$	$\textbf{83.1\%}(\textbf{4.8\%})$	$\textbf{74.6\%}(\textbf{5.9\%})$	$8.30\times 10^{-2}$
Deep	LSTM	$0.915(0.032)$	$82.8\%(5.1\%)$	$73.9\%(4.9\%)$	$6.74\times 10^{-2}$
Event + Timing	SVM [ 10 ]	$0.888(0.052)$	$\textbf{82.8\%}~(\textbf{5.5\%})$	$72.1\%(4.7\%)$	$1.86\times 10^{-2}$
Event + Order	Transformer Encoder [ 15 ]	$0.884(0.039)$	$79.0\%(5.6\%)$	$68.1\%(3.4\%)$	$9.77\times 10^{-4}$

TABLE III: Performance comparison of different models on the task transition detection dataset. Values in parentheses indicate the standard deviation over 11 leave-one-participant-out cross validation folds. Among the cited models, only the SVM models were evaluated on this dataset in prior work [ 10 ] ; all other models were freshly implemented and trained under the same experimental protocol for fair comparison. Wilcoxon

p

-values were obtained from paired signed-rank tests comparing each model’s AUC against EvtTiSA across the corresponding cross-validation folds.

Temporal Attention

Temporal attention models how information across time should be selected and combined in video, sequences, and time-series data.

See all 64 papers

Jul 30, 2026

HealthCAT: An Interpretable Encoder-only Transformer Framework for Health Indicator Prediction and Temporal Interpretation of Wearable Sensor Data

Wearable sensors continuously capture fine-grained multivariate time-series data, providing opportunities to model behavioural patterns associated with health outcomes. However, existing deep learning methods prioritise predictive accuracy over interpretability, limiting their application in health research. In this study, we present HealthCAT, a flexible framework that integrates an Encoder-only Transformer with an Attentive Class Activation Token (AttentiveCAT) to generate class-specific, time-step-level interpretations. These interpretations can be mapped back onto behavioural cycles that are relevant to the domain (e.g., time-of-day), supporting individual-level analysis of wearable sensor data. We evaluated HealthCAT using two real-world wearable sensor datasets (306 participants in total). HealthCAT outperformed deep learning baselines by up to 17% in F1-score and 12% in accuracy on both datasets (

p<0.05

). In masking experiments, the time steps identified by HealthCAT carried significantly more predictive value than random selection across all masking conditions (

p<0.05

), indicating that the identified time steps are predictively informative. By coupling predictive performance with validated time-step-level interpretability, HealthCAT moves wearable sensor analysis beyond aggregated metrics towards temporal patterns that support health monitoring, behavioural pattern analysis, and intervention design in health research. The significance of this work is that it enables accurate prediction of health indicators from wearable sensor data while providing insights into when and how physical activity patterns occur, rather than relying solely on aggregated summary measures.

Xiaotong Yu, Joshua Y. Kim, HaeJin Lee +1

Transformer Interpretability Explainable Time Series Forecasting Explainable Artificial Intelligence Time Series Classification Wearable Health Monitoring Time Series Transformers

Ask PDF

May 6, 2026

Temporal Structure Matters for Efficient Test-Time Adaptation in Wearable Human Activity Recognition

Wearable human activity recognition (WHAR) models often suffer from performance degradation under real-world cross-user distribution shifts. Test-time adaptation (TTA) mitigates this degradation by adapting models online using unlabeled test streams, yet existing methods largely inherit assumptions from vision tasks and underexploit the inherent inter-window temporal structure in WHAR streams. In this paper, we revisit such temporal structure as a feature-conditioned inference signal rather than merely an output-space smoothing prior. We derive the insight that temporal continuity and observation-induced feature deviations provide complementary cues for determining when to preserve or release temporal inertia and where to route prediction refinement during likely transitions. Building upon this insight, we propose SIGHT, a lightweight and backpropagation-free TTA framework for WHAR, enabling real-time edge deployment. SIGHT estimates predictive surprise by comparing the current feature with a prototype-based expected state, and then uses the resulting feature deviation to guide geometry-aware transition routing based on prototype alignment and stream-level marginal habit tracking. Evaluations on real-world datasets confirm that SIGHT outperforms existing TTA baselines while reducing computational and memory costs.

Zishu Zhou, Zaipeng Xie, Xuanyao Jie

College of Computer Science and Software Engineering, Hohai University

Test-Time Adaptation Human Activity Recognition Wearable Sensing

Ask PDF

Mar 13, 2026

Explainable AI Using Inherently Interpretable Components for Wearable-based Health Monitoring

The use of wearables in medicine and wellness, enabled by AI-based models, offers tremendous potential for real-time monitoring and interpretable event detection. Explainable AI (XAI) is required to assess what models have learned and build trust in model outputs, for patients, healthcare professionals, model developers, and domain experts alike. Explaining AI decisions made on time-series data recorded by wearables is especially challenging due to the data's complex nature and temporal dependencies. Too often, explainability using interpretable features leads to performance loss. We propose a novel XAI method that combines explanation spaces and concept-based explanations to explain AI predictions on time-series data. By using Inherently Interpretable Components (IICs), which encapsulate domain-specific, interpretable concepts within a custom explanation space, we preserve the performance of models trained on time series while achieving the interpretability of concept-based explanations based on extracted features. Furthermore, we define a domain-specific set of IICs for wearable-based health monitoring and demonstrate their usability in real applications, including state assessment and epileptic seizure detection.

Maurice Kuschel, Solveig Vieluf, Claus Reinsberger +2

Signal and System Theory Group, Paderborn University, Paderborn, Germany · Department of Medicine I, Ludwig Maximilian University, Munich, Germany · Konrad Zuse School of Excellence in Reliable AI, Munich, Germany +3

Explainable Time Series Forecasting Explainable Artificial Intelligence Interpretable ML Concept-Based Explanations Wearable Health Monitoring

Ask PDF

Abstract

Temporal Attention

Explore similar work

HealthCAT: An Interpretable Encoder-only Transformer Framework for Health Indicator Prediction and Temporal Interpretation of Wearable Sensor Data

Temporal Structure Matters for Efficient Test-Time Adaptation in Wearable Human Activity Recognition

Explainable AI Using Inherently Interpretable Components for Wearable-based Health Monitoring