MMTA: Multi Membership Temporal Attention for Fine-Grained Stroke Rehabilitation Assessment

Halil Ismail Helvaci; Justin Huber; Jihye Bae; Sen-ching Samson Cheung

cs.CVMar 1, 2026

MMTA: Multi Membership Temporal Attention for Fine-Grained Stroke Rehabilitation Assessment

Authors: Halil Ismail Helvaci, Justin Huber, Jihye Bae, Sen-ching Samson Cheung

Organizations: Department of Electrical and Computer Engineering, University of Kentucky, Lexington, KY, USA · College of Medicine, University of Kentucky, Lexington, KY, USA · Department of Electrical and Computer Engineering, University of California, Davis, CA, USA

Abstract

To empower the iterative assessments involved during a person's rehabilitation, automated assessment of a person's abilities during daily activities requires temporally precise segmentation of fine-grained actions in therapy videos. Existing temporal action segmentation (TAS) models struggle to capture sub-second micro-movements while retaining exercise context, blurring rapid phase transitions and limiting reliable downstream assessment of motor recovery. We introduce Multi-Membership Temporal Attention (MMTA), a high-resolution temporal transformer for fine-grained rehabilitation assessment. Unlike standard temporal attention, which assigns each frame a single attention context per layer, MMTA lets each frame attend to multiple locally normalized temporal attention windows within the same layer. We fuse these concurrent temporal views via feature-space overlap resolution, preserving competing local contexts near transitions while enabling longer-range reasoning through layer-wise propagation. This increases boundary sensitivity without additional depth or multi-stage refinement. MMTA supports both video and wearable IMU inputs within a unified single-stage architecture, making it applicable to both clinical and home settings. MMTA consistently improves over the Global Attention transformer, boosting Edit Score by +1.3 (Video) and +1.6 (IMU) on StrokeRehab while further improving 50Salads by +3.3. Ablations confirm that performance gains stem from multi-membership temporal views rather than architectural complexity, offering a practical solution for resource-constrained rehabilitation assessment.

Figures & tables

Figure 1: MMTA replaces global attention with locally normalized attention over overlapping temporal windows. Frames in overlap regions receive multiple window-conditioned updates, reconciled via overlap-resolution aggregation.

Backbone	Model	Video		IMU
Backbone	Model	ES $\uparrow$	AER $\downarrow$	ES $\uparrow$	AER $\downarrow$
TCN	MS-TCN* [ 6 ]	60.7	0.408	66.9	0.372
	MS-TCN [ 6 ]	62.2	0.392	68.9	0.330
	MS-TCN + [ 6 ]	62.7	0.390	68.8	0.317
Boundary	ASRF* [ 9 ]	56.9	0.449	68.2	0.328
Boundary	ASRF [ 9 ]	58.7	0.436	67.9	0.349

Table 1: Comparison with state-of-the-art methods on StrokeRehab (top) and 50Salads (bottom). Both transformer baselines share an identical backbone; Global Attention uses full self-attention, while MMTA (ours) replaces it with the proposed module.

Figure 2: Qualitative comparison of predicted (Pred) and ground-truth (GT) action labels on an example test sequence for IMU data (top) and video data (bottom).

Action	Video			IMU
Action	Sens.	Spec.	F1	Sens.	Spec.	F1
Rest	0.70	0.92	0.66	0.69	0.93	0.67
Reach	0.52	0.96	0.59	0.59	0.94	0.64
Retract	0.60	0.97	0.62	0.64	0.97	0.69
Stabilize	0.59	0.91	0.63	0.65	0.90	0.60
Transport	0.77	0.85	0.70	0.72	0.88	0.71

Table 2: Evaluation of frame-level action classification across the StrokeRehab Video and IMU datasets

StrokeRehab
Model	Video		IMU		Window Size	Stride
Model	ES $\uparrow$	AER $\downarrow$	ES $\uparrow$	AER $\downarrow$	Window Size	Stride
MMTA	62.0	0.388	67.6	0.324	1500	500
	66.8	0.332	68.9	0.314	1000	500
	68.2	0.318	66.9	0.331	800	500
	68.7	0.313	70.5	0.295	500	10

Table 3: Ablation study on the impact of window size and stride on the StrokeRehab Video and IMU datasets.

Temporal Action Segmentation

Temporal action segmentation labels when actions begin and end in continuous video, enabling fine-grained analysis of human and robotic activity.

See all 49 papers

Enhancing Clinician Decision-Making via Uncertainty-Aware Multi-Expert Fusion for Stroke Rehabilitation
Jun 23, 2026Tamim Ahmed, Thanassis RikakisClinical Decision-Making Multimodal Model Interpretability
Bioemdical Engineering, University of Soutehrn California,, Los Angeles, 90007, CA, USA.
Combining Boundary Supervision and Segment-Level Regularization for Fine-Grained Action Segmentation
Apr 2, 2026Hinako Mitsuoka, Kazuhiro HottaTemporal Action Segmentation Boundary-Aware Segmentation
Meijo University, 1-501 Shiogamaguchi, Tempaku-ku, Nagoya 468-8502, Japan
Improving Temporal Action Segmentation via Constraint-Aware Decoding
May 11, 2026Yeo Keat Ee, Debaditya Roy, Chen Li +2Temporal Action Segmentation Constrained Decoding
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Indian Institute of Technology Kharagpur, India +1

Jun 23, 2026J/K move · Enter open · S save

Enhancing Clinician Decision-Making via Uncertainty-Aware Multi-Expert Fusion for Stroke Rehabilitation

Tamim Ahmed, Thanassis Rikakis

Bioemdical Engineering, University of Soutehrn California,, Los Angeles, 90007, CA, USA.

Clinical Decision-Making Multimodal Model Interpretability Multimodal Fusion Uncertainty Calibration

Open paper AskPDF arXiv

Tailoring stroke rehabilitation requires assessing how movements are organized, not merely if they succeed. Currently, this assessment is a rate-limiting bottleneck. Instruments like the Action Research Arm Test (ARAT) compress rich behavioral observations into single ordinal endpoints, discarding the movement-quality details that distinguish recovery from compensation. Automated alternatives typically chase accuracy on noisy, single-observer labels to output opaque scores - a technology-centric approach that rarely reaches clinical practice. To address this, we present xAARA: an engine designed to augment rather than replace clinical judgment. From multi-view video, xAARA returns ARAT assessments with calibrated uncertainty and explanations across task, movement-phase, and movement-quality levels. Treating clinical scoring as an ill-posed inference problem, xAARA composes 692 calibrated multimodal models via a Dynamic Bayesian Network with entropy-based gating. It qualifies results against clinical validity rules and defers low-confidence cases. In 105 stroke survivors (788 exercises), xAARA achieved 94.2% task accuracy (Cohen's kappa=0.934) and 81.3% movement-phase accuracy (kappa=0.727), reducing predictive uncertainty by 96.1% compared to single-clinician scoring. For subjective cases, it matched at least one rater 100% of the time and never returned out-of-range scores. Four independent clinicians validated the assessments and indicated willingness to adopt the system. We argue that principled uncertainty quantification and clinician-aligned explainability are the critical bridges moving automated assessment from technical demonstration to a deployable clinical tool.

Abstract

Temporal Action Segmentation

Explore similar work

Enhancing Clinician Decision-Making via Uncertainty-Aware Multi-Expert Fusion for Stroke Rehabilitation

Combining Boundary Supervision and Segment-Level Regularization for Fine-Grained Action Segmentation

Improving Temporal Action Segmentation via Constraint-Aware Decoding