A Neuro-Symbolic System for Interpretable Multimodal Physiological Signals Integration in Human Fatigue Detection

Mohammadreza Jamalifard; Yaxiong Lei; Parasto Azizinezhad; Javier Fumanal-Idocin; Javier Andreu-Perez

cs.HCMar 25, 2026

A Neuro-Symbolic System for Interpretable Multimodal Physiological Signals Integration in Human Fatigue Detection

Authors: Mohammadreza Jamalifard, Yaxiong Lei, Parasto Azizinezhad, Javier Fumanal-Idocin, Javier Andreu-Perez

Organizations: School of Computer Science and Electronic Engineering, University of Essex, United Kingdom

Abstract

We propose a neuro-symbolic architecture that learns four interpretable physiological concepts, oculomotor dynamics, gaze stability, prefrontal hemodynamics, and multimodal, from eye-tracking and neural hemodynamics, functional near-infrared spectroscopy, (fNIRS) windows using attention-based encoders, and combines them with differentiable approximate reasoning rules using learned weights and soft thresholds, to address both rigid hand-crafted rules and the lack of subject-level alignment diagnostics. We apply this system to fatigue classification from multimodal physiological signals, a domain that requires models that are accurate and interpretable, with internal reasoning that can be inspected for safety-critical use. In leave-one-subject-out evaluation on 18 participants (560 samples), the method achieves 72.1% +/- 12.3% accuracy, comparable to tuned baselines while exposing concept activations and rule firing strengths. Ablations indicate gains from participant-specific calibration (+5.2 pp), a modest drop without the fNIRS concept (-1.2 pp), and slightly better performance with Lukasiewicz operators than product (+0.9 pp). We also introduce concept fidelity, an offline per-subject audit metric from held-out labels, which correlates strongly with per-subject accuracy (r=0.843, p < 0.0001).

Figures & tables

Component	Core details
Eye-tracking	EyeLink 1000 Plus (2000 Hz); timestamps reconstructed to remove LSL jitter/duplicates.
fNIRS	8-channel CW; bandpass 0.01–0.2 Hz; flat channels ( $\sigma<10^{-10}$ ) removed.
Pupil preprocessing	Blink detection at $2.5\sigma$ below median, 50 ms dilation, linear interpolation; bandpass 0.01–4.0 Hz.
Windows	Modalities aligned to 10 Hz; 10 s windows with 50% overlap; eye features computed at native rate then aggregated.
Labels	Alert (baseline) vs. Fatigued (post-induction); 560 samples (280/280).

TABLE I: Acquisition and preprocessing summary

Modality	Feature Groups & Metrics	Dim.
Eye: Pupil	Stat: $\mu,\sigma$ , range, skew, kurtosis. Dyn: $\mu,\sigma,\max$ of $\|\dot{p}\|$ and $\|\ddot{p}\|$ . Spec: Bandpower (LF, HF, Ratio), SampEn, Coeff. Var	16
Eye: Oculomotor	Dispersion: $\sigma_{x},\sigma_{y}$ , corr $(x,y)$ , spatial $H$ . Kinematics: Velocity $v$ ( $\mu,\sigma,\max,P_{90}$ ), accel. stats. Events: Saccade rate, fixation prop. Trend: Linear fit slope $(x,y)$ , angular $\Delta$ stats, SampEn $(v)$ .	18
Eye: Eyelid	Blink: Rate, duration/IBI stats ( $\mu,\sigma,\min/\max$ ), Percentage of Eye Closure (PERCLOS) (total & weighted).	8
fNIRS	Global: $\mu,\sigma$ , skew, range of channel-mean & deriv. Spec: VLF, LF, HF powers & ratios. Regional: $\mu,\sigma$ , SampEn per ROI (8 groups). Sym: L/R diff stats ( $\mu,\sigma,\nabla$ ), corr(L,R), A/P contrast. Cmplx: Global SampEn, Hurst exp., outlier prop.	48

TABLE II: Summary of extracted multimodal features (90 dims total).

Panel of Figure 3 — Fig. 1: Neuro-symbolic architecture: participant-normalized features $\to$ concept bottleneck (C1–C4) $\to$ differentiable Logic Rules $\to$ fatigue score.

Rule	Antecedent $\rightarrow$ Consequent	Firing Strength
$R_{1}$	IF $\tilde{C}_{1}$ is HIGH THEN Fatigue is LIKELY	$f_{1}=\beta_{1}\cdot\tilde{C}_{1}$
$R_{2}$	IF ( $\tilde{C}_{2}$ is HIGH) OR ( $\tilde{C}_{3}$ is HIGH) THEN Fatigue is LIKELY	$f_{2}=\beta_{2}\cdot(\tilde{C}_{2}\oplus\tilde{C}_{3})$
$R_{3}$	IF (Weighted Concept Sum) is HIGH THEN Fatigue is LIKELY	$f_{3}=\beta_{3}\cdot\left(\sum_{i=1}^{4}\alpha_{i}\tilde{C}_{i}\right)$

TABLE III: Differentiable approximate reasoning rule base with learned weights. Each rule maps concept activations to fatigue evidence.

Tuned Baselines (nested CV):
Method	Accuracy (%) †	95% CI
NeSy (Ours)	72.1 $\pm$ 12.3	[66.4, 77.8]
No Logic (Ablation)	72.0 $\pm$ 12.1	[66.4, 77.6]
SVM-RBF	69.4 $\pm$ 13.5	[63.2, 75.7]
Random Forest	66.6 $\pm$ 12.3	[60.9, 72.3]
Extra Trees	68.3 $\pm$ 14.3	[61.7, 74.9]

TABLE IV: Classification Accuracy (LOSO Cross-Validation)

∗ Population statistics estimated from the training cohort; may not transfer to unseen users
Strategy	Accuracy (%)	Std (%)
Global ∗	80.3	10.1
Participant-Aware (Calibration) †	72.1	12.3
No-Calibration (Train-Only)	66.9	8.3
† Primary configuration; calibration, no test labels used

TABLE V: Ablation: Normalization Strategies

Configuration	Accuracy (%)	$\Delta$ (pp)
Full Model (Calibration)	72.1	–
Without $C_{1}$ (Oculomotor Dynamics)	73.4	+1.3
Without $C_{2}$ (Gaze Stability)	72.4	+0.3
Without $C_{3}$ (Prefrontal Hemodynamics)	70.9	-1.2
Without $C_{4}$ (Multimodal)	71.2	-0.9

TABLE VI: Ablation: Individual Concept Contributions

Operator	Accuracy (%)	Std (%)	$\Delta$
Product	72.0	12.0	–
Łukasiewicz	72.9	12.2	+0.9
Gödel (min/max)	72.2	12.3	+0.2

TABLE VII: Ablation: Logic Operators

Configuration	Accuracy (%)	Std (%)
Learned Thresholds	72.1	12.3
Fixed Thresholds ( $\tau=0.5$ )	70.6	12.2

TABLE VIII: Ablation: Learned vs. Fixed Thresholds

ID	Acc	C-Fid	R-Fid	$R_{1}$	$R_{2}$	$R_{3}$
P007	96.7	0.75	0.83	-0.03	+0.09	-0.01
P006	86.4	0.55	0.56	-0.05	+0.10	-0.00
P010	85.0	0.46	0.48	-0.05	+0.10	-0.00
P018	85.0	0.44	0.61	-0.04	+0.06	-0.01
P011	83.5	0.45	0.44	-0.04	+0.08	-0.00
P016	81.4	0.48	0.47	+0.01	+0.05	+0.00

TABLE IX: Per-subject fidelity and rule discrimination

Fig. 2: Case study (P007): fatigue prediction and labels (top), concept activations

C_{1}

–

C_{4}

(middle), and rule

R_{2}

firing (bottom); dashed line marks the Alert

\rightarrow

Fatigued transition.

Multimodal Model Interpretability

Multimodal model interpretability studies how AI systems combine and use images, text, audio, video, and other signals, and how to explain their decisions.

See all 122 papers

Next-Token Prediction Learns Generalisable Representations of Sleep Physiology
Jun 8, 2026Jonathan F. Carter, Lionel TarassenkoSleep Stage Classification Time Series Classification
Institute of Biomedical Engineering University of Oxford
Synheart Capacity: A Theory-Driven Physiological Representation of Cognitive Capacity Dynamics from Wearable Signals
May 23, 2026Yisak Debele, Henok Ademtew, Israel GoytomCognitive Modeling Wearable Sensing
Synheart AI, Vancouver, Canada
SleepPairNet: Learning Multimodal Pediatric Sleep Representations with Random-Pair Cross-Attention
Oct 19, 2026Saurav Raj Pandey, Harlin LeeMultimodal Pretraining Cross-Modal Representation Learning
Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA · School of Data and Information Sciences, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

Jun 8, 2026J/K move · Enter open · S save

Next-Token Prediction Learns Generalisable Representations of Sleep Physiology

Jonathan F. Carter, Lionel Tarassenko

Institute of Biomedical Engineering University of Oxford

Sleep Stage Classification Time Series Classification Self-Supervised Time Series Representation Learning Multimodal Pretraining Next-Token Prediction Multimodal Foundation Models+2

Open paper AskPDF arXiv

Foundation models offer a promising route to compress multi-modal physiological signals into compact representations of human health, with broad applications across sleep medicine, cardiology, neurology and other healthcare domains. Existing models have typically been trained with masked-reconstruction or contrastive objectives. However, masked reconstruction may be poorly suited to the stochastic nature of these signals, while contrastive approaches rely on positive-pair definitions despite the semantic invariances of physiological signals being poorly understood. In this work, we show that next-token prediction is a simple and scalable alternative. We develop Hypnos, a multi-modal sleep foundation model trained using eight different sensing modalities (e.g. EEG, ECG, respiratory signals) drawn from over 20,000 overnight polysomnography recordings. We tokenize each modality into streams of discrete tokens using residual vector quantization, then train a large auto-regressive RQ-Transformer to jointly predict the next token across all modalities in parallel. After training, Hypnos can be applied to continuous streams of sensor data from any subset of supported modalities, generating embeddings for downstream tasks. Across a range of benchmarks, Hypnos significantly outperforms existing foundation models. In sleep stage classification, we match the performance of strong supervised baselines on held-out test sets whilst using

100\times

less labelled data. Hypnos even generalises to daytime physiology, surpassing a dedicated ECG foundation model at detecting atrial fibrillation. Our results demonstrate that next-token prediction is a strong self-supervised objective for representation learning from multi-modal physiological signals.

Abstract

Multimodal Model Interpretability

Explore similar work

Next-Token Prediction Learns Generalisable Representations of Sleep Physiology

Synheart Capacity: A Theory-Driven Physiological Representation of Cognitive Capacity Dynamics from Wearable Signals

SleepPairNet: Learning Multimodal Pediatric Sleep Representations with Random-Pair Cross-Attention