Organizations: School of Computing, The Australian National University, Canberra, ACT, Australia · Computational Media Lab, The Australian National University, Canberra, ACT, Australia · The Australian National University, Canberra, ACT, Australia
User performance is crucial in interactive systems, capturing how effectively users engage with task execution. Prospectively predicting performance enables the timely identification of users struggling with task demands. While ocular and cardiac signals are widely used to characterise performance-relevant visual behaviour and physiological activation, their potential for early prediction and for revealing the physiological mechanisms underlying performance differences remains underexplored. We conducted a within-subject experiment in a game environment with naturally unfolding complexity, using early ocular and cardiac signals to predict later performance and to examine physiological and self-reported group differences. Results show that the ocular-cardiac fusion model achieves a balanced accuracy of 0.86, and the ocular-only model shows comparable predictive power. High performers exhibited targeted gaze and adjusted visual sampling, and sustained more stable cardiac activation as demands intensified, with a more positive affective experience. These findings demonstrate the feasibility of cross-session prediction from early physiology, providing interpretable insights into performance variation and facilitating future proactive intervention.
Figures & tables
Figure 1. Overview of the prospective prediction pipeline. Early ocular and cardiac features from the low-complexity game session are used to predict performance outcomes in the subsequent high-complexity session. Ocular features capture visual behaviour, while cardiac features reflect autonomic arousal. Differences in visual patterns, autonomic profiles, and affective states are observed between high- and low-performing groups. Overview of the prospective prediction pipeline. The figure shows how early physiological signals predict later game performance. A participant plays a PC game while eye tracking and a wrist-worn sensor record visual behaviour and cardiac activity. Example plots in the middle illustrate ocular and cardiac features from the low-complexity session. These early features feed into a prospective prediction step that forecasts outcomes in the later high-complexity session. The right side shows two possible outcomes: high performance, linked to strategic visual attention and stable heart rate, and low performance, linked to narrow visual search and heart-rate instability.
Figure 2. Example game scenes illustrating the two experimental sessions. (a) Low-complexity session, characterised by fewer resources, weaker enemies, and a simple card deck; (b) High-complexity session, featuring more resources available, stronger enemies, and complex cards with intricate effects. Screenshots of game scenes across low-complexity and high-complexity phases. The figure comprises two screenshots from the game, illustrating the core experimental phases. Panel (a) shows the low-complexity session, where the interface displays fewer items, an opponent with lower health, and simple card options. Panel (b) shows the high-complexity session, featuring a boss fight with more items visible, a stronger opponent, and cards with more complex effects.
Figure 3. Eye tracking system (Tobii Pro Fusion). Eye tracker used in the experiment. The figure shows the Tobii Pro Fusion eye tracker, a rectangular bar-shaped device designed to be positioned below a monitor to record eye movements.
Figure 4. Cardiac recording device (Empatica EmbracePlus Wristband). Cardiac recording device used in the experiment. The figure shows the Empatica EmbracePlus, a black wristband with sensors on the inner side, worn on the wrist to collect cardiac signals.
Figure 5. Experimental procedure with annotated session structure and physiological recording. Experimental procedure showing sessions, physiological recordings, and self-reports. The figure presents the sequence of the study. Participants begin with a tutorial of about 3 minutes. They then enter the first game session, lasting about 25 minutes, which contains the low-complexity phase. A second game session of about 10 minutes follows and includes the high-complexity boss fight. Eye-tracking and cardiac signals are recorded continuously across both game sessions. After each session, participants provide subjective difficulty and affective ratings. The procedure ends with a performance outcome classified as win or loss.
Category
Feature Names
Functional Role
Fixation
Fixation Duration, Fixation Count
Periods where gaze remains relatively stable, indicating focused attention.
Saccade
Saccade Amplitude, Saccade Duration, Saccade Velocity, Saccade Count, Saccade to fixation ratio
Rapid eye movements between fixations, reflecting visual scanning and search behaviour.
Pupillometry
Pupil Diameter
Pupil size changes, associated with cognitive effort and emotional arousal.
AOI Features
AOI Hit Rate, AOI Proportion
Distribution and allocation of visual attention across predefined regions of interest on the game interface.
Table 1. Overview of extracted ocular features and their functional roles.
Category
Feature Names
Functional Role
Heart Rate Metrics
Heart Rate, Heart Rate Range
Reflect overall cardiovascular activation.
Heart Rate Variability Metrics
Root Mean Square of Successive Differences (RMSSD), Mean NN Interval
Reflect autonomic nervous system regulation.
Table 2. Overview of extracted cardiac features and their functional roles.
Figure 6. Processing framework integrating ocular, cardiac, and fused modalities. Framework from signal recording to model evaluation. The figure shows two parallel processing streams for ocular and cardiac data. The top stream illustrates the ocular pathway: raw eye-tracking data undergoes artifact correction, filtering, and segmentation. Feature engineering then extracts pupil diameter, saccades, fixations, and gaze measures. These features are used to train an ocular model that outputs a probability of high or low performance. The bottom stream illustrates the cardiac pathway: cardiac signals are denoised, filtered, and segmented. Then, feature engineering extracts blood volume pulse, heart rate, and heart-rate variability. These features train a cardiac model that outputs its own performance probability. Both model outputs are combined through decision-level fusion to generate the final performance prediction of win or loss.
Figure 7. Subjective difficulty ratings across game sessions. Bars represent mean ratings ( ± SD), separated by Low Complexity and High Complexity sessions, with performance groups shown as grouped bars (Win = high-performing; Loss = low-performing). Dots indicate individual participants’ ratings. Asterisks indicate significant difference between sessions (** p<.01 ). Bar chart of complexity ratings by session and outcome. The figure presents self-reported difficulty ratings across two game sessions. Ratings are higher in the high-complexity session compared to the low-complexity session. Within each session, bars are grouped by outcome, showing separate values for win and loss participants. Individual ratings are displayed as dots, and asterisks indicate a significant difference at p<.01.
Figure 8. Subjective ratings on the SAM scales grouped by performance (Win = high-performing; Loss = low-performing). Bars show mean ( ± SD), with dots indicating individual participants. Left: Valence (0 = unpleasant, 5 = pleasant). Middle: Arousal (0 = calm, 5 = aroused). Right: Dominance (0 = low control, 5 = high control). Asterisks indicate significant differences (** p<.01 , * p<.05 ). SAM ratings for valence, arousal, and dominance by complexity and outcome. The figure shows self-assessment manikin ratings across two sessions and outcome groups. For valence, participants in the win group reported higher pleasantness than those in the loss group. For arousal, ratings were significantly higher in the high-complexity session than in the low-complexity session for both performance groups. For dominance, perceived control was significantly lower in the high-complexity session than in the low-complexity session for both performance groups. Significant differences are marked with asterisks at *p<.05 and **p<.01.
Modality
Model
BAcc
Macro-Pr
Macro-Re
Macro-F1
MCC
Ocular
CatBoost
0.83
0.83
0.82
0.83
0.66
XGBoost
0.80
0.77
0.76
0.79
0.54
Linear SVM
0.57
0.79
0.56
0.48
0.27
Cardiac
CatBoost
0.66
0.67
0.66
0.66
0.33
XGBoost
0.70
0.72
0.70
0.70
0.42
Linear SVM
0.50
0.27
0.50
0.35
0.00
Table 3. Cross-subject LOSO performance across the ocular, cardiac, and fused feature sets for all models. The table reports the best subject-level results obtained under LOSO, including balanced accuracy, macro-precision, macro-recall, macro-F1, and MCC.
Figure 9. Cross-subject classification performance (balanced accuracy) comparison across modalities. The grouped bars illustrate the incremental predictive power from unimodal (cardiac, ocular) to multimodal (fused) approaches. CatBoost and XGBoost are shown as the top-performing classifiers Grouped bar chart of model performance (balanced accuracy) comparison across modalities. The figure displays balanced accuracy values for CatBoost and XGBoost across three feature sets: cardiac, ocular, and fused. Each feature set contains two adjacent bars corresponding to the two classifiers. Fused models show the tallest bars overall, followed by ocular models, and cardiac models have the shortest bars. CatBoost bars are consistently higher than XGBoost bars within each feature set.
Figure 10. Confusion matrices for the ocular (left) and fused (right) models, selected as the top-performing classifiers from the cross-subject evaluation. Confusion matrices for ocular vs fused classifiers. The figure shows two square confusion matrices, each with predicted labels on the x-axis and true labels on the y-axis. In both matrices, rows represent actual outcomes (Loss and Win), and columns represent predicted outcomes (Loss and Win). The left matrix corresponds to the ocular model: 12 correct Loss classifications in the top-left cell, 4 misclassified Loss samples in the top-right cell, 2 misclassified Win samples in the bottom-left cell, and 17 correct Win classifications in the bottom-right cell. The right matrix corresponds to the fused model: 14 correct Loss classifications, 2 misclassified Loss samples, 3 misclassified Win samples, and 16 correct Win classifications. Cell values are numerically indicated, with darker shading representing larger counts.
Component Removed
BAcc
Macro F1
MCC
Δ BAcc
\
0.83
0.83
0.66
0.00
Pupil
0.83
0.83
0.66
0.00
Fixation
0.83
0.83
0.66
0.00
Saccade
0.69
0.68
0.37
0.14
AOI Gaze
0.40
0.39
-0.22
0.43
Table 4. Ablation of ocular feature modules, where one component is removed at a time. Δ BAcc denotes the change in balanced accuracy relative to the full ocular model. The row \ corresponds to the full ocular model ( N=35 , LOSO).
Component Removed
BAcc
Macro F1
MCC
Δ BAcc
\
0.86
0.86
0.72
0.00
Consensus Check
0.77
0.77
0.54
0.09
Table 5. Ablation of the consensus check in the stacking architecture, where one component is removed at a time. Δ BAcc denotes the change in balanced accuracy relative to the full fusion architecture. The row \ corresponds to the full fusion architecture ( N=35 , LOSO).
Figure 11. Raincloud plots of cardiovascular features comparing performance groups (Win = high-performing; Loss = low-performing). (A) Mean heart rate. (B) Heart rate range. For all panels, raincloud plots depict individual datapoints (dots), probability density (violin), and boxplots summarising the median and interquartile range. Raincloud plots of cardiovascular features by performance group. The figure compares cardiovascular features between the win and loss groups. Panel A shows mean heart rate, and Panel B shows heart rate range. In all panels, individual participants are shown as dots, the distribution is shown as a violin representing probability density, and a boxplot summarises the median and interquartile range.
Figure 12. Dynamic trends of cardiac features across three phases (tutorial, low-complexity, high-complexity), comparing performance groups (Win = high-performing; Loss = low-performing). Lines show group means with shaded bands indicating ± 1 SEM. Line plots of cardiac features across sessions by outcome group. The figure presents two line plots showing cardiac features across tutorial, low-complexity, and high-complexity sessions for win and loss groups. Solid lines indicate group means, with shaded bands representing variability within groups at plus or minus one standard error of the mean. Asterisks above the markers denote sessions where between-group differences are statistically significant, tested using either independent-samples t-tests or Mann–Whitney U-tests. PanelA shows mean heart rate, and PanelB shows heart rate range.
Figure 13. Raincloud plots of ocular features comparing performance groups (Win = high-performing; Loss = low-performing). (A) AOI gaze proportion: Potions. (B) AOI gaze proportion: Hand Cards. (C) Saccade rate (Hz). (D) Mean saccade amplitude (°). (E) Saccade-to-fixation ratio. For all panels, raincloud plots depict individual datapoints (dots), probability density (violin), and boxplots summarising the median and interquartile range. Asterisks indicate significant differences (* p < .05). Raincloud plots of ocular features comparing performance groups, with high-performing players labelled as Win and low-performing players labelled as Loss. The figure consists of five panels. Panel A shows AOI gaze proportion for Potions. Panel B shows AOI gaze proportion for Hand Cards. Panel C shows saccade rate measured in hertz. Panel D shows mean saccade amplitude measured in degrees. Panel E shows the saccade-to-fixation ratio. In all panels, individual participants are shown as dots, the distribution is shown as a violin representing probability density, and a boxplot summarises the median and interquartile range. Asterisks indicate statistically significant differences between groups at p less than 0.05.
Figure 14. Dynamic trends of ocular features across three phases (tutorial, low-complexity, high-complexity), comparing performance groups (Win = high-performing; Loss = low-performing). Lines show group means with shaded bands indicating ± 1 SEM. Asterisks indicate significant between-group differences at each session (* p < .05). Line plots of ocular indices across game sessions. The figure presents five line plots of ocular indices across three game phases: tutorial, low-complexity, and high-complexity. Panel A shows AOI gaze proportion for Potions. Panel B shows AOI gaze proportion for Hand Cards. Panel C shows saccade rate measured in hertz. Panel D shows mean saccade amplitude measured in degrees. Panel E shows the saccade-to-fixation ratio. Solid green lines show the win group and dashed purple lines show the loss group. Circles indicate win-group means, and squares indicate loss-group means. Asterisks mark statistically significant between-group differences at p<.05.
Ratio of total saccade duration to total fixation duration. Formula: SFR=TfixationTsaccade .
Ratio
Appendix
Table 7. Description and Statistical Metrics of Eye-Tracking Features.
Cardiac Feature
Description with Formula
Statistical Metrics
Heart Rate (HR)
Number of heartbeats per window. Derived from interbeat intervals (IBIs). Formula: HR=IBI60 , where IBI is in seconds.
Mean
Heart Rate Variability (HRV)
Two indices computed from NN intervals. RMSSD=N−11∑i=1N−1(NNi+1−NNi)2MeanNN=N1∑i=1NNNi
Mean
Blood Volume Pulse (BVP) Statistics
Signal statistics across a window. Mean: μ=N1∑i=1Nxi Std: σ=N1∑i=1N(xi−μ)2
Mean, Std
Appendix
Table 8. Description and Statistical Metrics of Cardiovascular Features.
Modality
Complete Extracted Feature Set
Final Features Used
Ocular
Pupil: Left/Right/Avg diameter (Mean, Std, Min, Max); Saccade: Count mean; Rate; Amplitude (Mean, Max); Velocity max; Saccade to fixation ratio; Fixation: Duration (Max, Mean, Sum); AOI: Hand cards (Proportion, Rate); Top bar potion (Proportion, Rate)
Saccade: Count mean; Rate; Amplitude (Mean, Max); Velocity max; Saccade to fixation ratio; AOI: Hand cards (Proportion, Rate); Top bar potion (Proportion, Rate)
Cardiac
BVP Stats: Min; Max; Mean; Std; Variance; RMS; Skewness; Kurtosis; HRV: Mean NN; RMSSD; pNN50; pNN20; Ratio
Heart Rate: Mean Heart Rate; Heart Rate Range
Appendix
Table 9. Complete extracted feature set and final features used for each modality.
Element
Function / Effect
Typical On-Screen Position
Relics
Permanent passive bonuses that affect the run (e.g., extra energy, healing, card draw).
Top-left area of the screen
Potions
One-time use consumable items used during combat to grant effects (e.g., heal, buff, damage).
Top-middle of the combat interface, in potion slots
Hand Cards
The cards the player currently can play, showing available actions (Attacks, Skills, Powers).
Bottom centre of the screen
Enemy Tooltip
Shows what the enemy intends to do next (e.g., attack amount, defense, buffs/debuffs), guiding player strategy.
Above the enemy on the combat screen
Appendix
Table 10. Key interface elements in Slay the Spire with their positions and functions.
Figure 15. Areas of interest (AOIs) defined for gaze feature extraction in Slay the Spire . AOIs included (1) hand cards at the bottom centre (decision options), (2) relics at the top-left (passive bonuses), (3) potions at the top (consumable items), and (4) enemy intent indicators above the opponent, including both the icon and tooltip (upcoming enemy action). These regions provided the basis for deriving gaze features such as fixation proportion and dwell time. AOIs feature illustration The figure shows a combat screen from Slay the Spire with AOIs outlined in red. At the bottom centre are hand cards, representing actions available to the player. At the top-left are relic icons, which provide passive effects. At the top are potion slots, showing consumable items. On the right, above the enemy, the intent icon and tooltip indicate the opponent’s next planned move. These AOIs were used to quantify gaze allocation to task-relevant interface elements.
Feature
Importance
Potion bar (AOI proportion)
45.99
Potion bar (AOI rate)
40.88
Hand cards (AOI proportion)
4.29
Hand cards (AOI rate)
3.49
Saccade–fixation ratio
2.36
Saccade rate
1.51
Appendix
Table 11. CatBoost feature importance for the ocular modality (mean across LOSO folds).
Feature
Importance
HR Range
0.56
Mean Heart Rate
0.44
Appendix
Table 12. XGBoost feature importance for the Cardiac modality (mean across LOSO folds).
Human cognitive performance is constrained by limited mental resources, yet continuous computational estimation of cognitive capacity dynamics remains an open challenge. We propose a theory-driven multimodal learning framework that models capacity-related cognitive state as a two-dimensional physiological representation defined by voluntary resource allocation (mental effort) and overload-related strain (stress). The proposed architecture combines dual-stream encoding of cardiac (IBI/HRV) and electrodermal (EDA) signals with late fusion and task-specific output heads that independently estimate probabilistic effort and stress states. Evaluation on the SWELL-KW dataset using strict leave-one-subject-out cross-validation demonstrates cross-individual generalization (stress: 70.0% balanced accuracy; effort: 72.2%), with significant gains from multimodal integration and theory-guided supervision. Rather than collapsing physiological dynamics into a single workload label, the proposed effort--stress state-space enables structured differentiation between distinct cognitive regimes, including productive engagement and overload-related strain. Predicted state trajectories exhibit significant demand-sensitive shifts under controlled workload manipulations, with effort and stress responding differentially across interruption and time-pressure conditions. These results suggest that physiologically grounded multidimensional state representations may provide a foundation for adaptive systems capable of continuous capacity-aware monitoring and human-centered interaction.
We present OLIVE, a framework for adapting a foundation model to provide real-time assistance in temporally demanding, high-stakes, and dynamic tasks. We show that passive EEG, fused online with behavioral evidence, can meaningfully extend the number of targets users detect and engage beyond their unaided action bandwidth. OLIVE learns from both explicit behavioral signals (the targets the user shoots down in an XR first-person shooter game) and implicit physiological signals (fixation-locked EEG) to provide timely guidance, continuously adapting a frozen vision-language model's inference on which items are task-relevant by jointly estimating per-source reliability without manual labels or offline training. Through three user studies, including two live deployments of an assistive agent driven by OLIVE in XR, we show that OLIVE Pareto-dominates prior test-time adaptation frameworks, achieving the highest convergence rate at comparable convergence speed. Combining implicit physiological and explicit behavioral signals, the OLIVE agent produces the largest and most reliable within-session improvement to a user's ability to detect and engage targets, largely independent of the individual's skill. When the target switches silently, the agent that uses both behavioral and physiological signals reconverges significantly faster than the behavior-only agent (1.27 times faster on average, p = .008), restoring trustworthy guidance at the moment the task changes, precisely when reliable assistance matters most.
We propose a neuro-symbolic architecture that learns four interpretable physiological concepts, oculomotor dynamics, gaze stability, prefrontal hemodynamics, and multimodal, from eye-tracking and neural hemodynamics, functional near-infrared spectroscopy, (fNIRS) windows using attention-based encoders, and combines them with differentiable approximate reasoning rules using learned weights and soft thresholds, to address both rigid hand-crafted rules and the lack of subject-level alignment diagnostics. We apply this system to fatigue classification from multimodal physiological signals, a domain that requires models that are accurate and interpretable, with internal reasoning that can be inspected for safety-critical use. In leave-one-subject-out evaluation on 18 participants (560 samples), the method achieves 72.1% +/- 12.3% accuracy, comparable to tuned baselines while exposing concept activations and rule firing strengths. Ablations indicate gains from participant-specific calibration (+5.2 pp), a modest drop without the fNIRS concept (-1.2 pp), and slightly better performance with Lukasiewicz operators than product (+0.9 pp). We also introduce concept fidelity, an offline per-subject audit metric from held-out labels, which correlates strongly with per-subject accuracy (r=0.843, p < 0.0001).