cs.LGMar 19, 2026
SaveSignals of Success and Struggle: Early Prediction and Physiological Signatures of Human Performance across Task Complexity
Organizations: School of Computing, The Australian National University, Canberra, ACT, Australia · Computational Media Lab, The Australian National University, Canberra, ACT, Australia · The Australian National University, Canberra, ACT, Australia
Abstract
User performance is crucial in interactive systems, capturing how effectively users engage with task execution. Prospectively predicting performance enables the timely identification of users struggling with task demands. While ocular and cardiac signals are widely used to characterise performance-relevant visual behaviour and physiological activation, their potential for early prediction and for revealing the physiological mechanisms underlying performance differences remains underexplored. We conducted a within-subject experiment in a game environment with naturally unfolding complexity, using early ocular and cardiac signals to predict later performance and to examine physiological and self-reported group differences. Results show that the ocular-cardiac fusion model achieves a balanced accuracy of 0.86, and the ocular-only model shows comparable predictive power. High performers exhibited targeted gaze and adjusted visual sampling, and sustained more stable cardiac activation as demands intensified, with a more positive affective experience. These findings demonstrate the feasibility of cross-session prediction from early physiology, providing interpretable insights into performance variation and facilitating future proactive intervention.
Figures & tables
Figure 1. Overview of the prospective prediction pipeline. Early ocular and cardiac features from the low-complexity game session are used to predict performance outcomes in the subsequent high-complexity session. Ocular features capture visual behaviour, while cardiac features reflect autonomic arousal. Differences in visual patterns, autonomic profiles, and affective states are observed between high- and low-performing groups. Overview of the prospective prediction pipeline. The figure shows how early physiological signals predict later game performance. A participant plays a PC game while eye tracking and a wrist-worn sensor record visual behaviour and cardiac activity. Example plots in the middle illustrate ocular and cardiac features from the low-complexity session. These early features feed into a prospective prediction step that forecasts outcomes in the later high-complexity session. The right side shows two possible outcomes: high performance, linked to strategic visual attention and stable heart rate, and low performance, linked to narrow visual search and heart-rate instability.
Figure 2. Example game scenes illustrating the two experimental sessions. (a) Low-complexity session, characterised by fewer resources, weaker enemies, and a simple card deck; (b) High-complexity session, featuring more resources available, stronger enemies, and complex cards with intricate effects. Screenshots of game scenes across low-complexity and high-complexity phases. The figure comprises two screenshots from the game, illustrating the core experimental phases. Panel (a) shows the low-complexity session, where the interface displays fewer items, an opponent with lower health, and simple card options. Panel (b) shows the high-complexity session, featuring a boss fight with more items visible, a stronger opponent, and cards with more complex effects.
Figure 3. Eye tracking system (Tobii Pro Fusion). Eye tracker used in the experiment. The figure shows the Tobii Pro Fusion eye tracker, a rectangular bar-shaped device designed to be positioned below a monitor to record eye movements.
Figure 4. Cardiac recording device (Empatica EmbracePlus Wristband). Cardiac recording device used in the experiment. The figure shows the Empatica EmbracePlus, a black wristband with sensors on the inner side, worn on the wrist to collect cardiac signals.
Figure 5. Experimental procedure with annotated session structure and physiological recording. Experimental procedure showing sessions, physiological recordings, and self-reports. The figure presents the sequence of the study. Participants begin with a tutorial of about 3 minutes. They then enter the first game session, lasting about 25 minutes, which contains the low-complexity phase. A second game session of about 10 minutes follows and includes the high-complexity boss fight. Eye-tracking and cardiac signals are recorded continuously across both game sessions. After each session, participants provide subjective difficulty and affective ratings. The procedure ends with a performance outcome classified as win or loss.
| Category | Feature Names | Functional Role |
| Fixation | Fixation Duration, Fixation Count | Periods where gaze remains relatively stable, indicating focused attention. |
| Saccade | Saccade Amplitude, Saccade Duration, Saccade Velocity, Saccade Count, Saccade to fixation ratio | Rapid eye movements between fixations, reflecting visual scanning and search behaviour. |
| Pupillometry | Pupil Diameter | Pupil size changes, associated with cognitive effort and emotional arousal. |
| AOI Features | AOI Hit Rate, AOI Proportion | Distribution and allocation of visual attention across predefined regions of interest on the game interface. |
Table 1. Overview of extracted ocular features and their functional roles.
| Category | Feature Names | Functional Role |
| Heart Rate Metrics | Heart Rate, Heart Rate Range | Reflect overall cardiovascular activation. |
| Heart Rate Variability Metrics | Root Mean Square of Successive Differences (RMSSD), Mean NN Interval | Reflect autonomic nervous system regulation. |
Table 2. Overview of extracted cardiac features and their functional roles.
Figure 6. Processing framework integrating ocular, cardiac, and fused modalities. Framework from signal recording to model evaluation. The figure shows two parallel processing streams for ocular and cardiac data. The top stream illustrates the ocular pathway: raw eye-tracking data undergoes artifact correction, filtering, and segmentation. Feature engineering then extracts pupil diameter, saccades, fixations, and gaze measures. These features are used to train an ocular model that outputs a probability of high or low performance. The bottom stream illustrates the cardiac pathway: cardiac signals are denoised, filtered, and segmented. Then, feature engineering extracts blood volume pulse, heart rate, and heart-rate variability. These features train a cardiac model that outputs its own performance probability. Both model outputs are combined through decision-level fusion to generate the final performance prediction of win or loss.
Figure 7. Subjective difficulty ratings across game sessions. Bars represent mean ratings ( SD), separated by Low Complexity and High Complexity sessions, with performance groups shown as grouped bars (Win = high-performing; Loss = low-performing). Dots indicate individual participants’ ratings. Asterisks indicate significant difference between sessions (** ). Bar chart of complexity ratings by session and outcome. The figure presents self-reported difficulty ratings across two game sessions. Ratings are higher in the high-complexity session compared to the low-complexity session. Within each session, bars are grouped by outcome, showing separate values for win and loss participants. Individual ratings are displayed as dots, and asterisks indicate a significant difference at .
Figure 8. Subjective ratings on the SAM scales grouped by performance (Win = high-performing; Loss = low-performing). Bars show mean ( SD), with dots indicating individual participants. Left: Valence (0 = unpleasant, 5 = pleasant). Middle: Arousal (0 = calm, 5 = aroused). Right: Dominance (0 = low control, 5 = high control). Asterisks indicate significant differences (** , * ). SAM ratings for valence, arousal, and dominance by complexity and outcome. The figure shows self-assessment manikin ratings across two sessions and outcome groups. For valence, participants in the win group reported higher pleasantness than those in the loss group. For arousal, ratings were significantly higher in the high-complexity session than in the low-complexity session for both performance groups. For dominance, perceived control was significantly lower in the high-complexity session than in the low-complexity session for both performance groups. Significant differences are marked with asterisks at * and **.
| Modality | Model | BAcc | Macro-Pr | Macro-Re | Macro-F1 | MCC |
| Ocular | CatBoost | 0.83 | 0.83 | 0.82 | 0.83 | 0.66 |
| XGBoost | 0.80 | 0.77 | 0.76 | 0.79 | 0.54 | |
| Linear SVM | 0.57 | 0.79 | 0.56 | 0.48 | 0.27 | |
| Cardiac | CatBoost | 0.66 | 0.67 | 0.66 | 0.66 | 0.33 |
| XGBoost | 0.70 | 0.72 | 0.70 | 0.70 | 0.42 | |
| Linear SVM | 0.50 | 0.27 | 0.50 | 0.35 | 0.00 |
Table 3. Cross-subject LOSO performance across the ocular, cardiac, and fused feature sets for all models. The table reports the best subject-level results obtained under LOSO, including balanced accuracy, macro-precision, macro-recall, macro-F1, and MCC.
Figure 9. Cross-subject classification performance (balanced accuracy) comparison across modalities. The grouped bars illustrate the incremental predictive power from unimodal (cardiac, ocular) to multimodal (fused) approaches. CatBoost and XGBoost are shown as the top-performing classifiers Grouped bar chart of model performance (balanced accuracy) comparison across modalities. The figure displays balanced accuracy values for CatBoost and XGBoost across three feature sets: cardiac, ocular, and fused. Each feature set contains two adjacent bars corresponding to the two classifiers. Fused models show the tallest bars overall, followed by ocular models, and cardiac models have the shortest bars. CatBoost bars are consistently higher than XGBoost bars within each feature set.
Figure 10. Confusion matrices for the ocular (left) and fused (right) models, selected as the top-performing classifiers from the cross-subject evaluation. Confusion matrices for ocular vs fused classifiers. The figure shows two square confusion matrices, each with predicted labels on the x-axis and true labels on the y-axis. In both matrices, rows represent actual outcomes (Loss and Win), and columns represent predicted outcomes (Loss and Win). The left matrix corresponds to the ocular model: 12 correct Loss classifications in the top-left cell, 4 misclassified Loss samples in the top-right cell, 2 misclassified Win samples in the bottom-left cell, and 17 correct Win classifications in the bottom-right cell. The right matrix corresponds to the fused model: 14 correct Loss classifications, 2 misclassified Loss samples, 3 misclassified Win samples, and 16 correct Win classifications. Cell values are numerically indicated, with darker shading representing larger counts.
| Component Removed | BAcc | Macro F1 | MCC | BAcc |
| \ | 0.83 | 0.83 | 0.66 | 0.00 |
| Pupil | 0.83 | 0.83 | 0.66 | 0.00 |
| Fixation | 0.83 | 0.83 | 0.66 | 0.00 |
| Saccade | 0.69 | 0.68 | 0.37 | 0.14 |
| AOI Gaze | 0.40 | 0.39 | -0.22 | 0.43 |
Table 4. Ablation of ocular feature modules, where one component is removed at a time. BAcc denotes the change in balanced accuracy relative to the full ocular model. The row \ corresponds to the full ocular model ( , LOSO).
| Component Removed | BAcc | Macro F1 | MCC | BAcc |
| \ | 0.86 | 0.86 | 0.72 | 0.00 |
| Consensus Check | 0.77 | 0.77 | 0.54 | 0.09 |
Table 5. Ablation of the consensus check in the stacking architecture, where one component is removed at a time. BAcc denotes the change in balanced accuracy relative to the full fusion architecture. The row \ corresponds to the full fusion architecture ( , LOSO).
Figure 11. Raincloud plots of cardiovascular features comparing performance groups (Win = high-performing; Loss = low-performing). (A) Mean heart rate. (B) Heart rate range. For all panels, raincloud plots depict individual datapoints (dots), probability density (violin), and boxplots summarising the median and interquartile range. Raincloud plots of cardiovascular features by performance group. The figure compares cardiovascular features between the win and loss groups. Panel A shows mean heart rate, and Panel B shows heart rate range. In all panels, individual participants are shown as dots, the distribution is shown as a violin representing probability density, and a boxplot summarises the median and interquartile range.
Figure 12. Dynamic trends of cardiac features across three phases (tutorial, low-complexity, high-complexity), comparing performance groups (Win = high-performing; Loss = low-performing). Lines show group means with shaded bands indicating 1 SEM. Line plots of cardiac features across sessions by outcome group. The figure presents two line plots showing cardiac features across tutorial, low-complexity, and high-complexity sessions for win and loss groups. Solid lines indicate group means, with shaded bands representing variability within groups at plus or minus one standard error of the mean. Asterisks above the markers denote sessions where between-group differences are statistically significant, tested using either independent-samples t-tests or Mann–Whitney U-tests. PanelA shows mean heart rate, and PanelB shows heart rate range.
Figure 13. Raincloud plots of ocular features comparing performance groups (Win = high-performing; Loss = low-performing). (A) AOI gaze proportion: Potions. (B) AOI gaze proportion: Hand Cards. (C) Saccade rate (Hz). (D) Mean saccade amplitude (°). (E) Saccade-to-fixation ratio. For all panels, raincloud plots depict individual datapoints (dots), probability density (violin), and boxplots summarising the median and interquartile range. Asterisks indicate significant differences (* p < .05). Raincloud plots of ocular features comparing performance groups, with high-performing players labelled as Win and low-performing players labelled as Loss. The figure consists of five panels. Panel A shows AOI gaze proportion for Potions. Panel B shows AOI gaze proportion for Hand Cards. Panel C shows saccade rate measured in hertz. Panel D shows mean saccade amplitude measured in degrees. Panel E shows the saccade-to-fixation ratio. In all panels, individual participants are shown as dots, the distribution is shown as a violin representing probability density, and a boxplot summarises the median and interquartile range. Asterisks indicate statistically significant differences between groups at p less than 0.05.
Figure 14. Dynamic trends of ocular features across three phases (tutorial, low-complexity, high-complexity), comparing performance groups (Win = high-performing; Loss = low-performing). Lines show group means with shaded bands indicating 1 SEM. Asterisks indicate significant between-group differences at each session (* p < .05). Line plots of ocular indices across game sessions. The figure presents five line plots of ocular indices across three game phases: tutorial, low-complexity, and high-complexity. Panel A shows AOI gaze proportion for Potions. Panel B shows AOI gaze proportion for Hand Cards. Panel C shows saccade rate measured in hertz. Panel D shows mean saccade amplitude measured in degrees. Panel E shows the saccade-to-fixation ratio. Solid green lines show the win group and dashed purple lines show the loss group. Circles indicate win-group means, and squares indicate loss-group means. Asterisks mark statistically significant between-group differences at .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Hyperparameters |
| Linear SVM | C = 1.0 class_weight = {0: 1.0, 1: float(spw)} random_state = 42 |
| CatBoost | iterations = 600 depth = 6 learning_rate = 0.05 l2_leaf_reg = 3.0 loss_function = Logloss eval_metric = Logloss verbose = False random_state = 42 class_weights = [1.0, max(1.0, spw)] |
| XGBoost | n_estimators = 400 max_depth = 5 learning_rate = 0.05 subsample = 0.8 colsample_bytree = 0.8 min_child_weight = 5 reg_lambda = 1.5 reg_alpha = 0.0 gamma = 0.2 objective = binary:logistic class_weights = [1.0, max(1.0, spw)] tree_method = hist random_state = 42 eval_metric = logloss |
| XGBoost (Fusion Model) | n_estimators = 10 max_depth = 2 learning_rate = 0.08 subsample = 1.0 colsample_bytree = 1.0 min_child_weight = 1.0 reg_lambda = 1.0 objective = binary:logistic scale_pos_weight = float(spw) eval_metric = logloss n_jobs = 1 tree_method = hist random_state = 42 |
Table 6. Model hyperparameters for SVM, CatBoost and XGBoost.
| Eye Feature | Description with Formula | Statistical Metrics |
| Pupil Diameter | Pupil diameter calculated separately for left, right, and both eyes within the resampling window. | Mean, Min, Max, SD |
| Saccade Amplitudes | Angular distance of saccades in degrees. Formula: , where is the pixel distance and is the viewing distance. | Mean, Max |
| Saccade Velocity | Velocity of saccades in degrees/second. Formula: . | Mean, Max |
| Saccade & Fixation Count | Event counts within each resampling window. | Mean |
| Saccade & Fixation Rate | Per-second event rate. Formula: . | Rate |
| Saccade/Fixation Ratio | Ratio of total saccade duration to total fixation duration. Formula: . | Ratio |
Table 7. Description and Statistical Metrics of Eye-Tracking Features.
| Cardiac Feature | Description with Formula | Statistical Metrics |
| Heart Rate (HR) | Number of heartbeats per window. Derived from interbeat intervals (IBIs). Formula: , where IBI is in seconds. | Mean |
| Heart Rate Variability (HRV) | Two indices computed from NN intervals. | Mean |
| Blood Volume Pulse (BVP) Statistics | Signal statistics across a window. Mean: Std: | Mean, Std |
Table 8. Description and Statistical Metrics of Cardiovascular Features.
| Modality | Complete Extracted Feature Set | Final Features Used |
| Ocular | Pupil: Left/Right/Avg diameter (Mean, Std, Min, Max); Saccade: Count mean; Rate; Amplitude (Mean, Max); Velocity max; Saccade to fixation ratio; Fixation: Duration (Max, Mean, Sum); AOI: Hand cards (Proportion, Rate); Top bar potion (Proportion, Rate) | Saccade: Count mean; Rate; Amplitude (Mean, Max); Velocity max; Saccade to fixation ratio; AOI: Hand cards (Proportion, Rate); Top bar potion (Proportion, Rate) |
| Cardiac | BVP Stats: Min; Max; Mean; Std; Variance; RMS; Skewness; Kurtosis; HRV: Mean NN; RMSSD; pNN50; pNN20; Ratio | Heart Rate: Mean Heart Rate; Heart Rate Range |
Table 9. Complete extracted feature set and final features used for each modality.
| Element | Function / Effect | Typical On-Screen Position |
| Relics | Permanent passive bonuses that affect the run (e.g., extra energy, healing, card draw). | Top-left area of the screen |
| Potions | One-time use consumable items used during combat to grant effects (e.g., heal, buff, damage). | Top-middle of the combat interface, in potion slots |
| Hand Cards | The cards the player currently can play, showing available actions (Attacks, Skills, Powers). | Bottom centre of the screen |
| Enemy Tooltip | Shows what the enemy intends to do next (e.g., attack amount, defense, buffs/debuffs), guiding player strategy. | Above the enemy on the combat screen |
Table 10. Key interface elements in Slay the Spire with their positions and functions.
Figure 15. Areas of interest (AOIs) defined for gaze feature extraction in Slay the Spire . AOIs included (1) hand cards at the bottom centre (decision options), (2) relics at the top-left (passive bonuses), (3) potions at the top (consumable items), and (4) enemy intent indicators above the opponent, including both the icon and tooltip (upcoming enemy action). These regions provided the basis for deriving gaze features such as fixation proportion and dwell time. AOIs feature illustration The figure shows a combat screen from Slay the Spire with AOIs outlined in red. At the bottom centre are hand cards, representing actions available to the player. At the top-left are relic icons, which provide passive effects. At the top are potion slots, showing consumable items. On the right, above the enemy, the intent icon and tooltip indicate the opponent’s next planned move. These AOIs were used to quantify gaze allocation to task-relevant interface elements.
| Feature | Importance |
| Potion bar (AOI proportion) | 45.99 |
| Potion bar (AOI rate) | 40.88 |
| Hand cards (AOI proportion) | 4.29 |
| Hand cards (AOI rate) | 3.49 |
| Saccade–fixation ratio | 2.36 |
| Saccade rate | 1.51 |
Table 11. CatBoost feature importance for the ocular modality (mean across LOSO folds).
| Feature | Importance |
| HR Range | 0.56 |
| Mean Heart Rate | 0.44 |
Table 12. XGBoost feature importance for the Cardiac modality (mean across LOSO folds).