Bayesian Inference of Psychometric Variables From Brain and Behavior in Implicit Association Tests
Authors: Christian A. Kothe, Sean Mullen, Michael V. Bronstein, Grant Hanada, Marcelo Cicconet, Aaron N. McInnes, Tim Mullen, Marc Aafjes, Scott R. Sponheim, Alik S. Widge
Organizations: Intheon, La Jolla, CA, USA · Department of Psychiatry and Behavioral Sciences, University of Minnesota, MN, USA · Institute for Health Informatics, University of Minnesota, MN, USA · Deliberate AI, New York, NY, USA · Minneapolis VA Medical Center, MN, USA
Objective. We establish a principled method for inferring mental health related psychometric variables from neural and behavioral data using the Implicit Association Test (IAT) as the data generation engine, aiming to overcome the limited predictive performance (typically under 0.7 AUC) of the gold-standard D-score method, which relies solely on reaction times. Approach. We propose a sparse hierarchical Bayesian model that leverages multi-modal data to predict experiences related to mental illness symptoms in new participants. The model is a multivariate generalization of the D-score with trainable parameters, engineered for parameter efficiency in the small-cohort regime typical of IAT studies. Data from two IAT variants were analyzed: a suicidality-related E-IAT (n=39) and a psychosis-related PSY-IAT (n=34). Main Results. Our approach overcomes a high inter-individual variability and low within-session effect size in the dataset, reaching AUCs of 0.73 (E-IAT) and 0.76 (PSY-IAT) in the best modality configurations, though corrected 95% confidence intervals are wide (±0.18) and results are marginally significant after FDR correction (q=0.10). Restricting the E-IAT to MDD participants improves AUC to 0.79 [0.62,0.97] (significant at q=0.05). Performance is on par with the best reference methods (shrinkage LDA and EEGNet) for each task, even when the latter were adapted to the task, while the proposed method was not. Accuracy was substantially above near-chance D-scores (0.50-0.53 AUC) in both tasks, with more consistent cross-task performance than any single reference method. Significance. Our framework shows promise for enhancing IAT-based assessment of experiences related to entrapment and psychosis, and potentially other mental health conditions, though further validation on larger and independent cohorts will be needed to establish clinical utility.
Figures & tables
Figure 1: ( a ) Block/trial structure used for E-IAT and PSY-IAT task designs. A “congruent” or “incongruent” block represents a block of trials where the concept-attribute pair (i.e., “trapped/me”) displayed is in agreement or conflict, respectively, with the subject type (patient / control). ( b , c ) Screenshots of PSY-IAT task showing a trial from the two block types. ( b ) depicts an incongruent trial for a person with schizophrenia (a congruent trial for a person without schizophrenia). ( c ) depicts a congruent trial for a person with schizophrenia (incongruent for a person without schizophrenia).
Figure 2: ( a ) A diagram of the data capture setup showing the positioning of the sensors relative to the subject. ( b ) A photograph of the data collection setup at the University of Minnesota used for the E-IAT task data collection. The same setup was used for the collection of the PSY-IAT task at the Minneapolis VA Medical Center.
Figure 3: Diagram for the USBL model, shown for a combination of EEG and FAU modalities. Here T is the number of trials in a session and P is the number of participants in the dataset being analyzed. The modality superscripts EEG and FAU have been shortened to E and F, respectively. XE and XF are the preprocessed observations for the current trial and participant (subscripts omitted) and WE and WF are the associated weight matrices. y is the dependent variable for the current participant (same for all trials), α is the vector of per-modality weights αm , A is the latent EEG contrast matrix, βF is the unscaled weight matrix for the FAU modality and τ and λ are the global and local (per-channel) horseshoe scales, respectively. ΣU is the model-derived EEG spatial covariance matrix and σi is the learned innovation scale for the EEG smoothing (GRW) prior. γ is the vector of per-region hyper-parameters estimated as part of the model.
Method
Modalities
AUC
Sensitivity
Specificity
Proposed
EEG
0.65±0.21 [0.43, 0.86] pBH=0.42
0.64±0.21 [0.41, 0.86]
0.65±0.25 [0.39, 0.91]
EEG, Gaze
0.73±0.17 † [0.56, 0.91] pBH=0.053
0.48±0.24 [0.23, 0.73]
0.82±0.21 [0.61, 1.04]
EEG, FAU
0.48±0.22 [0.24, 0.71] pBH=0.88
0.49±0.23 [0.24, 0.73]
0.49±0.26 [0.21, 0.77]
EEG, Dyn
0.48±0.23 [0.24, 0.72] pBH=0.88
0.42±0.22 [0.18, 0.65]
0.52±0.27 [0.23, 0.80]
Gaze
0.40±0.21 [0.17, 0.62] pBH=0.61
0.56±0.19 [0.35, 0.76]
0.33±0.25 [0.07, 0.58]
FAU, Dyn
0.43±0.24 [0.18, 0.69] pBH=0.69
0.40±0.23 [0.15, 0.64]
0.50±0.27 [0.21, 0.78]
Table 1: Performance comparison between the proposed method and a battery of reference methods across different modality combinations, on E-IAT data. Alternative hyper-prior settings are reported in table 4 . Note results on FAU and Dyn features in the E-IAT were not significantly different from chance level for the baseline methods (similarly to the proposed method), and are not reproduced in the table for conciseness. Values shown as mean ± SD across 50 CV folds; bracketed intervals are Nadeau & Bengio corrected 95% CIs on the mean. CIs were not truncated at [0, 1] to show nominal coverage rate. For the proposed method, BH-corrected p -values ( pBH ) are reported below each AUC, corrected across all 16 instantiations of the (unmodified) proposed method (entries in tables 1 , 2 , and 5 ). Results marginally significant at q=0.10 are shown in bold and marked with †.
Method
Modalities
AUC
Sensitivity
Specificity
Proposed
EEG
0.62±0.26 [0.35, 0.90] pBH=0.61
0.75±0.22 [0.51, 0.98]
0.56±0.26 [0.28, 0.83]
EEG, Gaze
0.42±0.25 [0.16, 0.69] pBH=0.69
0.48±0.30 [0.16, 0.80]
0.47±0.31 [0.15, 0.80]
EEG, FAU, Gaze
0.67±0.18 [0.48, 0.87] pBH=0.22
0.65±0.20 [0.43, 0.86]
0.66±0.28 [0.37, 0.96]
EEG, FAU
0.69±0.20 [0.48, 0.90] pBH=0.22
0.69±0.22 [0.46, 0.92]
0.60±0.27 [0.31, 0.89]
EEG, Dyn
0.60±0.21 [0.38, 0.83] pBH=0.61
0.53±0.24 [0.28, 0.78]
0.58±0.29 [0.28, 0.89]
Gaze
0.58±0.23 [0.35, 0.82] pBH=0.69
0.74±0.27 [0.46, 1.02]
0.39±0.22 [0.16, 0.62]
Table 2: Performance comparison between the proposed method and a battery of reference methods across different modality combinations, on PSY-IAT data. Values shown as mean ± SD across 50 CV folds; bracketed intervals are Nadeau & Bengio corrected 95% CIs on the mean. For the proposed method, BH-corrected p -values ( pBH ) are reported below each AUC, corrected across the 16 instantiations of the proposed method (entries in tables 1 , 2 , and 5 ). Results marginally significant at q=0.10 are shown in bold and marked with †.
Task
Modalities
Model Variant
MSE
Cross-Entropy
E-IAT
EEG, Gaze
Base
0.249 [ 0.247 , 0.250 ]
0.690 [ 0.688 , 0.693 ]
Calibrated
0.228 [ 0.190 , 0.267 ]
0.650 [ 0.562 , 0.739 ]
PSY-IAT
FAU, Dyn
Base
0.242 [ 0.236 , 0.249 ]
0.678 [ 0.665 , 0.691 ]
Calibrated
0.233 [ 0.180 , 0.285 ]
0.685 [ 0.521 , 0.850 ]
FAU, Dyn, Gaze
Base
0.242 [ 0.236 , 0.248 ]
0.677 [ 0.666 , 0.688 ]
Calibrated
0.227 [ 0.180 , 0.275 ]
0.667 [ 0.527 , 0.807 ]
Table 3: Comparison of Brier scores (MSE) and cross-entropy between base and calibrated models, for all modality combinations with highest AUC results. Values shown as mean [95% CI]. Lower scores indicate better calibration.
Task / Modalities
Variant
AUC
Sensitivity
Specificity
E-IAT EEG, Gaze
σ =0.1 (default)
0.73±0.17 [0.56, 0.91]
0.48±0.24 [0.23, 0.73]
0.82±0.21 [0.61, 1.04]
E-IAT EEG, Gaze
σ =0.01
0.77±0.17 [0.59, 0.95]
0.48±0.23 [0.23, 0.72]
0.85±0.20 [0.64, 1.05]
PSY-IAT EEG
Dugh (default)
0.62±0.26 [0.35, 0.90]
0.75±0.22 [0.51, 0.98]
0.56±0.26 [0.28, 0.83]
PSY-IAT EEG
LR
0.56±0.20 [0.35, 0.78]
0.53±0.22 [0.30, 0.76]
0.48±0.29 [0.18, 0.79]
PSY-IAT FAU, Dyn
σ =0.1 (default)
0.73±0.19 [0.53, 0.93]
0.73±0.25 [0.47, 0.99]
0.65±0.27 [0.36, 0.93]
PSY-IAT FAU, Dyn
σ =0.01
0.71±0.20 [0.49, 0.92]
0.71±0.25 [0.44, 0.97]
0.61±0.29 [0.30, 0.91]
Table 4: Supplementary parameter variations of the proposed method, comparing an alternative smoothness hyper-prior ( σ =0.01; see discussion) and a simpler Bayesian low-rank (LR) model variant for EEG. Values shown as mean ± SD across 50 CV folds; bracketed intervals are Nadeau & Bengio corrected 95% CIs on the mean.
Method
Modalities
AUC
Sensitivity
Specificity
Proposed
EEG, Gaze
0.79±0.17 * [0.62, 0.97] pBH=0.032
0.47±0.23 [0.22, 0.71]
0.82±0.21 [0.60, 1.05]
EEG, Gaze ( σ =0.01)
0.84±0.15 [0.68, 1.00]
0.44±0.21 [0.22, 0.67]
0.91±0.18 [0.72, 1.09]
Table 5: Performance of the proposed method on the E-IAT when restricting the analysis to MDD participants only ( n=33 ), excluding the 6 control subjects. This addresses the concern (see section Ground Truthing) that our model may partly be detecting an MDD/CTL distinction rather than the entrapment contrast. Values shown as mean ± SD across 50 CV folds; bracketed intervals are Nadeau & Bengio corrected 95% CIs on the mean. The default- σ configuration is included in the BH-FDR family jointly with tables 1 / 2 ; the alternative σ =0.01 variant is included for informational purposes and was not enrolled in statistical tests. AUC results significant after BH correction ( pBH<0.05 ) are shown in bold and marked with *.
Figure 4: EEG and eye-tracking weights for the best-performing E-IAT model; ( a ) spatial EEG weight maps, ( b ) eye tracking weight time course (note 0.0 is not at the center of the vertical axis).
Figure 5: Mean ERP at channel F7 (top) by group (left=low entrapment, right=high entrapment) for each condition (“trapped/me” vs “free/me”) with the scalp topographies underneath at time t=500 msec after stimulus onset.
Figure 6: Mean ERP at channel T7 (top) by group (left=control, right=psychosis) for each condition (“Schizophrenia/me” vs “NOT Schizophrenia/me”) with the scalp topographies underneath at time t=300 msec after stimulus onset.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
orig y→ new y′ (C/I)
(+) Stim. Assoc.
( − ) Stim. Assoc.
(+) Participant ( y=1 )
1→1 (C)
1→0 (I)
( − ) Participant ( y=0 )
0→0 (I)
0→1 (C)
Appendix
Table 6: Basic operation of the “recoding” trick. The label gets flipped on the right-hand column and becomes equivalent to the (crossed) participant/trial congruency factor.
Figure 7: Weights for the PSY-IAT model using Dynam+AU modalities; ( a ) AU weights over time (note the high stationarity), ( b ) matrix of Dynam weights (noisy).
College of Information Science and Engineering Ritsumeikan University Ibaraki, Osaka, Japan · College of Computer Science and Technology Zhejiang University Hangzhou, Zhejiang, China
Machine Learning Core National Institute of Mental Health, National Institutes of Health · Section on Developmental Neurogenomics, Human Genetics Branch National Institute of Mental Health, National Institutes of Health
Department of Biostatistics, Columbia Mailman School of Public Health. · Department of Biostatistics, Peking University Health Science Center. · Beijing International Center for Mathematical Research, Peking University.