q-bio.NCOct 5, 2026
SaveCOMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal
Organizations: Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Mental Illness Research, Education and Clinical Center, James J. Peters VA Medical Center, Bronx, NY, USA · Berkman Klein Center for Internet & Society, Harvard University, Cambridge, MA, USA
Abstract
Language models can score psychiatric questionnaires from speech, but agreement with self-report may reflect the questionnaire rather than the person. We introduce psychometric representational similarity analysis, a framework for comparing the structure of speech-derived scores, self-report, item wording and theory, and implement it alongside person-level construct scoring in COMPASS 2.0. We show how similarly worded items induce covariance without psychological signal. In pre-registered discovery and confirmation analyses of clinical interviews from 275 participants, language-derived symptom geometry resembled wording more than self-report, with no structure beyond wording detected by the registered tests. Geometric agreement with self-report survived assigning participants someone else's answers, whereas person-paired scores captured distress more than specific symptoms. Complementary analyses examined counselling quality and wording structure across 34 instruments and the Research Domain Criteria (RDoC) framework. These findings distinguish agreement about psychological structure from evidence that language-derived assessments track individual people.
Figures & tables
Figure 1: Psychometric representational dissimilarity matrices give every source of psychometric information one format. a , For the same 25 items (8 PHQ-8, 17 PCL-C), each source yields an item-by-item RDM ( correlation or similarity; darker, more similar; DAIC-WOZ, 189 participants): speech scored by an embedding model or by an entailment model (participants’ mean profiles), the participants’ self-reports, the similarity of the item texts, the entailment scorer’s confusions between item probes, and a theory model in which items of the same scale cluster (PHQ-8; PCL-C re-experiencing, avoidance and numbing, hyperarousal) are similar. b , Spearman correlations between the RDMs’ entries over item pairs, for sources in DAIC-WOZ (D) and E-DAIC (E). Diagonal: split-half reliability of each sampled RDM across random halves of participants (Spearman–Brown corrected; 200 splits); the attainable correlation between two sources is about the square root of the product of their reliabilities; dashes mark sources without sampling noise.
Figure 2: COMPASS 2.0 reads speech through an instrument as representations (psychometric RSA) and as construct scores (COMPASS), with controls for what the probes impose. a , The two inputs: language (invented transcript lines) and the items of an instrument (four PHQ-8 items). b , Both are embedded in one space (multidimensional scaling of MiniLM embeddings of the example sentences); lines join utterances to the items they resemble, with width proportional to cosine similarity. c , Item activity profiles: the similarity of each speech window to each of the eight PHQ-8 items (z-scored per item), for simulated windows with no symptom signal, grouped by the question that elicited them, before centring. d , Psychometric RSA: the item geometry, here the correlations between the items’ activities across the windows in c after centring each window on its question’s mean. e , COMPASS: keyed construct scores; here a language-derived PHQ-8 score (entailment scorer, participants’ mean profiles) against the self-reported PHQ-8 total (DAIC-WOZ, 189 participants). f , The wording null: the geometry of speech without symptom signal resembles the similarity of the item texts. g , Whitening by the item-text similarity matrix factors out the covariance expected from the item texts alone. h , The whitened language geometry is compared with the self-reported item geometry while controlling for wording (DAIC-WOZ, 25 PHQ-8 and PCL-C items, MiniLM; diagonal omitted). i , Cosine similarity and entailment-model stance, , of an example utterance with a forward alliance statement and its reversal (paraphrases, not inventory items).
Figure 3: Item wording alone produces a convincing symptom structure. a , Cosine similarity between the texts of the 8 PHQ-8 and 17 PCL-C items (MiniLM). Circles mark near-duplicate items across the two instruments (sleep, concentration, loss of interest). b , Simulation: correlation of the language-derived item geometry with the true symptom structure as the symptom signal in speech grows (mean s.d. over 200 simulated samples of 189 people). Without any signal ( ), the naive geometry already correlates almost as strongly with the truth as the wording itself does (dashed line), because the truth partly resembles the wording. The whitened geometry starts at zero. c , Partial correlation with the truth beyond the part predicted by wording.
Figure 4: In clinical interviews, language-derived symptom structure follows the questionnaire and the script, and geometric agreement is blind to who said what. a , Share of the variance of each item’s activity across answer windows explained by the eliciting question (filled) and by the person (open); DAIC-WOZ, MiniLM. b , Item-pair correlations of the naive language-derived geometry (189 participants) against item-text similarity. c , The same correlations against the participants’ self-reported item correlations. d , The registered tests in discovery (DAIC-WOZ; circles, 95% intervals) and confirmation (E-DAIC; squares, Bonferroni-adjusted intervals). H1 compares the naive geometry’s correlation with wording and with self-report; H2 is the partial correlation of the whitened geometry with self-report beyond wording (no interval; Freedman–Lane test); H3 is the mean same-item minus different-item correlation between language and self-report. Accent marks a supported test. e , Exploratory: correlation between the general factors (first principal components) of language-derived and self-reported items (accent), and item-specific convergence after removing the self-reported general factor (grey; a same-minus-different contrast, not a variance share), with person-bootstrap 95% intervals; circles discovery, squares confirmation. f , Person shuffle (entailment scorer): agreement of the naive language geometry with the self-reported geometry, of the general factors, and of a COMPASS PHQ-8 score with the self-reported total, with true pairing (accent; circles discovery, squares confirmation) and after shuffling which self-report belongs to which participant (grey, 95% range of 1,000 shuffles).
Figure 5: Keyed similarity scores of the alliance track what a session talks about, and stance-aware scores separate session quality in a second corpus. a , Mean cosine similarity between Working Alliance Inventory client-form items with the same key (forward–forward) and with opposite keys (forward–reverse), per subscale (MiniLM). b , Keyed Bond score of client turns (our earlier scoring rule) against the mean similarity of the same turns to all Bond items regardless of key, per session. c , Therapist-minus-client difference in keyed Bond score against the difference in relationship-topic activation. d , Simulation: correlation with the true alliance of keyed cosine scores and of stance-aware scores. e , Known groups: area under the curve (AUC) for separating high- from low-quality counselling sessions by the stance score, by the keyed cosine score of the same 36 observer-form items and by a replica of our earlier scoring, in discovery (AnnoMI; circles) and confirmation (second corpus; squares), with stratified-bootstrap 95% intervals.
Figure 6: Item wording carries a low-dimensional, domain-structured geometry in every instrument examined. a , Classical multidimensional scaling of the cosine distances between 702 item texts and construct definitions from 35 probe banks (MiniLM), coloured by construct domain. b , Effective number of dimensions (participation ratio) of each bank’s item-similarity matrix, before (filled) and after (open) removing the direction shared by all item embeddings (bge; banks ordered by number of items; logarithmic scale). c , Share of each bank’s items whose nearest neighbour lies in another instrument (grey) or in another construct domain (accent; MiniLM).
Extended Data Fig. 1: Script structure and reliability in both interview corpora. a , Correlation of the language-derived item geometry with item-text similarity at the level of single windows and of participants (naive, question- or phase-centred, and centred and whitened); MiniLM. b , Split-half reliability (Spearman–Brown) of the population geometry, raw and whitened. c , Median split-half reliability, across random halves of each participant’s windows, of the participant’s own window-level item geometry (raw correlations, which share the wording) and of that geometry whitened against the other participants’ window covariance (the participant’s deviation); participants with at least 40 windows.
Extended Data Fig. 2: Construct definitions of the Research Domain Criteria cluster by domain in embedding space. a , Classical multidimensional scaling of the cosine distances between the 49 construct and subconstruct definitions (MiniLM), coloured by domain. b , Share of definitions whose nearest neighbour lies in the same domain for each embedder, against the share expected by chance.
Extended Data Fig. 3: Simulation details. a , Correlation with the true symptom structure of the naive geometry, the whitened geometry, exact deconvolution restricted to admissible fits (positive fitted signal variance and positive noise-subtracted diagonal; their number out of 200 is given) and a positive-semidefinite projection of all deconvolution fits, at (circles) and (squares); mean s.d. over 200 samples of 189 people. Whitening does not estimate the true structure itself, so this panel compares structural agreement. b , Partial correlation with the truth beyond wording of person-level geometry when people answer different subsets of questions, without and with centring on the question mean, at and : centring keeps the null calibrated and costs little power. c , False-positive rate of the Freedman–Lane test without person signal after whitening by the wording reference or by the within-speaker covariance, in four conditions: topic variation of the same shape between and within speakers, stronger between-speaker topic variation of the same shape, topic variation along different directions between and within speakers, and genuine within-speaker item dynamics (200 samples each; dashed line, 5%).
| Corpus | Content | People or sessions | Units | Use |
|---|---|---|---|---|
| DAIC-WOZ | Clinical interviews by a human-operated virtual interviewer; human transcripts | 189 | 10,533 | Discovery (symptoms) |
| E-DAIC | Interviews by an autonomous agent; automatic transcripts | 86 | 5,214 | Confirmation (symptoms) |
| Alexander Street | Transcribed psychotherapy sessions, four clinical collections | 786 | 180,547 | Alliance re-examination |
| AnnoMI | Motivational-interviewing demonstrations, high/low quality (110/23) | 133 | 9,687 | Discovery (alliance) |
| High/low-quality counselling | Counselling sessions not in AnnoMI, high/low quality (119/65) | 184 | 7,877 | Confirmation (alliance) |
Extended Data Table 1: Corpora. Units are answer windows (interviews) or speaker turns (sessions).
| Sample | Scorer | H1 | H2 | H3 |
|---|---|---|---|---|
| Discovery | MiniLM | 0.68 / 0.42 (0.15 to 0.47) | 0.009 ( = 0.93) | 0.004 ( 0.016 to 0.023) |
| bge | 0.69 / 0.47 (0.15 to 0.40) | 0.028 ( = 0.52) | 0.014 ( 0.002 to 0.031) | |
| Entailment | – | 0.051 ( = 0.27) | 0.038 (0.023 to 0.056) | |
| Confirmation | MiniLM | 0.63 / 0.36 (0.11 to 0.55) | 0.008 ( = 0.93) | 0.038 (0.007 to 0.072) |
| bge | 0.65 / 0.21 (0.33 to 0.58) | 0.096 ( = 0.15) | 0.042 (0.013 to 0.073) | |
| Entailment | – | 0.118 ( = 0.15) | 0.005 ( 0.022 to 0.035) |
Extended Data Table 2: Registered symptom tests. H1: correlation of the naive language geometry with item-text similarity / with self-reported item correlations (interval of the difference: 95% in discovery, 97.5% in confirmation). H2: partial correlation of the whitened geometry with self-report beyond wording (Freedman–Lane ; Holm-adjusted in confirmation). H3: mean same-item minus different-item correlation (95% or 98.3% interval).
| Partial after whitening by ( ), controlling for | |||||||
|---|---|---|---|---|---|---|---|
| Sample | Scorer | H2 interval | Block | Corr | registered | both | |
| D | MiniLM | [ 0.137, 0.129] | 0.925 | 0.904 | 0.027 (0.722) | 0.003 (0.972) | 0.017 (0.823) |
| D | bge | [ 0.067, 0.109] | 0.520 | 0.877 | 0.031 (0.522) | 0.060 (0.190) | 0.059 (0.282) |
| D | entailment | [ 0.012, 0.107] | 0.260 | 0.216 | 0.030 (0.567) | 0.005 (0.923) | 0.030 (0.551) |
| C | MiniLM | [ 0.098, 0.123] | 0.962 | 0.914 | 0.029 (0.697) | 0.046 (0.522) | 0.076 (0.291) |
| C | bge | [ 0.134, 0.001] | 0.057 | 0.889 | 0.079 (0.135) | 0.045 (0.367) | 0.004 (0.950) |
Supplementary Table 1: Sensitivity analyses of the symptom tests (exploratory; values unadjusted). H2 interval: person-bootstrap 95% interval of the registered partial correlation, conditional on the estimated question means. Block : Freedman–Lane test of the registered statistic (controls , and for the entailment scorer) with items permuted within the PHQ-8 and within the PCL-C. Within-speaker reference: the population geometry whitened by the pooled within-speaker covariance instead of ; is the correlation of the reference geometry with item-text similarity; partial with self-report ( ) controlling for Corr only, for the registered controls only (the change of reference alone), or for both. This is a sensitivity to the reference, valid as a test of person structure only if within-speaker variation is nuisance with the same shape as between-speaker nuisance (Extended Data Fig. 3c). D, discovery (DAIC-WOZ); C, confirmation (E-DAIC).
| Analysis | Sessions | Videos | Stance AUC [95% CI] | Keyed cosine AUC | Difference [95% CI] |
|---|---|---|---|---|---|
| Discovery (AnnoMI) | 133 | 119 | 0.547 [0.420, 0.671] | 0.662 | [ 0.275, 0.040] |
| Registered confirmation sample | 184 | 156 | 0.682 [0.583, 0.773] | 0.564 | [ 0.009, 0.243] |
| Registered sample, screen-negative only | 176 | 150 | 0.695 [0.594, 0.785] | 0.550 | [0.013, 0.275] |
| All screen-negative sessions without ID match | 192 | 166 | 0.678 [0.585, 0.765] | 0.554 | [ 0.0001, 0.246] |
| Registered sample without over-limit segments | 184 | 156 | 0.676 [0.581, 0.767] | 0.564 | [ 0.019, 0.239] |
Supplementary Table 2: Sensitivity analyses of the alliance tests (exploratory). Intervals resample whole source videos (5,000 resamples). AUC, area under the curve for high- versus low-quality sessions; difference, stance minus keyed cosine. Screen-negative: no AnnoMI session contains at least 0.3 of the session’s word 5-grams; this does not establish independence of actors, therapists or productions.
| Registered | Identifier | Transcript | |
|---|---|---|---|
| analysed | ID distinct | screen-negative | 176 |
| analysed | ID distinct | text match | 8 |
| excluded | ID distinct | screen-negative | 1 |
| excluded | ID match | screen-negative | 4 |
| excluded | ID match | text match | 49 |
| excluded | ID unresolved | screen-negative | 15 |
Supplementary Table 3: Provenance of the high- and low-quality counselling sessions. Left: all 258 sessions by registered status, source-identifier status and transcript screen (text match: at least 0.3 of the session’s word 5-grams occur in one AnnoMI session). Right: the 21 sessions that the registered loader excluded without an identifier match (missing addresses, and one Vimeo address that the loader did not parse). In the registered sample, containment was at most 0.027 or at least 0.540.
| COMPASS PHQ-8 score with PHQ-8 total | General factors | ||||
| Sample | Scorer | partial | partial | ||
| D | MiniLM | 0.15 [0.00, 0.30] | 0.07 [ 0.08, 0.21] | 0.07 [ 0.06, 0.22] | 0.04 [ 0.16, 0.11] |
| D | bge | 0.26 [0.13, 0.37] | 0.16 [0.03, 0.29] | 0.26 [0.13, 0.38] | 0.14 [ 0.01, 0.27] |
| D | entailment | 0.54 [0.44, 0.62] | 0.52 [0.43, 0.61] | 0.49 [0.38, 0.59] | 0.46 [0.35, 0.56] |
| C | MiniLM | 0.39 [0.24, 0.53] | 0.12 [ 0.06, 0.28] | 0.33 [0.14, 0.50] | 0.02 [ 0.26, 0.24] |
| C | bge | 0.53 [0.41, 0.65] | 0.40 [0.23, 0.56] | 0.52 [0.38, 0.65] | 0.36 [0.16, 0.52] |
Supplementary Table 4: Person-paired read-outs beyond interview length and topic (exploratory). COMPASS PHQ-8 score: mean of the participant’s z-scored, question-centred PHQ-8 item scores, correlated with the self-reported PHQ-8 total; general factors: first principal components of the language-derived and self-reported item scores. Partial: after removing the ranks of the number of answer windows, the number of participant words and topic activation (mean MiniLM cosine similarity of the participant’s windows to the 25 symptom items), none of which is fitted to the self-reports; log counts instead of ranks gave similar values. Lower panel: rank correlations of these baselines with the PHQ-8 total. Person-bootstrap 95% intervals (2,000 resamples). D, discovery (DAIC-WOZ, 189 participants); C, confirmation (E-DAIC, 86).
| Analysis | Estimand | Unit of resampling | Status |
|---|---|---|---|
| H1 | Agreement of the naive population geometry with item wording minus its agreement with self-reported geometry | Participants | Registered (OSF) |
| H2 | Partial correlation of the whitened population geometry with self-reported geometry, beyond wording | Items (permutation) | Registered (OSF), primary |
| H3 | Person-paired item correspondence: mean same-item minus different-item correlation | Participants | Registered (OSF) |
| A1, A2 | Separation of high- from low-quality sessions by stance scores, and its advantage over keyed cosine scores | Sessions | Pre-specified (fingerprinted plan) |
| Psychometric RDM comparisons (Fig. 1) | Second-order agreement of population geometries across sources; split-half reliabilities | Participants (reliability only) | Descriptive |
| General factors, COMPASS PHQ-8 score | Person-paired agreement of language and self-report, alone and beyond length and topic baselines | Participants | Exploratory |
Supplementary Table 5: What each analysis estimates, and its status.
| Bank | Domain | Items | Text used | Source |
|---|---|---|---|---|
| CAPE-42 | symptoms | 42 | items, verbatim | 78 |
| SAPS | symptoms | 34 | item names | 3 |
| C-SSRS | symptoms | 31 | items, verbatim | 68 |
| PANSS | symptoms | 30 | item names | 42 |
| SANS | symptoms | 25 | item names | 2 , 3 |
| BDI-II | symptoms | 21 | item titles | 5 |
Supplementary Table 6: Probe banks. Texts were taken from the published forms or manuals; where only item titles or names are openly available, those were used. Copyrighted texts are not redistributed.
| Item | Hypotheses |
|---|---|
| PHQ-8 1 NoInterest | I have little interest in doing things. I have little pleasure in doing things. |
| PHQ-8 2 Depressed | I feel down. I feel depressed. I feel hopeless. |
| PHQ-8 3 Sleep | I have trouble falling asleep. I have trouble staying asleep. I sleep too much. |
| PHQ-8 4 Tired | I feel tired. I have little energy. |
| PHQ-8 5 Appetite | I have a poor appetite. I overeat. |
| PHQ-8 6 Failure | I feel bad about myself. I feel like a failure. I have let myself or my family down. |
Supplementary Table 7: Entailment hypotheses. Each symptom item was written as one or more atomic first-person statements; an item’s score is the maximum over its statements of given the answer window.