No Transformer Beats Six Covariates: Long-Horizon Prediction of Depressive Symptoms from Childhood Essays

Daniel Kua; Emrul Hasan; John-Jose Nunez; Frances Chen

cs.CLOct 6, 2026

No Transformer Beats Six Covariates: Long-Horizon Prediction of Depressive Symptoms from Childhood Essays

Authors: Daniel Kua, Emrul Hasan, John-Jose Nunez, Frances Chen

Organizations: Department of Statistics, University of British Columbia, Vancouver, Canada · Vector Institute, Toronto, Canada · Department of Psychiatry, University of British Columbia, Vancouver, Canada · Department of Psychology, University of British Columbia, Vancouver, Canada

Abstract

Natural language processing (NLP) models can detect depression-related language in text written near the time symptoms are measured, but whether pretrained transformers can predict depressive symptoms from text written twelve years earlier is largely untested. In the National Child Development Study, a British birth cohort, we predict probable depressive symptoms at age 23 from essays the same people wrote at age 11. Our baseline, a logistic regression on six childhood covariates, outperforms every text model that sees only the essay: seven fine-tuned transformers, a bag-of-words model, frozen embeddings and four zero-shot large language models. Its area under the receiver operating characteristic curve (AUC-ROC) is 0.737 against 0.670 for the best transformer on the primary seed, and no added text score detectably raises the baseline's AUC-ROC. None of the five domain-pretrained transformers detectably beats its general-domain control after Bonferroni correction. For long-horizon prediction, the baseline remains the model to beat.

Figures & tables

Panel of Figure 1 — Figure 1: Text models against the covariate baseline. The dashed path adds the text score to the covariates.

Step	N
Essays parsed, non-empty (UKDA-8313)	10,509
Plus sex available (UKDA-5565)	10,507
Plus all six childhood covariates	9,333
Plus Malaise at 23 (UKDA-5566)	7,226
Below the cut (Malaise $<8$ )	6,729 (93.1%)
At or above the cut (Malaise $\geq 8$ )	497 (6.9%)

Table 1: Cohort flow. Bold : analytic sample.

General-domain language models
Model	AUC-ROC $\uparrow$ [95% CI]	AUC-ROC ${}_{\text{5-run}}$ $\uparrow$	AUC-PR $\uparrow$	Brier $\downarrow$
Covariate baseline (logistic regression)	0.737 [0.688, 0.783]	–	0.188	0.060
TF-IDF + logistic regression (non-neural)	0.644 [0.591, 0.697]	–	0.115	0.063
BERT-base-uncased	0.669 [0.617, 0.718]	0.663 $\pm$ 0.026	0.128	0.064
RoBERTa-base	0.620 [0.567, 0.670]	0.661 $\pm$ 0.005	0.097	0.063
Domain-specific language models

Table 2: Test-set performance on the primary seed. CI: confidence interval. AUC-PR: area under the precision-recall curve. AUC-ROC

{}_{\text{5-run}}

: mean

\pm

standard deviation (SD) over five runs on the longer schedule (Section 3.2 ). Arrows mark the better direction, bold the best per column and underline the best text model on unrounded values. ‘–’: deterministic or zero-shot. Chance: AUC-PR

0.068

(the test prevalence), Brier

0.064

(predicting the training prevalence for everyone). Zero-shot rows: well-being prompt, rating mapped to risk (Appendix B.2 ).

Figure 3: Test-set ROC (left) and precision-recall (PR, right) curves. Encoders: mean of five runs. Others: one fit. Zero-shot LLMs: dashed (Qwen2.5-32B dash-dot). Covariate-only LR (logistic regression): covariate baseline.

Figure 4: Paired DeLong tests:

\Delta

AUC-ROC with 95% bootstrap CI on the primary seed and for the five-seed ensemble. Top: the covariate baseline minus each encoder. Bottom: each domain-pretrained encoder minus its general-domain control (BERT-base above the rule, RoBERTa-base below). *: survives Bonferroni correction (top

p<0.05/7

, bottom

p<0.05/5

Score added	$\Delta$ AUC [95% CI]	$p$
BERT-base	$+0.0006$ $[-0.0032,+0.0044]$	0.74 / 0.32
RoBERTa-base	$+0.0005$ $[-0.0014,+0.0026]$	0.60 / 0.68
ClinicalBERT	$-0.0011$ $[-0.0031,+0.0009]$	0.29 / 0.61
BioBERT	$-0.0052$ $[-0.0132,+0.0026]$	0.20 / 0.24
MentalBERT	$+0.0004$ $[-0.0021,+0.0030]$	0.73 / 0.39
MentalRoBERTa	$+0.0001$ $[-0.0019,+0.0022]$	0.92 / 0.66

Table 3: Change (

\Delta

AUC) in the covariate baseline’s AUC-ROC of

0.737

when one text score, cross-fitted if trained, is added.

p

: paired DeLong (test rows) / nested likelihood-ratio (training rows). The encoders (top block, shorter schedule) and the cross-fitted TF-IDF differ from those in Table 2 (Appendix B.3 ). Zero-shot rows use three decimals (parse rates: Table A1 ).

	$R^{2}$	AUC-ROC of the score
Text model	cov.	whole	cov.-fitted	residual
MentalBERT	0.401	0.671	0.720	0.549
BERT-base	0.343	0.663	0.721	0.542
RoBERTa-base	0.415	0.661	0.725	0.522
MentalRoBERTa	0.427	0.654	0.717	0.525
BioBERT	0.289	0.651	0.733	0.506

Table 4: Text-score decomposition (five-run means). Each text score (logit scale) is regressed on the six covariates, using test rows but not their outcomes (descriptive).

R^{2}

cov.: variance explained. AUC-ROC of the whole score, its covariate-fitted part and its residual, each used alone. † Deterministic. Bold : highest whole AUC-ROC.

Age-11 target	BERT-base	MentalBERT	TF-IDF	Cov. ref.
Sex	0.991	0.991	0.985	0.583
Ability	0.864	0.864	0.827	0.723
Internalising	0.682	0.693	0.658	0.742
Externalising	0.699	0.693	0.623	0.769

Table 5: Concurrent control: test AUC-ROC (

n=1{,}446

) for four of the six covariates as age-11 targets, with cognitive ability cut at its bottom third and behaviour ratings at their top tenth. Cov. ref.: logistic regression on the other five covariates. Bold : best per target. Underline : best text model where Cov. ref. wins. One run per model (encoders: primary seed, six-epoch recipe). Variables: Table B1 .

Figure 5: Calibration of the primary-seed test predictions. Points: tenths of the test rows by predicted risk, with Wilson 95% CIs. Belt

p

: calibration-belt test, failed if

p<0.05

(uncorrected). Slope and intercept: ideal

1

and

0

, with the intercept fitted at slope

1

(Wald 95% CI). Covariate-only LR (logistic regression): covariate baseline.

Encoder	Stock	Adapted
BERT-base	0.663 $\pm$ 0.026	0.673 $\pm$ 0.012
RoBERTa-base	0.661 $\pm$ 0.005	0.687 $\pm$ 0.017
MentalBERT	0.671 $\pm$ 0.007	0.678 $\pm$ 0.012
bert-base-cased	0.665 $\pm$ 0.012	0.682 $\pm$ 0.015
DeBERTa-v3-base †	0.657 $\pm$ 0.009	0.678 $\pm$ 0.022
DeBERTa-v3-large †	0.665 $\pm$ 0.036	0.673 $\pm$ 0.026

Table 6: Five-run test AUC-ROC, mean

\pm

SD, of seven encoders before (Stock) and after (Adapted) task-adaptive pretraining on the essays. † The masked-language-model head starts untrained (Appendix B.2 ). Bold : best in each column.

Figure 6: Test AUC-ROC by training-set size for the covariate baseline, BERT-base (the general-domain control) and MentalBERT (best five-run mean in Table 2 ). As new refits (six-epoch recipe, Table B2 ), the

4{,}624

-essay points need not match Table 2 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Parse	Odds ratio per SD [95% CI]
Qwen2.5-7B	$0.999$	0.906 [0.803, 1.022]
Qwen2.5-32B	$1.000$	0.881 [0.782, 0.994]
Gemma-2-9B	$1.000$	0.930 [0.825, 1.048]
Llama-3.1-8B	$1.000$	0.948 [0.839, 1.070]

Appendix

Table A1: Zero-shot LLM well-being ratings. Parse: share of the

7{,}226

essays whose reply parsed. Odds ratio: for the outcome per SD of the rating, adjusted for the six covariates (unpenalised fit on the training rows, Wald 95% CI). Below

1

, a higher rating goes with lower odds. Boehm et al. (2024) report

0.87

[0.78,0.98]

per SD of hand-coded well-being, in a differently restricted sample, also adjusted for word count.

Variable	What it measures (NCDS code)	Reported by	Range	Age-11 target
Malaise total (outcome)	derived total mal of 24 yes/no questions on emotional and bodily symptoms ( n6016 to n6039 )	participant, age 23	0–20	outcome: positive if $\geq 8$
Sex	female or male ( n622 )	cohort records	0 or 1	female
Father’s social class	father’s or male head’s occupation: non-manual, manual or no male head ( n1685 , 1966 General Register Office scheme)	parent, age 11	1–3	not used
Parental psychiatric history	psychiatric condition coded for the mother ( n1406 , n1407 ) or father ( n1415 , n1416 ) among chronic or serious illnesses since age 7, other codes 0	parent, age 11	0 or 1	not used
Cognitive ability	items correct of 80 (40 verbal, 40 non-verbal) on a 30-minute general ability test ( n920 )	test, age 11	0–79	low: $\leq 37$ (bottom third)
BSAG internalising	sum of depression ( n980 ), withdrawal ( n977 ), unforthcomingness ( n974 ) and writing off adults ( n989 )	teacher, age 11	0–31	high: $\geq 10$ (top tenth)

Appendix

Table B1: Variables. Rows 2 to 7 are the covariate baseline’s inputs, standardised (sex: 1 for female, social class: one ordinal term). Range: in the analytic sample. A BSAG score counts the behaviour descriptions the teacher underlined. Last column: targets of Table 5 , cut at training-row quantiles, not clinical thresholds.

Learning rate	$2\times 10^{-5}$	Weight decay	0.01
Batch size	16	Warmup	10%
Train epochs	6	Rate decay	linear
Early-stop patience	3 epochs	Max gradient norm	1.0

Appendix

Table B2: Six-epoch recipe: settings from StressRoBERTa ( Alqahtani et al., 2026 ) , except our warmup, rate decay and clipping. It covers the primary-seed runs, the five-split refit and the learning curve. Five-run refits use a longer schedule (10 epochs, rate

10^{-5}

) and cross-fitting a shorter one (Appendix B.3 ). The two large encoders use batch size 8 in training and validation.

Hugging Face model

Fine-tuned encoders (Table 2 )

bert-base-uncased

roberta-base

emilyalsentzer/Bio_ClinicalBERT

dmis-lab/biobert-base-cased-v1.1

mental/mental-bert-base-uncased

Appendix

Table B3: Model checkpoints. The seven encoders of Table 2 have

108

125

million parameters.

Baseline minus model AUC-ROC, both relabelled
Outcome cut	6	7	8	9	10
Test positives	218	154	99	59	37
Baseline, relabelled	0.690	0.712	0.737	0.733	0.739
Baseline, refitted	0.693	0.714	0.737	0.734	0.746
BERT-base	0.043	0.051	0.068	0.069	0.048 †
RoBERTa-base	0.068	0.086	0.117	0.104	0.096

Appendix

Table B4: Primary-seed test AUC-ROC by outcome cut for models trained at cut 8 (relabelled) or at the cut (refitted). † Paired DeLong

p\geq 0.05

(uncorrected).

Explore similar work

Natural Language Processing Psychometrics
Aug 7, 2026Edoardo Sebastiano De Duro, Emma Franchino, Massimo StellaPsychometric Properties Mental Health
CogNoscoLab, Department of Psychology and Cognitive Science, University of Trento, Italy
EmoTrack: Clinical-Semantic Modeling for Text-Based Depression Severity Estimation
May 21, 2026Zhaomin Wu, Jiayi Li, Bingsheng HeDepression Transcript
Department of Computer Science National University of Singapore
Reading between the Lines: Leveraging Large Language Models for Global Dementia and Depression Assessment from Clinical Interviews
Jun 16, 2026Franziska Braun, Alea Rüggeberg, Thomas Ranzenberger +4Depression Interviews
1TH Nürnberg · 2FAU Erlangen · 3PMU Klinikum Nürnberg, Germany

Aug 7, 2026, cs.CLJ/K move · Enter open · S save

Natural Language Processing Psychometrics

Edoardo Sebastiano De Duro, Emma Franchino, Massimo Stella

CogNoscoLab, Department of Psychology and Cognitive Science, University of Trento, Italy

Psychometric Properties Mental Health Natural Language Processing Demographics

Open paper AskPDF arXiv

Natural Language Processing (NLP) models predicting mental health outcomes rarely specify what they measure: contextual knowledge, emotional content, or syntactic structure. NLP Psychometrics treats psychological prediction from text as a psychometric problem, linking scores to interpretable linguistic evidence and testing beyond the training text format. Nine LLMs, conditioned on controlled personas (cognitive digital shadows), completed psychometric questionnaires with textual explanations per item. We extracted emotional profiles and syntactic-semantic structure via textual forma mentis networks, combined with personality and sociodemographic variables in ablated random forest (RF) regressors, using SHAP to identify which features drove performance and in which direction. Full RF models explained up to 70.8% of variance in life satisfaction (SWLS), 55.7% in depression (PHQ-9), and, for DASS-21, 68.5% depression, 76.0% anxiety, 72.4% stress. Sociodemographics alone explained no meaningful variance in depression, anxiety, or stress, but did so for life satisfaction, where emotion features and income were the strongest predictors; neuroticism and network topology instead dominated depression and anxiety, reversing direction between them. Without retraining, RF models separated diaries from low- and high-score personas (

r

up to 0.91) and, using only network/emotion features, classified clinical from control participants in real transcripts with up to 68% accuracy. These results show the promise and limits of synthetic data: LLM personas can expose model biases, recover patterns consistent with clinical rumination, and support psychometric prediction from human text without a matched questionnaire, but cannot substitute for human validation. NLP Psychometrics makes these distinctions explicit, measurable, and testable through interpretable AI and network/emotional features.

Abstract

Explore similar work

Natural Language Processing Psychometrics

EmoTrack: Clinical-Semantic Modeling for Text-Based Depression Severity Estimation

Reading between the Lines: Leveraging Large Language Models for Global Dementia and Depression Assessment from Clinical Interviews