Organizations: Department of Computer Science, University of California, Los Angeles, CA, USA · Department of Psychology, The University of Texas at Austin, Austin, TX, USA · Department of Psychiatry, NYU Grossman School of Medicine, New York, NY, USA
Mental health disorders affect over 1 billion people worldwide, yet access to care remains limited by workforce shortages and cost constraints. While AI systems show therapeutic promise, current alignment approaches optimize objectives independently, failing to balance patient preferences with clinical safety. We survey 335 individuals with lived mental health experience to collect preference rankings across therapeutic dimensions, then develop a multi-objective alignment framework using direct preference optimization. We train reward models for six criteria -- empathy, safety, active listening, self-motivated change, trust/rapport, and patient autonomy -- and systematically compare multi-objective approaches against single-objective optimization, supervised fine-tuning, and parameter merging. Multi-objective DPO (MODPO) achieves superior balance (77.6% empathy, 62.6% safety) compared to single-objective optimization (93.6% empathy, 47.8% safety), and therapeutic criteria outperform general communication principles by 17.2%. Blinded clinician evaluation confirms MODPO is consistently preferred, with LLM-evaluator agreement comparable to inter-clinician reliability.
Figures & tables
Figure 1: Primary therapeutic preference distribution across persona sets.
Figure 2: Safety-empathy performance across training approaches. Each point shows a model’s average win rate computed across all pairwise head-to-head comparisons.
Figure 3: Statistical significance matrices from pairwise head-to-head comparisons using McNemar’s test ( α =0.05, n=600). △ = row wins, ∇ = column wins, = = no significant difference.
Figure 4: Phase 2 safety-overall preference trade-off. All trained models substantially outperform baseline on overall preference while maintaining or improving safety. Therapeutic-specific criteria (MODPO_Survey, MODPO_Survey4) achieve higher overall preference than general principles (MODPO_Maxim).
Figure 5
Figure 6: Human validation results. ( 5(a) ) Win rates by criterion; bars show the fraction of questions where the clinician majority selected MODPO_Survey, base_model, or tie. ( 5(b) ) Non-tie (binary) agreement with human reliability context; we compare leave-one-out human vs consensus and LLM vs consensus on the same non-tie subset, reporting accuracy, Cohen’s κ , and Gwet’s AC1; Fleiss’ κ provides overall inter-rater reliability context. ( 5(c) ) Agreement levels (3-class, all questions including ties); exact-match agreement is computed for human–human (pairwise across clinician pairs), individual human–LLM, and LLM–human majority. ( 6(a) , 6(b) ) 3-class confusion matrices (LLM vs clinician majority); rows are clinician-majority labels and columns are LLM predictions. ( 6(c) ) LLM accuracy stratified by clinician consensus strength (non-tie questions only).
Figure 7: Complete methodological pipeline showing the flow from dataset preparation through final evaluation.
Figure 8: Example of a persona profile. Displayed values are synthetic to prevent participant re-identification.
Figure 9: Example of the blinded clinician evaluation protocol. Clinicians reviewed the patient scenario and two anonymized responses, then rated each response pair across seven therapeutic criteria using a forced-choice or tie-based comparison.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Demographic composition across complete pool (n=335), selected subset (n=150), training set (n=100), and test set (n=50).
Figure 12: Overall performance rankings calculated as average win rate across all pairwise head-to-head comparisons, showing complete reversal in model rankings between dimensions.
Figure 13: Complete head-to-head win rates for all 21 pairwise comparisons. Each cell displays the row model’s win rate against the column model. Models ordered by performance (best to worst) within each dimension.
Figure 14: Phase 2 performance rankings calculated as average win rate across all pairwise comparisons.
Figure 15: Complete head-to-head win rates for all pairwise comparisons. Models ordered by performance within each dimension.
Figure 16: Statistical significance matrices using McNemar’s test ( α =0.05, n=600). △ = row wins, ∇ = column wins, = = no significant difference.
Figure 17: Representative examples from confusion matrix cells. Response cells are color-coded: purple (both evaluators agree), teal (human winner), pink (LLM winner), white (neither selected).
Figure 18
Figure 19: Prolific validation: Base Model vs. MODPO_Survey (n=200 questions, 10 annotators each, with attention checks). ( 18(a) ) Win rates by criterion; SURVEY is consistently preferred across all seven evaluation criteria. ( 18(b) ) Non-tie binary agreement with human reliability context. ( 18(c) ) Agreement levels (3-class, all questions including ties). ( 19(a) , 19(b) ) 3-class confusion matrices (LLM vs. human majority); rows are human-majority labels, columns are LLM predictions. ( 19(c) ) LLM accuracy stratified by human consensus (non-tie questions only).
Figure 20
Figure 21: Prolific validation: MODPO_Maxim vs. MODPO_Survey (n=100 questions, 18 annotators each, with attention checks). ( 20(a) ) Win rates by criterion. ( 20(b) ) Non-tie binary agreement with human reliability context. ( 20(c) ) Agreement levels (3-class, all questions including ties). ( 21(a) , 21(b) ) 3-class confusion matrices (LLM vs. human majority); rows are human-majority labels, columns are LLM predictions. ( 21(c) ) LLM accuracy stratified by human consensus (non-tie questions only).