PRiME: Pragmatic Reasoning In MEntal health

Sneha Oram; Tejomay Kishor Padole; Pushpak Bhattacharyya; Suyash P. Awate

cs.CLJul 31, 2025

PRiME: Pragmatic Reasoning In MEntal health

Authors: Sneha Oram, Tejomay Kishor Padole, Pushpak Bhattacharyya, Suyash P. Awate

Organizations: Indian Institute of Technology, Bombay

Abstract

In this work, the task and domain interaction of reasoning and mental health are investigated. As large language models (LLMs) present exceptional text understanding and fluent text generation capability, we analyze whether they can reason in the high-stakes domain of mental health. We begin by introducing three datasets and three tasks leveraging the existing work in pragmatic reasoning. The analysis constitutes an evaluation of $12$ LLMs, including MentaLLaMA, which is a mental health reasoning and explanation generation model. The performance of the models remains on par with human evaluation, with the Mistral-7B model giving the best performance. We further analyze the performance of the models with error asymmetry ratio and conduct first-order attention rollout experiment with MentaLLaMA-7B. We present the results for the community reporting the architectural behaviour of the models for the input text and recommend further careful analysis for the high-stakes domain.

Figures & tables

Figure 1: Overview of our streamlined pragmatics reasoning tasks in mental health with examples.

Tasks	Label counts & proportions
Task 1: Agreement Detection	#Agree: $700$ ( $50\%$ ) #Disagree: $700$ ( $50\%$ )
Task 2: Implicature NLI	#Definitely true: $500$ ( $35.78\%$ ) #Definitely false: $497$ ( $35.57\%$ ) #Might be true: $400$ ( $28.63\%$ )
Task 3: Presupposition NLI	#Definitely true: $497$ ( $35.70\%$ ) #Definitely false: $494$ ( $35.48\%$ ) #Might be true: $401$ ( $28.80\%$ )

Table 1: PRiME dataset statistics for three redefined tasks for implicature and presupposition in mental health.

Models
mistralai/Mistral-7B-Instruct-v0.2 Jiang et al. (2023)
mistralai/Mistral-7B-Instruct-v0.3 Jiang et al. (2024)
Qwen/Qwen3-4B-Instruct-2507 Team (2025b)
Qwen/Qwen2.5-7B-Instruct Team (2024b)
Qwen/Qwen2.5-1.5B-Instruct Team (2024b)
google/gemma-2-2b-it Team (2024a)

Table 2: List of twelve LLMs used for evaluation on the PRiME dataset

Table 3: Performance of LLMs on the Agreement Detection task. ZS, KS, and CoT refer to zero-shot, k-shot, and chain-of-thought prompt settings, respectively.

Figure 2: Accuracy of the models with increasing model size on zero-shot (ZS) prompt setting.

Figure 3: First-order attention rollout experiment on the MentaLLaMA-7B model for the tasks Agreement Detection, Implicature NLI, and Presupposition NLI from left to right, respectively.

Table 6: Error ratio of Mistral

3

7

B for the NLI tasks

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Models names Details
Mistral $2$ - $7$ B mistralai/Mistral-7B-Instruct-v0.2
Mistral $3$ - $7$ B mistralai/Mistral-7B-Instruct-v0.3
Qwen $3$ - $4$ B Qwen/Qwen3-4B-Instruct-2507
Qwen $2.5$ - $7$ B Qwen/Qwen2.5-7B-Instruct
Qwen $2.5$ - $1.5$ B Qwen/Qwen2.5-1.5B-Instruct
Gemma $2$ - $2$ B google/gemma-2-2b-it

Appendix

Table 7: Models used for the evaluation of our proposed PRiME benchmark

Figure 4: Average F1-score aggregated for the prompt setting across each label for the three tasks.

Tasks	GT-Human	Human-LLM
Task 1: Agreement	0.921	0.769
Task 2: Implicature	0.695	0.541
Task 3: Presupposition	0.601	0.486

Appendix

Table 8: Phi coefficient

(\phi)

correlations among Ground Truth (GT), Human evaluators (Human), and Mistral

3

7

B (LLMs).

Table 9: Error ratio of the models with respect to the label ‘might be true’for the Implicature NLI task.

Table 10: Error ratio of the models with respect to the label ‘might be true’for the Presupposition NLI task.

Figure 5: Versatility of the models on the three tasks based on their performance in the zero-shot setting.

Table 11: Bootstrapped accuracy of all models on the Agreement Detection task with a 95% confidence interval. Values in square brackets indicate the 2.5th and 97.5th percentile accuracies.

Table 12: Bootstrapped accuracy of all models on the Implicature NLI task with a 95% confidence interval. Values in square brackets indicate the 2.5th and 97.5th percentile accuracies.

Table 13: Bootstrapped accuracy of all models on the Presupposition NLI task with a 95% confidence interval. Values in square brackets indicate the 2.5th and 97.5th percentile accuracies.

Figure 6: Bootstrapped accuracies for each model for Agreement Detection in a zero-shot setting.

Figure 7: First-order attention rollout on the premise and hypothesis pair from implicature NLI task with label ‘true’.

Figure 8: First-order attention rollout on the premise and hypothesis pair from implicature NLI task with label ‘false’.

Figure 9: First-order attention rollout on the premise and hypothesis pair from implicature NLI task with label ‘might be true’.

Language Model Reasoning Evaluation

See all 93 papers

OMIND: Framework for Knowledge Grounded Finetuning and Multi-Turn Dialogue Benchmark for Mental Health LLMs
Mar 26, 2026Suraj Racha, Prashant Harish Joshi, Utkarsh Maurya +6Medical LLMs LLM Fine-Tuning
Indian Institute of Technology Bombay
HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench
Aug 25, 2026Matthew Flathers, Phuong Anh Nguyen, Jill Noorily +7Clinical Language Model Evaluation LLM Evaluation
Division of Digital Psychiatry, Beth Israel Deaconess Medical Center, Boston, MA · Department of Psychiatry, Beth Israel Deaconess Medical Center, Boston, MA
Mental-R1: Aligning LLM Reasoning for Mental Health Assessment
Jun 11, 2026Xin Wang, Boyan Gao, Yibo Yang +1Cognitive Modeling Medical Diagnosis
Department of Engineering Science, University of Oxford, Oxford, U.K · Department of Engineering Science, University of Oxford, U.K. · Oxford Suzhou Centre for Advanced Research, Suzhou, China

Mar 26, 2026J/K move · Enter open · S save

OMIND: Framework for Knowledge Grounded Finetuning and Multi-Turn Dialogue Benchmark for Mental Health LLMs

Suraj Racha, Prashant Harish Joshi, Utkarsh Maurya, Nitin Yadav+5

Indian Institute of Technology Bombay

Medical LLMs LLM Fine-Tuning LLM Grounding Clinical Language Model Evaluation

Open paper AskPDF arXiv

Large Language Models (LLMs) have shown remarkable capabilities for complex tasks, yet adaptation in medical domain, specifically mental health, poses specific challenges. Mental health is a rising concern globally with LLMs having large potential to help address the same. We highlight three primary challenges for LLMs in mental health - lack of high quality interpretable and knowledge grounded training data; training paradigms restricted to core capabilities, and evaluation of multi turn dialogue settings. Addressing it, we present oMind framework which includes training and aligning LLM agents for diverse capabilities including conversations; high quality ~164k multi-task SFT dataset, as a result of our generation pipeline based on Structured Knowledge retrieval, LLM based pruning, and review actions. We also introduce oMind-Chat - a novel multi turn benchmark dataset with expert annotated turn level and conversation level rubrics. Our diverse experiments on both core capabilities and conversations shows oMind LLMs consistently outperform baselines. oMind-LLM also shows significantly better reasoning with up to 80% win rate.

Abstract

Language Model Reasoning Evaluation

Explore similar work

OMIND: Framework for Knowledge Grounded Finetuning and Multi-Turn Dialogue Benchmark for Mental Health LLMs

HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench

Mental-R1: Aligning LLM Reasoning for Mental Health Assessment