PsyJev: Psychological Analysis via System-1 Judgment

Fangyuan Liu; Sirui Zhao; Xiaohao Wang; Guo Cheng; Peng Zhang; Enhong Chen

cs.AIOct 27, 2026

PsyJev: Psychological Analysis via System-1 Judgment

Authors: Fangyuan Liu, Sirui Zhao, Xiaohao Wang, Guo Cheng, Peng Zhang, Enhong Chen

Organizations: State Key Laboratory of Cognitive Intelligence, School of Computer Science and Technology, University of Science and Technology of China · Computer Science, Beijing Institute of Technology · School of Information Science and Technology, University of Science and Technology of China

Abstract

Self-disclosures on the Web support diverse psychological analyses, from recognizing expressed experiences to identifying symptom-relevant information. We study a shared, question-conditioned decision interface for this heterogeneous problem in content analysis. We organize PsyJudgeBench, comprising 13 annotated datasets and 82 questions across six psychological areas, while preserving source-specific criteria and masks for unobserved labels. We introduce PsyJev, a System-1 model that directly produces distributions over answers specified in each request. Joint supervised training establishes the shared judgment function, followed by Judgment-Anchored RLCD (JA-RLCD), which adapts logit-space reward optimization to heterogeneous psychological judgments through balanced supervision, selective retention of correct SFT predictions, and question-group reward optimization. The Qwen3.5-based 2B, 4B, and 9B models achieve 64.33, 66.21, and 67.56 average test F1, respectively. The 9B model exceeds the strongest external comparison by 2.54 points. Controlled 2B comparisons examine the training stages and anchored refinement. Further analyses examine question-conditioned judgment, inference cost, and probability quality, complemented by qualitative comparisons across models.

Figures & tables

Figure 1. PsyJudgeBench coverage. The inner groups organize 13 source datasets into six psychological content areas. Q denotes the number of questions associated with each source, and the outer bars and numbers indicate source-record counts. A concentric benchmark overview groups thirteen datasets into six psychological content areas and displays each dataset's question count and number of source records.

Figure 2. PsyJev overview. (a) System-1-style direct judgment through question-conditioned input serialization and constrained legal-answer scoring. (b) Two-stage training with joint SFT followed by JA-RLCD, combining judgment anchoring with logit-space exploration and question-group reward optimization. The prediction pathway serializes the text, question, criterion, and labeled candidates, encodes them with a Qwen3.5 backbone, and normalizes answer-position vocabulary logits over the legal answer identifiers. The training pathway applies supervised fine-tuning followed by JA-RLCD, which explores answer logits, evaluates Brier-based question-group rewards, and updates the judgment policy with pathwise reward gradients and a judgment anchor.

Classification and Decision Models
	Depression / Anxiety				Self / Relationships			Stress / Distress		Cognition		Wellness	Crisis	Avg.	Inference
Model	DepEmo	ANGST	PRIMATE	BDISen	IRF	LoST	LonXpl	SAD	CAMS	ThinkTrap	TherapQA	MultiWD	CRADLE	Avg.	Latency (ms)
MentalBERT	52.09	43.38	27.03	8.61	65.00	17.10	77.43	66.60	38.46	0.00	7.82	37.88	0.00	33.95	0.46
MentalRoBERTa	60.74	58.21	33.11	5.13	75.45	31.55	80.66	67.87	39.06	0.00	10.51	38.69	0.00	38.54	0.46
ModernBERT-large	68.95	64.22	48.06	42.90	82.65	67.12	83.17	71.31	46.76	15.47	19.25	53.49	20.08	52.57	1.16
GLiClass	75.78	69.70	56.97	57.36	83.41	68.01	84.77	73.47	49.66	12.95	19.62	62.99	38.88	57.97	1.90

Table 1. Performance comparison on PsyJudgeBench (F1, %). Avg. denotes the unweighted mean across 13 datasets. Bold indicates the best result within each benchmark-adapted model group. Jev is evaluated zero-shot. Latency measures model computation (ms/judgment) on the fixed H100 CUDA Graph probe. Abbreviations: DepEmo (DepressionEmo), BDISen (BDI-Sen), LonXpl (LonXplain), ThinkTrap (Thinking Traps), and TherapQA (TherapistQA).

(a) Training pipeline
Training configuration	Avg. F1 $\uparrow$	$\Delta$ F1
SFT only	60.68	-3.65
SFT $\rightarrow$ continued CE	61.72	-2.61
SFT $\rightarrow$ adapted RLCD	58.87	-5.46
SFT $\rightarrow$ JA-RLCD (full)	64.33	+0.00
(b) Refinement and matched controls

Table 2. 2B ablations and matched-budget controls.

\Delta

F1 is relative to full JA-RLCD, computed before rounding.

JA-RLCD vs.	All Q $\Delta$ F1	Difficult Q $\Delta$ F1	Improved Q (%)
SFT	+6.23	+19.45	57.32
Continued CE	+4.36	+13.38	56.10
Anchor only	+1.38	+3.02	58.54

Table 3. Question-averaged test-F1 gains of JA-RLCD across all 82 questions and 16 validation-selected difficult questions.

Model	Both correct $\uparrow$	Same-answer errors $\downarrow$
GLiClass	55.84	41.88
Gemma-2-9B	67.85	29.84
Qwen3.5-4B answer	66.24	31.95
PsyJev-4B	70.06	28.09

Table 4. Question-conditioned judgment on opposite-label question pairs. Both correct and same-answer errors are reported as source-balanced percentages.

Figure 3. F1–latency comparison of three 4B configurations. PsyJev gains 2.38 F1 points over answer-SFT first-token scoring at comparable GPU latency, while running 2.10

\times

faster than full-answer scoring under the shared timing protocol. Horizontal bars compare PsyJev at 12.01 milliseconds per judgment, first-token scoring at 11.87 milliseconds, and full-answer scoring at 25.25 milliseconds. An arrow indicates the speedup over full-answer scoring, while an annotation marks the F1 improvement over first-token scoring.

Model	F1 $\uparrow$	NLL $\downarrow$	Brier $\downarrow$	ECE $\downarrow$
MentalBERT	33.95	0.659	0.3619	5.82
MentalRoBERTa	38.54	0.620	0.3405	4.75
ModernBERT-large	52.57	0.580	0.3089	5.75
GLiNER2	59.01	0.544	0.2869	4.95
FLAN-T5-XL	52.88	0.588	0.3220	6.41
PsyJev-4B	66.21	0.536	0.2777	4.58

Table 5. Classification performance and probability quality on PsyJudgeBench. F1 and ECE are percentages. NLL is normalized by the log answer-set size. Bold indicates the best result among the displayed models.

Figure 4. Source-mean ECE accumulated over the full 0–100% confidence range. Curves sum contributions from all 15 bins; their 100% endpoints reproduce Table 5 . Lower endpoints indicate better overall calibration. Six curves show cumulative contributions to source-mean ECE for PsyJev-4B, GLiNER2, ModernBERT-large, FLAN-T5-XL, MentalBERT, and MentalRoBERTa. Both axes start at zero. The horizontal axis covers all 15 confidence bins from zero to 100 percent; the vertical axis measures cumulative ECE in percentage points. PsyJev-4B ends at 4.58, the lowest overall ECE among the six models, with a relative reduction of 3.5 to 28.5 percent over the five baselines.

Gold	GLiClass	Gemma-2-9B	Qwen3.5-4B Answer SFT	PsyJev-4B SFT	PsyJev-4B JA-RLCD
A. Co-occurring emotions DepressionEmo : Multi-label emotion recognition
“ Very lonely in high school, then very lonely in the military. Eventually I was able to have a group of friends for a while and it filled the void , but I ditched them, I get lonely now and again ”
Emptiness, Loneliness	Loneliness $\times$	Emptiness, Loneliness	Emptiness, Loneliness, Sadness $\times$	Emptiness, Loneliness, Sadness $\times$	Emptiness, Loneliness
Interpretation. PsyJev recovers Emptiness and Loneliness without the additional Sadness label predicted by the answer model and SFT.
B. Competing distress cues CAMS : Primary distress reason
“i have a part-time job and struggle to even get through 4 hour shifts . so i don’t make enough money to do anything but attend work basically and smoke some cigarettes. […] i’m also struggling with a severely broken heart from an on-and-off 10 year thing”

Table 6. Illustrative judgment corrections by PsyJev-4B after JA-RLCD.

\times

indicates disagreement with source annotations. Displayed texts are excerpts of the original test inputs.

Scale	Model / training	F1 (%)
2B	Qwen3.5 answer-SFT	74.32
	PsyJev SFT	73.91
	PsyJev JA-RLCD	77.76
4B	Qwen3.5 answer-SFT	76.53
	PsyJev SFT	77.87
	PsyJev JA-RLCD	78.70

Table 7. Dreaddit transfer on all 715 test examples (F1, %). No Dreaddit supervision is used.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Depression / anxiety
Source	Q	Train	Val.	Test
DepressionEmo	8	4,219	907	911
ANGST	2	6,769	763	2,872
PRIMATE	9	1,593	207	203
BDI-Sen	21	1,759	261	503
Self / relationships

Appendix

Table 8. Source records and coordinated splits. Q counts registry questions: 79 binary and three multiclass in total.

Source	Registry instruction and legal answers
DepressionEmo	Does the author express sadness in this post? Answers: no; yes
PRIMATE	Does this post provide enough information to answer the PHQ-9 item about Feeling down depressed or hopeless? Judge answerability, not whether the symptom is present. Answers: no; yes
BDI-Sen	Does this sentence provide information relevant to the BDI-II symptom Sadness? Judge the source relevance label, not severity or symptom presence. Answers: no; yes
SAD	What is the primary stressor category expressed in this sentence? Answers: Financial Problem; Other; Everyday Decision Making; Emotional Turmoil; School; Family Issues; Social Relationships; Work; Health, Fatigue, or Physical Pain

Appendix

Table 9. Verbatim registry instructions and legal answers for four judgment types in PsyJudgeBench.

Cross-partition check	Full text	512 tokens
Exact normalized matches	0 (0.00%)	0 (0.00%)
Jaccard $\geq 0.85$ , $\geq 8$ shingles	0	0
Containment ( $\geq 80$ characters)	0	0

Appendix

Table 10. Cross-partition overlap on full texts and 512-token inputs.

Figure 5. Quality–latency comparison of the 16 models with average F1

\geq 57\%

in Table 1 . A scatter plot focuses on models with average F1 of at least 57 percent and uses a linear latency axis. It shows PsyJev at 2B, 4B, and 9B, alongside thirteen baselines. Dashed arrows pair PsyJev and Qwen3.5 answer-SFT at matching backbone sizes.

Source	Brier $\downarrow$	ECE (%) $\downarrow$
ANGST	0.4436	18.54
BDI-Sen	0.1499	1.85
CAMS	0.5408	4.33
CRADLE	0.0644	0.76
DepressionEmo	0.2327	1.06
IRF	0.1895	3.97

Appendix

Table 11. Probability quality of the final PsyJev-4B by source. The mean gives equal weight to all 13 sources.

Model / training	F1 (%)
GLiClass	66.56
GLiNER2	70.94
FLAN-T5-XL	74.84
Llama-3.2-3B	78.00
Phi-4-mini	73.27
Gemma-3-4B	78.26

Appendix

Table 12. Complete Dreaddit transfer results (F1, %) without additional training on Dreaddit.

Model	Both correct $\uparrow$	Same answer	Both wrong
GLiClass	55.84	41.88	2.28
Gemma-2-9B	67.85	29.84	2.31
Qwen3.5-4B answer	66.24	31.95	1.82
PsyJev-4B SFT	63.55	34.37	2.09
PsyJev-4B	70.06	28.09	1.84

Appendix

Table 13. Different questions about the same text. Outcomes (%) use identical opposite-label binary pairs and equal source weights. “Same answer” means exactly one prediction is wrong in this subset.

Case	Answer probability / margin	SFT	JA-RLCD
A	Emptiness	0.752	0.825
A	Loneliness	0.999	1.000
A	Sadness	0.571	0.445
B	Jobs and careers	0.303	0.402
B	Relationship	0.406	0.389
B	$p_{y}-p_{c}$	-0.103	+0.013

Appendix

Table 14. Answer probabilities before and after JA-RLCD for the three cases in Table 6 .

Computational Psycholinguistics

Computational psycholinguistics models how people learn, produce, and understand language using formal theories, algorithms, and behavioral data.

See all 89 papers

PsycheMME: A Psychology-Informed Benchmark for Social Understanding and Interaction in Videos
Oct 20, 2026Heran Hu, Yuhang Dong, Zhucun Xue +3Video-Language Model Evaluation Video Understanding
Zhejiang University, APRIL Lab · China University of Geosciences
Evaluating and Benchmarking the System One Model Jev
Sep 29, 2026Tobias Deußer, Lorenz Sparrenberg, Rafet SifaMultilingual Language Model Evaluation LLM Evaluation
University of Bonn, Bonn, Germany · Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany · Fraunhofer IAIS, Sankt Augustin, Germany
System1-Arena: Which Decisions Can Be Left to System 1?
Oct 15, 2026Tianyu Chen, Xinyang Liu, Ruyi An +2LLM Decision-Making LLM Evaluation
The University of Texas at Austin

Oct 20, 2026J/K move · Enter open · S save

PsycheMME: A Psychology-Informed Benchmark for Social Understanding and Interaction in Videos

Heran Hu, Yuhang Dong, Zhucun Xue, Kaiwei Chen+2

Zhejiang University, APRIL Lab · China University of Geosciences

Video-Language Model Evaluation Video Understanding Video Reasoning Human Behavior Modeling Language Model Reasoning Evaluation

Open paper AskPDF arXiv

Understanding human needs and selecting responses require distinct judgments that aggregate accuracy can obscure. We introduce PsycheMME, a video benchmark informed by psychology, with eight capabilities spanning grounding, states, needs, interaction decisions, memory, and risk. Drawing on Social Information Processing and the Perception-Action Cycle, this organization separates human understanding from response selection. We construct 1,000 questions from 630 episodes through independent answering with proposed references withheld and expert review. Our Reasoning-Verified protocol weights answer credit by the support and logical consistency of explicit rationales against annotated cues. Across 17 models, Timing has the lowest Verified scores and credit retention, with weaker support judgments for correct Timing answers. Models with similar Need scores still differ in answer completeness on open questions. Input controls and SFT/GRPO examine context dependence and adaptation. Evaluating answers alongside rationales locates where models fail across understanding, response selection, and justification.

Abstract

Computational Psycholinguistics

Explore similar work

PsycheMME: A Psychology-Informed Benchmark for Social Understanding and Interaction in Videos

Evaluating and Benchmarking the System One Model Jev

System1-Arena: Which Decisions Can Be Left to System 1?