cs.CLFeb 13, 2026

Discovering Semantic Structures in Psychological Scales: A Response-Free Front-End to Efficient Simplification

Authors: Bo Wang, Yuxuan Zhang, Yueqin Hu, Hanchao Hou, Kaiping Peng, Shiguang Ni

Organizations: Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China · Shenzhen International Graduate School, Tsinghua University, Shenzehen, China. · Faculty of Psychology, Beijing Normal University, Beijing, China. · Beijing Key Laboratory of Applied Experimental Psychology, National Demonstration Center for Experimental Psychology Education, Faculty of Psychology, Beijing Normal University, Beijing, China

Abstract

Psychological scale refinement traditionally relies on response-based methods such as factor analysis, item response theory, and network psychometrics to uncover latent constructs and optimize item composition. Although rigorous, these approaches require large samples and may be constrained by data availability and cross-cultural comparability. Recent advances in natural language processing suggest that the wording of scale items contains construct-relevant semantic organization, offering a complementary response-free perspective. We introduce a topic-modeling framework that operationalizes item-level semantic structure for systematic scale simplification. Items are encoded using contextual sentence embeddings and grouped through density-based clustering to identify semantic topics without predefining their number. Within each cluster, class-based term weighting identifies representative keywords that summarize each topic and support an interpretable semantic pre-structure of the item pool. Semantically related clusters are hierarchically merged, and representative items are selected using structure-aware membership criteria within an integrated reduction pipeline. We evaluated the framework on three widely used instruments: DASS, IPIP, and EPOCH, covering negative affect, personality traits, and well-being. Results showed that the identified semantic topics often formed factor-like groupings aligned with established constructs. Representative items selected through semantic membership probabilities reduced item counts by 60.5% on average while maintaining acceptable psychometric properties and preserving factor correlation patterns. Our framework formalizes semantic topic structure as an inspectable front-end for scale construction and simplification. To facilitate adoption, we provide a visualization-supported tool enabling one-click semantic analysis and structured simplification.

Figures & tables

Explore similar work

Oct 27, 2026

semanticfa: An R Package for Response-Free Semantic Factor Analysis of Psychometric Scales

The items of a reflective psychometric scale can be read as a proposal about a construct's structure. Language models make that proposal measurable by converting each item into an embedding, a list of numbers, similar for similar meanings. We introduce semanticfa, an R package for semantic factor analysis, which factors item-embedding similarities. It also interprets factors, flags redundant items, builds short forms, and screens new items. A worked example analyzes the 50-item International Personality Item Pool Big-Five Factor Markers, embedded with Qwen3-Embedding-8B, and compares the results with a factor analysis of 874,434 respondents. Retention criteria tied between five and six factors, and the tie rule kept five. Three corresponded mainly to single domains, and two blended most Extraversion items with five Agreeableness items. Each factor's congruence with its best-matching human factor (1 = identical) ranged from .92 for Openness to .52 for Agreeableness. Item pairs more similar in meaning tended to be answered more alike (r = .46 with raw responses), but same-domain reverse-keyed opposites, answered oppositely, were about as similar as cross-domain pairs (.58 vs. .57). Sign-flipping reverse-keyed items' embeddings did not improve agreement with raw responses and worsened it with reverse-scored ones. With five factors, a 25-item short form matched the domains better than all 200 random five-per-domain sets, but parallel analysis kept only three. We propose semantic-behavioral coherence, the agreement between semantic and response structures, as a measurable property of a scale, given an embedding model. Semantic factor analysis complements, and does not replace, response-based factor analysis.
Sep 25, 2026

Toward Embedding-Based Psychometrics: Structural Modeling of Assessment-Item Semantics With Contextual Scores

Contextual scores represent assessment items through their similarities to reference words in an external corpus. We examine the semantic structure of scores for 40 TIMSS mathematics scored units using a partially specified two-step factor procedure. A search across factor counts identifies a persistent seven-group structure under the featured construction. Subsequent comparisons consistently favor a general dimension alongside group associations, although individual group memberships remain sensitive to some specification choices. Item examples distinguish recurring, cross-domain, sensitive, and imposed associations. Simpler and unrestricted references clarify the contribution and limits of the anchored representation: it improves on a single factor but does not achieve the lowest working Bayesian information criterion (BIC). A separate response benchmark compares three initial Q constructions and their Hull-PVAF revisions under higher-order and saturated attribute distributions. Among these diagnostic models, BIC favors the official content framework and the Akaike information criterion (AIC) favors its direct four-factor augmentation, but a matched unidimensional two-parameter logistic model has lower AIC and BIC than all twelve conditions. These findings support a conditional semantic representation while limiting direct diagnostic interpretation. We discuss learned text-assisted response calibration as a prospective application requiring a larger calibrated item bank and independent evaluation.
May 6, 2026

Measuring Psychological States Through Semantic Projection: A Theory-Driven Approach to Language-Based Assessment

Recent advances in natural language processing have enabled increasingly accurate estimation of psychological traits from language. However, most existing approaches rely on supervised models trained to predict questionnaire scores, limiting interpretability and generalizability across contexts. The present study introduces a theory-driven and fully unsupervised framework for measuring psychological states directly from natural language using semantic projection. Psychological constructs were operationalized as interpretable semantic axes derived from lexical anchors and items from validated clinical scales assessing depression, anxiety, and worry. Participants textual responses were embedded using Sentence-BERT and projected onto these axes to generate continuous psychological scores across multiple response formats, including selected words, generated words, phrases, and free-text responses. Projection scores were evaluated through correlations with standardized clinical measures , split-half reliability analyses, attenuation corrections, distributional similarity using Wasserstein distance, and comparisons with lexicon-based sentiment analysis (VADER). Results showed strong associations between projection scores and clinical measures, particularly for structured formats such as selected words, written words, and phrases. Free-text responses produced weaker results when analyzed as whole texts, but performance improved substantially when sentence-level aggregation strategies were applied. These findings support semantic projection as an interpretable and scalable alternative to supervised language models for psychological assessment and highlight the importance of response format and text-processing strategies in language-based mental health measurement.