Discovering Semantic Structures in Psychological Scales: A Response-Free Front-End to Efficient Simplification
Organizations: Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China · Shenzhen International Graduate School, Tsinghua University, Shenzehen, China. · Faculty of Psychology, Beijing Normal University, Beijing, China. · Beijing Key Laboratory of Applied Experimental Psychology, National Demonstration Center for Experimental Psychology Education, Faculty of Psychology, Beijing Normal University, Beijing, China
Abstract
Psychological scale refinement traditionally relies on response-based methods such as factor analysis, item response theory, and network psychometrics to uncover latent constructs and optimize item composition. Although rigorous, these approaches require large samples and may be constrained by data availability and cross-cultural comparability. Recent advances in natural language processing suggest that the wording of scale items contains construct-relevant semantic organization, offering a complementary response-free perspective. We introduce a topic-modeling framework that operationalizes item-level semantic structure for systematic scale simplification. Items are encoded using contextual sentence embeddings and grouped through density-based clustering to identify semantic topics without predefining their number. Within each cluster, class-based term weighting identifies representative keywords that summarize each topic and support an interpretable semantic pre-structure of the item pool. Semantically related clusters are hierarchically merged, and representative items are selected using structure-aware membership criteria within an integrated reduction pipeline. We evaluated the framework on three widely used instruments: DASS, IPIP, and EPOCH, covering negative affect, personality traits, and well-being. Results showed that the identified semantic topics often formed factor-like groupings aligned with established constructs. Representative items selected through semantic membership probabilities reduced item counts by 60.5% on average while maintaining acceptable psychometric properties and preserving factor correlation patterns. Our framework formalizes semantic topic structure as an inspectable front-end for scale construction and simplification. To facilitate adoption, we provide a visualization-supported tool enabling one-click semantic analysis and structured simplification.
Figures & tables
| Parameter | Meaning | Typical value used in paper |
|---|---|---|
| UMAP | ||
| n_neighbors | Controls the balance between preserving local versus global structure by setting the number of neighboring points considered. | 3 |
| n_components | Specifies the dimensionality of the reduced embedding space. | 5 |
| min_dist | Determines how tightly points are packed together in the low-dimensional embedding. | 0.0 |
| HDBSCAN | ||
| min_cluster_size | Defines the minimum number of items required to form a cluster. | 2 |
| Scale name | Scale origin | Number of items | Number and names of the factors | Data origin | Number of data records |
| DASS | ( Lovibond, 1995 ) | 42 | 3, Depression, Anxiety, Stress | ( OpenPsychometrics.org, 2019 ) | 39,775 |
| IPIP | ( Goldberg and others, 1999 ) | 50 | 5, Extraversion, Agreeableness, Conscientiousness, Emotional Stability, Intellect | ( OpenPsychometrics.org, 2018 ) | 1,015,341* |
| EPOCH (Chinese version) | ( Kern et al., 2016 ) (English), ( Kern et al., 2019 ) (Chinese) | 20 | 5, Engagement, Perseverance, Optimism, Connectedness, Happiness. | ( Zeng et al., 2019b ; Zeng et al., 2019a ) | 17,854 |
| *: We filtered records with missing item(s), and randomly allocated 20,000 data for evaluation. All reversed items were | |||||
| rewritten into their counterparts prior to analysis to prevent the encoding and topic modeling from being driven by | |||||
| superficial polarity cues than substantive item meaning (refer to Appendix A - IPIP for original and rephrased items, | |||||
| Topic No. | Corresponding Factor | Selected Items | Topic Keywords (Top 3) |
|---|---|---|---|
| 0 | Anxiety | Q4, Q7, Q20, Q25 | absence, physical, heart |
| 1 | Depression | Q3, Q5, Q10, Q13 | feel, couldn, life |
| 2 | Stress | Q1, Q6, Q8, Q11 | get, upset, hard |
| Model | df | /df | RMSEA | CFI | TLI | SRMR | |
|---|---|---|---|---|---|---|---|
| Three-factor | 8354.955 | 51 | 163.823 | 0.064 | 0.956 | 0.943 | 0.041 |
| One-factor | 26483.266 | 54 | 490.431 | 0.111 | 0.859 | 0.828 | 0.061 |
| Stress | Depression | Anxiety | |||
|---|---|---|---|---|---|
| Item | Factor Loading | Item | Factor Loading | Item | Factor Loading |
| Q1 | 0.788 | Q3 | 0.778 | Q4 | 0.730 |
| Q6 | 0.696 | Q5 | 0.751 | Q7 | 0.703 |
| Q8 | 0.673 | Q10 | 0.770 | Q20 | 0.703 |
| Q11 | 0.800 | Q13 | 0.821 | Q25 | 0.687 |
| Stress | Depression | Anxiety | |||
| Item | CITC | Item | CITC | Item | CITC |
| Q1 | 0.696 | Q3 | 0.714 | Q4 | 0.653 |
| Q6 | 0.639 | Q5 | 0.684 | Q7 | 0.619 |
| Q8 | 0.556 | Q10 | 0.719 | Q20 | 0.553 |
| Q11 | 0.706 | Q13 | 0.719 | Q25 | 0.614 |
| Cronbach’s | 0.824 | Cronbach’s | 0.862 | Cronbach’s | 0.797 |
| Topic No. | Corresponding Factor | Selected Items | Topic Keywords (Top 3) |
|---|---|---|---|
| 0 | EXT (Extraversion) | Q1, Q2, Q3, Q5 (original item ids: EXT {1,2,3,5}) | people, talk, attention |
| 1 | EST (Emotional Stability, or Neuroticism) | Q12, Q13, Q14, Q16 (EST {2,3,4,6}) | stay, calm, relax |
| 2 | CSN (Conscientiousness) | Q31, Q32, Q33, Q34 (CSN {1,2,3,4}) | order, thing, keep |
| 3 | AGR (Agreeableness) | Q24, Q25, Q26, Q28 (AGR {4,5,6,8}) | feel, people, other |
| 4 | OPN (Openness, or Intellect) | Q42, Q43, Q45, Q46 (OPN {2,3,5,6}) | idea, understand, imagination |
| Model | df | /df | RMSEA | CFI | TLI | SRMR | |
|---|---|---|---|---|---|---|---|
| Five-factor | 11258.01 | 160 | 70.363 | 0.059 | 0.860 | 0.833 | 0.057 |
| One-factor | 53927.25 | 173 | 311.718 | 0.125 | 0.320 | 0.253 | 0.129 |
| Topic No. | Corresponding Factor | Selected Items | Topic Keywords (Top 3) |
|---|---|---|---|
| 0 | E (Engagement) | Q7, Q12 (original item ids: E2, E4) | time, forget,involve |
| 1 | O (Optimism) | Q13, Q18 (O2, O4) | believe, good, thing |
| 2 | H (Happiness) | Q4, Q20 (H1, H4) | happy, fun, life |
| 3 | P (Perseverance) | Q2, Q9 (P1, P2) | finish, until, stick |
| 4 | C (Connectedness) | Q1, Q10 (C1, C2) | care, people, share |
| Model | df | /df | RMSEA | CFI | TLI | SRMR | |
|---|---|---|---|---|---|---|---|
| Five-factor | 771.534 | 25 | 30.861 | 0.041 | 0.983 | 0.970 | 0.018 |
| One-factor | 5895.535 | 35 | 168.444 | 0.097 | 0.870 | 0.833 | 0.050 |
| Semantic cluster | Dominant factor | Proportion | Number of items |
|---|---|---|---|
| Topic 0 | Anxiety | 0.8 | 15 |
| Topic 1 | Depression | 1 | 14 |
| Topic 2 | Stress | 1 | 11 |
| Topic 3 | Anxiety | 1 | 2 |
| ARI | 0.745 | ||
| Semantic cluster | Dominant factor | Proportion | Number of items |
|---|---|---|---|
| Topic 0 | Extraversion (EXT) | 0.83 | 12 |
| Topic 1 | Emotional Stability (EST) | 1 | 10 |
| Topic 2 | Conscientiousness (CSN) | 1 | 10 |
| Topic 3 | Agreeableness (AGR) | 1 | 8 |
| Topic 4 | Intellect/Openness (OPN) | 1 | 8 |
| Topic 5 | Intellect/Openness (OPN) | 1 | 2 |
| Semantic cluster | Dominant factor | Proportion | Number of items |
|---|---|---|---|
| Topic 0 | Engagement (E) | 1 | 4 |
| Topic 1 | Optimism (O) | 1 | 4 |
| Topic 2 | Happiness (H) | 1 | 4 |
| Topic 3 | Perseverance (P) | 1 | 4 |
| Topic 4 | Connectedness (C) | 1 | 4 |
| ARI | 1.000 | ||
| No. of items per topic | CFI | TLI |
|---|---|---|
| 3 | 0.857 | 0.812 |
| 4 | 0.860 | 0.833 |
| 5 | 0.808 | 0.782 |
| 6 | 0.821 | 0.803 |
| 7 | 0.807 | 0.791 |
| 8 | 0.776 | 0.760 |
| Topics (nr_topic) | CFI | TLI |
|---|---|---|
| 2 | 0.841 | 0.811 |
| 3 | 0.864 | 0.838 |
| 4 | 0.845 | 0.815 |
| 5 | 0.875 | 0.852 |
| 6 | 0.860 | 0.833 |
| ‘auto’ (6) | 0.857 | 0.830 |
| Parameter | Setting | Reduction result | Jaccard | Kept | Changed |
|---|---|---|---|---|---|
| default | n_neighbors = 3 min_cluster_size=2 min_samples = 1 | {1,3,4,5,6,7,8,10,11,13,20,25} | |||
| n_neighbors | 2 | {1,3,4,5,7,8,9,10,12,13,14,19} | 0.500 | 8 | 4 |
| (UMAP) | 10 | {1,2,3,4,5,6,7,8,10,13,16,17} | 0.600 | 9 | 3 |
| min_cluster_size | 4 | {1,3,4,6,7,9,10,11,15,16,17,18} | 0.412 | 7 | 5 |
| (HDBSCAN) | 6 | {1,3,4,5,6,7,10,11,12,15,16,19} | 0.500 | 8 | 4 |
| Parameter | Setting | Reduction result | Jaccard | Kept | Changed |
|---|---|---|---|---|---|
| default | n_neighbors = 3 min_cluster_size=2 min_samples = 1 | {1,2,3,5,12,13,14,16,24,25,26,28, 31,32,33,34,42,43,45,46} | |||
| n_neighbors | 2 | {1,2,3,6,7,11,12,13,14,23,24,25, 32,33,34,35,42,43,44,46} | 0.538 | 14 | 6 |
| (UMAP) | 10 | {1,2,3,11,12,13,14,15,16,17,20,21, 22,31,32,33,34,41,42,43} | 0.481 | 13 | 7 |
| min_cluster_size | 4 | {1,2,3,4,11,14,15,16,21,23,24,25, 31,32,33,34,41,42,43,44} | 0.481 | 13 | 7 |
| (HDBSCAN) | 6 | {1,2,3,4,5,6,8,9,11,14,15,16, 31,32,34,36,41,42,43,44} | 0.379 | 11 | 9 |
| Parameter | Setting | Reduction result | Jaccard | Kept | Changed |
|---|---|---|---|---|---|
| default | n_neighbors = 3 min_cluster_size=2 min_samples = 1 | {1,2,4,7,9,10,12,13,18,20} | |||
| n_neighbors | 2 | {2,4,5,6,7,9,10,14,15,18} | 0.429 | 6 | 4 |
| (UMAP) | 10 | {1,2,4,5,6,7,9,10,13,18} | 0.667 | 8 | 2 |
| min_cluster_size | 4 | {1,2,3,4,5,6,7,9,10,13} | 0.538 | 7 | 3 |
| (HDBSCAN) | 6 | Unassigned* | / | / | / |
| Scale or item-pool condition | Expected suitability | Interpretive implication |
|---|---|---|
| High within-factor semantic coherence and clear between-factor boundaries (e.g., EPOCH-CN) | Higher suitability | Semantic topics are more likely to align with established dimensions and support direct candidate short-form generation. |
| Broad, facet-rich constructs with heterogeneous content within each dimension (e.g., IPIP) | Moderate suitability, with caution | Semantic selection may preserve content coverage, but retained items may show lower internal homogeneity in very short subscales. |
| Semantically overlapping or “sticky” constructs (e.g., DASS) | Moderate suitability, with caution | Topic–factor deviations may reveal meaningful construct overlap rather than simple algorithmic error. |
| Narrow wording-specific or symptom-specific subthemes (e.g., somatic-symptom items in DASS) | Useful for diagnosis, but may over-fragment | The method can highlight content granularity, but researchers should decide whether such subthemes are theoretically central or peripheral. |
| Constructs primarily defined by response processes, item difficulty, or external criteria rather than item meaning | Lower suitability as a stand-alone guide | Semantic structure should be treated only as preliminary content evidence and must be supplemented by psychometric validation. |