cs.CLOct 27, 2026
Savesemanticfa: An R Package for Response-Free Semantic Factor Analysis of Psychometric Scales
Organizations: Department of Psychology, University of Alberta, Edmonton, Alberta, Canada
Abstract
The items of a reflective psychometric scale can be read as a proposal about a construct's structure. Language models make that proposal measurable by converting each item into an embedding, a list of numbers, similar for similar meanings. We introduce semanticfa, an R package for semantic factor analysis, which factors item-embedding similarities. It also interprets factors, flags redundant items, builds short forms, and screens new items. A worked example analyzes the 50-item International Personality Item Pool Big-Five Factor Markers, embedded with Qwen3-Embedding-8B, and compares the results with a factor analysis of 874,434 respondents. Retention criteria tied between five and six factors, and the tie rule kept five. Three corresponded mainly to single domains, and two blended most Extraversion items with five Agreeableness items. Each factor's congruence with its best-matching human factor (1 = identical) ranged from .92 for Openness to .52 for Agreeableness. Item pairs more similar in meaning tended to be answered more alike (r = .46 with raw responses), but same-domain reverse-keyed opposites, answered oppositely, were about as similar as cross-domain pairs (.58 vs. .57). Sign-flipping reverse-keyed items' embeddings did not improve agreement with raw responses and worsened it with reverse-scored ones. With five factors, a 25-item short form matched the domains better than all 200 random five-per-domain sets, but parallel analysis kept only three. We propose semantic-behavioral coherence, the agreement between semantic and response structures, as a measurable property of a scale, given an embedding model. Semantic factor analysis complements, and does not replace, response-based factor analysis.
Figures & tables
| Diagnostic | What it measures | How to read it |
|---|---|---|
| Kaiser–Meyer–Olkin (KMO) | How large the correlations among items are compared with the partial correlations between pairs of items once all other items are held constant | From 0 to 1. Higher values mean the matrix is better suited to factor analysis. |
| Root mean square residual (RMSR) | The typical size of the differences between the observed similarities and those the factors reproduce | Lower is better. The range from random embeddings (see Note) shows how little the conventional cutoff discriminates for embeddings. A value above that range is not evidence of misfit. |
| Common part accounted for (CAF) | How much of the common variance the factors capture. It is one minus the KMO of what the factors leave unexplained | From 0 to 1. Higher is better. A low value means factorable structure remains after the factors are removed. |
| Total entropy fit index (TEFI) | How cleanly the factors divide the items, measured with Von Neumann entropy ( golino-etal-2021 ) | Lower is better. It has no conventional cutoff and is read against other solutions. The range from random embeddings (see Note) shows only whether the solution can be told apart from one fitted to random embeddings. |
| McDonald’s omega total | The share of the variance of the summed score of a factor’s items that the factor accounts for, computed from the factor’s loadings. It indexes internal consistency | From 0 to 1. Higher is better. It is computed twice, over the items whose largest loading is on the factor and over the items of the theoretical domain the factor matches best (ten items in the worked example). |
Table 1: Diagnostics reported by sfa() and how to read them
| Theoretical domain | |||||||
| Factor (matched domain) | O | C | E | A | N | Items | Reverse-keyed |
| MR4 (O) | 7 | 0 | 0 | 0 | 0 | 7 | 1 |
| MR1 (C) | 1 | 10 | 1 | 1 | 1 | 14 | 6 |
| MR5 (E) | 0 | 0 | 5 | 2 | 0 | 7 | 0 |
| MR2 (A) | 2 | 0 | 4 | 3 | 1 | 10 | 10 |
| MR3 (N) | 0 | 0 | 0 | 4 | 8 | 12 | 1 |
Table 2: Assignment of the 50 items to the five semantic factors, by theoretical domain
Figure 1: Item-by-factor loading heatmap for the five-factor semantic solution (mean-centered Pearson encoding of the Qwen3-Embedding-8B embeddings). Rows are items, ordered by theoretical domain (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) and separated by horizontal lines, and columns are factors, ordered by explained variance. Below each factor label is the domain the factor matches best, the domain on whose items its mean absolute loading is largest, as in Table 2 . Two of the factors are blended: MR5 gathers forward-keyed sociability and warmth items and MR2 reverse-keyed items describing the absence of a disposition. Color shows the oblimin pattern loading, from red (negative) through white to blue (positive), on the scale at the right. The call plot(fit, type = "loadings") draws the same map with items sorted by their dominant factor. Item codes give the domain (O, C, E, A, N) and the item’s position in the inventory. MR = minimum residual, the prefix of the psych package’s automatic factor labels.
Figure 2: Cosine similarity ( atomic encoding) among the 50 International Personality Item Pool Big-Five Factor Marker items, grouped by domain in the order Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism ( sfa_corplot() ). All pairwise similarities are positive (range = .394–.938), so the color scale runs from 0.3 (white) to 1 (dark blue). The diagonal is 1 by definition and is excluded from the stated range. Darker cells mark higher similarity. The block structure along the diagonal is clearest among the Neuroticism items N15 to N19 and faint in the other domains. Item codes give the domain (O, C, E, A, N) and the item’s position in the inventory.
Figure 3: Eigenvalues of the mean-centered Pearson similarity matrix of the Qwen3-Embedding-8B embeddings of the 50 items in decreasing order (a scree plot, on a logarithmic scale). The dashed red line is the parallel-analysis threshold, the 95th percentile of the corresponding eigenvalues of 100 similarity matrices of random embeddings, and the dotted line marks an eigenvalue of 1, the Kaiser rule’s cutoff. The first eigenvalue (29.63) is far larger than the rest, and five eigenvalues exceed the parallel-analysis threshold.
Figure 4: Mean-centered Pearson similarity matrices of the 50 International Personality Item Pool items, grouped by domain in the order Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism, for Qwen3-Embedding-0.6B (left), 4B (center), and 8B (right). Each panel shows all 50 items against all 50, and the axes mark each domain’s block with the domain’s initial letter (O, C, E, A, N) at the middle of the block. The three panels share one color scale of mean-centered Pearson similarity, from white at 0.3, the lowest similarity in any of the three matrices rounded down to the nearest tenth, to dark blue at 1, and the color key is marked in steps of 0.1. The diagonal is 1 by definition. The Neuroticism block along the diagonal is clearly visible at every size, and the blocks of the other domains are faint. The text reports two measures of this separation. The point-biserial correlation between sharing a domain and pair similarity is .398 (0.6B), .458 (4B), and .416 (8B), and the mean similarity of pairs within a domain exceeds that of pairs between domains by .077, .077, and .079, respectively.
Figure 5: Two-dimensional item maps of the 50 item embeddings produced by sfa_itemplot() : -SNE (top left), UMAP (top right), PCA (bottom left), and classical MDS (bottom right), with items colored by theoretical domain in the same colors in all four panels (legend below the panels). The maps are computed from the Qwen3-Embedding-8B item embeddings (4,096 coordinates) after the default encoding, which centers each embedding across its coordinates and scales it to unit length. In the -SNE and UMAP maps, item codes, added for this figure, give the domain (O, C, E, A, N) and the item’s position in the inventory and are placed where they cover no other item or code. The PCA and MDS panels color the items by domain without item codes, because the codes overprint in those panels. -SNE = -distributed stochastic neighbor embedding, UMAP = uniform manifold approximation and projection, PCA = principal component analysis, MDS = multidimensional scaling.
| Human factor | |||||
|---|---|---|---|---|---|
| Semantic factor | O (MR5) | C (MR2) | E (MR4) | A (MR3) | N (MR1) |
| O (MR4) | .92 | .14 | .18 | .17 | .02 |
| C (MR1) | .28 | .84 | .27 | .10 | .11 |
| E (MR5) | .16 | .11 | .64 | .52 | .07 |
| A (MR2) | .27 | .05 | .41 | .52 | .01 |
| N (MR3) | .12 | .13 | .03 | .30 | .80 |
Table 3: Tucker congruence coefficients between the semantic factors (rows) and the human-response factors (columns)
Figure 6: Semantic similarity (mean-centered Pearson encoding of the Qwen3-Embedding-8B item vectors) against human inter-item correlations from raw responses ( ), for all 1,225 item pairs. Each point is one pair of items. The red line is the linear fit ( ). The dashed black curve is the quadratic fit, which accounts for .40 of the variance in the semantic similarities, against .21 for the line. Similarity rises steeply toward the positive end of the horizontal axis and only slightly toward the negative end.
Figure S1: Depth selection for the Qwen3-Embedding-8B item embeddings. Each panel shows one quantity at every depth evaluated, the number of leading coordinates kept (147 depths spaced 28 apart from 3 to 4,091, plus the full 4,096). Left: normalized mutual information (NMI) between the exploratory graph analysis communities and the theoretical domains. Right: the total entropy fit index (TEFI), for which lower values indicate a cleaner division. The curves show the unrescaled values. The selected depth, marked by the dashed line, is 1,431, the depth at which is largest after each quantity is rescaled across the depths to run from 0 to 1. NMI there is .704, the maximum of a jagged curve.
| Theory | Human data | Diagnostics | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Encoding or matrix | NMI | ARI | KMO | TEFI | RMSR | CAF | ||||
| atomic | .527 | .402 | .744 | .434 | .456 | .707 | .971 | .032 | .412 | |
| atomic_reversed | .527 | .402 | .450 | .137 | .444 | – | .971 | .032 | .412 | |
| squid | .391 | .272 | .622 | .567 | .490 | 1.000 | .022 | .072 | .996 | |
| mean_centered_pearson | .527 | .402 | .744 | .434 | .456 | .707 | .971 | .032 | .412 | |
| nli | .548 | .448 | .286 | .063 | .561 | – | .551 | .094 | .688 | |
Table S1: Agreement of each five-factor solution with theory and with human data, and its diagnostics, for the four embedding encodings and the NLI matrix
| Item | Domain | MR3 | MR1 | MR2 | MR5 | MR4 |
|---|---|---|---|---|---|---|
| E1 | Extraversion | .093 | .308 | .076 | .354 | .148 |
| E2 | Extraversion | .051 | .196 | .449 | .203 | .122 |
| E3 | Extraversion | .084 | .061 | .127 | .752 | .013 |
| E4 | Extraversion | .001 | .546 | .206 | .161 | .008 |
| E5 | Extraversion | .037 | .205 | .117 | .385 | .257 |
| E6 | Extraversion | .093 | .175 | .432 | .065 | .190 |
Table S2: Factor loadings of the 50 International Personality Item Pool Big-Five Factor Marker items in the five-factor semantic solution (mean-centered Pearson encoding of the Qwen3-Embedding-8B embeddings, minimum residual extraction, oblimin rotation)
| Item | Domain | MR4 | MR1 | MR3 | MR2 | MR5 |
|---|---|---|---|---|---|---|
| E1 | Extraversion | .712 | .042 | .010 | .015 | .012 |
| E2 | Extraversion | .713 | .086 | .053 | .026 | .008 |
| E3 | Extraversion | .623 | .189 | .179 | .061 | .068 |
| E4 | Extraversion | .755 | .023 | .039 | .018 | .027 |
| E5 | Extraversion | .714 | .012 | .123 | .067 | .034 |
| E6 | Extraversion | .522 | .005 | .073 | .010 | .235 |
Table S3: Factor loadings of the 50 International Personality Item Pool Big-Five Factor Marker items in the five-factor solution of the human responses (correlations of reverse-scored responses, minimum residual extraction, oblimin rotation, respondents after exclusions)
Figure S2: Signed natural language inference (NLI) similarity, the probability of entailment minus the probability of contradiction, among the 50 International Personality Item Pool Big-Five Factor Marker items, grouped by domain. The color scale runs from (red) through 0 (white) to 1 (blue), and the diagonal is 1. Red cells are negative and mark pairs that the cross-encoder judged to contradict each other. The matrix is shown before the repair that made it positive definite for factoring. Every similarity in the cosine matrix of Figure 2 in the main text is positive. Here, by contrast, pairs of items of the same domain averaged .20 when keyed the same way and -.79 when keyed oppositely, which produces negative bands within the domain blocks. Many pairs from different domains are negative as well (71% of them, and 26% below ). Item codes give the domain (O, C, E, A, N) and the item’s position in the inventory.
Figure S3: Comparison-data misfit profiles for the similarity matrices of three embedding models (Qwen3-Embedding-0.6B, -4B, and -8B) and for one random subsample of respondents’ reverse-scored responses, drawn with the seed 42. The misfit at each candidate number of factors is the median, across 500 random samples from the comparison population, of the eigenvalue RMSR, the root mean square difference between a sample’s eigenvalues and the observed eigenvalues. The eigenvalue RMSR is a different quantity from the root mean square residual (RMSR) that sfa() reports as a fit diagnostic. The embedding profiles treat the embedding dimensions (1,024 to 4,096) as cases, and the human profile treats respondents as cases. The three model sizes are drawn with different line types (0.6B dotted, 4B dashed, 8B solid), and the human profile is drawn in red. Left: misfit divided by its value at one factor. Right: relative improvement, the percentage reduction in misfit from the preceding number of factors. The human profile shows a sharp elbow at five factors, where the improvement peaks at 57.2% and then drops. The embedding profiles decline gradually. Their largest improvement, 22%, is less than half the human improvement into five factors.
| Scale | Model | 95% interval | Gap | NMI | ARI | Consensus | |
|---|---|---|---|---|---|---|---|
| Big Five | 0.6B | .398 | [.286, .515] | .077 | .497 | .393 | 6 |
| 4B | .458 | [.372, .548] | .077 | .693 | .574 | 6 | |
| 8B | .416 | [.316, .535] | .079 | .527 | .402 | 5 | |
| DASS | 0.6B | .481 | [.344, .612] | .099 | .412 | .271 | 5 |
| 4B | .505 | [.349, .649] | .106 | .700 | .650 | 4 | |
| 8B | .510 | [.343, .664] | .124 | .834 | .801 | 4 |
Table S4: Separation of the theoretical scales and agreement of the factor solutions with them, by model size, for the Big Five inventory and the DASS
Figure S4: Mean-centered Pearson similarity matrices of the 42 items of the Depression Anxiety Stress Scales (DASS), grouped by scale in the order Depression, Anxiety, and Stress, for Qwen3-Embedding-0.6B (left), 4B (center), and 8B (right). The axes mark each scale’s block with its initial letter (D, A, S) at the middle of the block. The three panels share one color scale of mean-centered Pearson similarity, from white at 0.2, the lowest similarity in any of the three matrices rounded down to the nearest tenth, to dark blue at 1. The diagonal is 1 by definition. The Depression and Stress blocks stand out more with each larger model, and the Anxiety block less.
More in this topic
Psychometric Validation
Psychometric validation tests whether assessments reliably measure their intended traits or abilities across people, settings, and uses.
See all 67 papers