cs.CLOct 27, 2026

semanticfa: An R Package for Response-Free Semantic Factor Analysis of Psychometric Scales

Authors: Devon Yanitski, Chris Westbury

Organizations: Department of Psychology, University of Alberta, Edmonton, Alberta, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The items of a reflective psychometric scale can be read as a proposal about a construct's structure. Language models make that proposal measurable by converting each item into an embedding, a list of numbers, similar for similar meanings. We introduce semanticfa, an R package for semantic factor analysis, which factors item-embedding similarities. It also interprets factors, flags redundant items, builds short forms, and screens new items. A worked example analyzes the 50-item International Personality Item Pool Big-Five Factor Markers, embedded with Qwen3-Embedding-8B, and compares the results with a factor analysis of 874,434 respondents. Retention criteria tied between five and six factors, and the tie rule kept five. Three corresponded mainly to single domains, and two blended most Extraversion items with five Agreeableness items. Each factor's congruence with its best-matching human factor (1 = identical) ranged from .92 for Openness to .52 for Agreeableness. Item pairs more similar in meaning tended to be answered more alike (r = .46 with raw responses), but same-domain reverse-keyed opposites, answered oppositely, were about as similar as cross-domain pairs (.58 vs. .57). Sign-flipping reverse-keyed items' embeddings did not improve agreement with raw responses and worsened it with reverse-scored ones. With five factors, a 25-item short form matched the domains better than all 200 random five-per-domain sets, but parallel analysis kept only three. We propose semantic-behavioral coherence, the agreement between semantic and response structures, as a measurable property of a scale, given an embedding model. Semantic factor analysis complements, and does not replace, response-based factor analysis.

Figures & tables

DiagnosticWhat it measuresHow to read it
Kaiser–Meyer–Olkin (KMO)How large the correlations among items are compared with the partial correlations between pairs of items once all other items are held constantFrom 0 to 1. Higher values mean the matrix is better suited to factor analysis.
Root mean square residual (RMSR)The typical size of the differences between the observed similarities and those the factors reproduceLower is better. The range from random embeddings (see Note) shows how little the conventional cutoff discriminates for embeddings. A value above that range is not evidence of misfit.
Common part accounted for (CAF)How much of the common variance the factors capture. It is one minus the KMO of what the factors leave unexplainedFrom 0 to 1. Higher is better. A low value means factorable structure remains after the factors are removed.
Total entropy fit index (TEFI)How cleanly the factors divide the items, measured with Von Neumann entropy ( golino-etal-2021 )Lower is better. It has no conventional cutoff and is read against other solutions. The range from random embeddings (see Note) shows only whether the solution can be told apart from one fitted to random embeddings.
McDonald’s omega totalThe share of the variance of the summed score of a factor’s items that the factor accounts for, computed from the factor’s loadings. It indexes internal consistencyFrom 0 to 1. Higher is better. It is computed twice, over the items whose largest loading is on the factor and over the items of the theoretical domain the factor matches best (ten items in the worked example).
Table 1: Diagnostics reported by sfa() and how to read them
 Theoretical domain  
Factor (matched domain)OCEANItemsReverse-keyed
MR4 (O)7000071
MR1 (C)110111146
MR5 (E)0052070
MR2 (A)204311010
MR3 (N)00048121
Table 2: Assignment of the 50 items to the five semantic factors, by theoretical domain
Figure 1: Item-by-factor loading heatmap for the five-factor semantic solution (mean-centered Pearson encoding of the Qwen3-Embedding-8B embeddings). Rows are items, ordered by theoretical domain (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) and separated by horizontal lines, and columns are factors, ordered by explained variance. Below each factor label is the domain the factor matches best, the domain on whose items its mean absolute loading is largest, as in Table 2 . Two of the factors are blended: MR5 gathers forward-keyed sociability and warmth items and MR2 reverse-keyed items describing the absence of a disposition. Color shows the oblimin pattern loading, from red (negative) through white to blue (positive), on the scale at the right. The call plot(fit, type = "loadings") draws the same map with items sorted by their dominant factor. Item codes give the domain (O, C, E, A, N) and the item’s position in the inventory. MR = minimum residual, the prefix of the psych package’s automatic factor labels.
Figure 2: Cosine similarity ( atomic encoding) among the 50 International Personality Item Pool Big-Five Factor Marker items, grouped by domain in the order Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism ( sfa_corplot() ). All pairwise similarities are positive (range = .394–.938), so the color scale runs from 0.3 (white) to 1 (dark blue). The diagonal is 1 by definition and is excluded from the stated range. Darker cells mark higher similarity. The block structure along the diagonal is clearest among the Neuroticism items N15 to N19 and faint in the other domains. Item codes give the domain (O, C, E, A, N) and the item’s position in the inventory.
Figure 3: Eigenvalues of the mean-centered Pearson similarity matrix of the Qwen3-Embedding-8B embeddings of the 50 items in decreasing order (a scree plot, on a logarithmic scale). The dashed red line is the parallel-analysis threshold, the 95th percentile of the corresponding eigenvalues of 100 similarity matrices of random embeddings, and the dotted line marks an eigenvalue of 1, the Kaiser rule’s cutoff. The first eigenvalue (29.63) is far larger than the rest, and five eigenvalues exceed the parallel-analysis threshold.
Figure 4: Mean-centered Pearson similarity matrices of the 50 International Personality Item Pool items, grouped by domain in the order Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism, for Qwen3-Embedding-0.6B (left), 4B (center), and 8B (right). Each panel shows all 50 items against all 50, and the axes mark each domain’s block with the domain’s initial letter (O, C, E, A, N) at the middle of the block. The three panels share one color scale of mean-centered Pearson similarity, from white at 0.3, the lowest similarity in any of the three matrices rounded down to the nearest tenth, to dark blue at 1, and the color key is marked in steps of 0.1. The diagonal is 1 by definition. The Neuroticism block along the diagonal is clearly visible at every size, and the blocks of the other domains are faint. The text reports two measures of this separation. The point-biserial correlation between sharing a domain and pair similarity is .398 (0.6B), .458 (4B), and .416 (8B), and the mean similarity of pairs within a domain exceeds that of pairs between domains by .077, .077, and .079, respectively.
Figure 5: Two-dimensional item maps of the 50 item embeddings produced by sfa_itemplot() : tt -SNE (top left), UMAP (top right), PCA (bottom left), and classical MDS (bottom right), with items colored by theoretical domain in the same colors in all four panels (legend below the panels). The maps are computed from the Qwen3-Embedding-8B item embeddings (4,096 coordinates) after the default encoding, which centers each embedding across its coordinates and scales it to unit length. In the tt -SNE and UMAP maps, item codes, added for this figure, give the domain (O, C, E, A, N) and the item’s position in the inventory and are placed where they cover no other item or code. The PCA and MDS panels color the items by domain without item codes, because the codes overprint in those panels. tt -SNE = tt -distributed stochastic neighbor embedding, UMAP = uniform manifold approximation and projection, PCA = principal component analysis, MDS = multidimensional scaling.
 Human factor
Semantic factorO (MR5)C (MR2)E (MR4)A (MR3)N (MR1)
O (MR4).92.14.18.17.02
C (MR1).28.84.27.10.11
E (MR5).16.11.64.52.07
A (MR2).27.05.41.52−- .01
N (MR3).12.13.03.30.80
Table 3: Tucker congruence coefficients between the semantic factors (rows) and the human-response factors (columns)
Figure 6: Semantic similarity (mean-centered Pearson encoding of the Qwen3-Embedding-8B item vectors) against human inter-item correlations from raw responses ( N=874,434N=874{,}434 ), for all 1,225 item pairs. Each point is one pair of items. The red line is the linear fit ( r=.46r=.46 ). The dashed black curve is the quadratic fit, which accounts for .40 of the variance in the semantic similarities, against .21 for the line. Similarity rises steeply toward the positive end of the horizontal axis and only slightly toward the negative end.
Figure S1: Depth selection for the Qwen3-Embedding-8B item embeddings. Each panel shows one quantity at every depth evaluated, the number of leading coordinates kept (147 depths spaced 28 apart from 3 to 4,091, plus the full 4,096). Left: normalized mutual information (NMI) between the exploratory graph analysis communities and the theoretical domains. Right: the total entropy fit index (TEFI), for which lower values indicate a cleaner division. The curves show the unrescaled values. The selected depth, marked by the dashed line, is 1,431, the depth at which 0.70×NMI−0.30×TEFI0.70\times\mathrm{NMI}-0.30\times\mathrm{TEFI} is largest after each quantity is rescaled across the depths to run from 0 to 1. NMI there is .704, the maximum of a jagged curve.
 TheoryHuman dataDiagnostics
Encoding or matrixNMIARIϕˉ\bar{\phi}rkeyedr_{\text{keyed}}rrawr_{\text{raw}}rdisr_{\text{dis}}KMOTEFIRMSRCAF
atomic.527.402.744.434.456.707.971−27.72-27.72.032.412
atomic_reversed.527.402.450.137.444–.971−27.72-27.72.032.412
squid.391.272.622.567.4901.000.022−44.39-44.39.072.996
mean_centered_pearson.527.402.744.434.456.707.971−27.71-27.71.032.412
nli.548.448.286.063.561–.551−54.15-54.15.094.688
Table S1: Agreement of each five-factor solution with theory and with human data, and its diagnostics, for the four embedding encodings and the NLI matrix
ItemDomainMR3MR1MR2MR5MR4
E1Extraversion.093.308.076.354.148
E2Extraversion.051.196.449.203.122
E3Extraversion.084.061.127.752.013
E4Extraversion−- .001.546.206.161.008
E5Extraversion.037.205.117.385.257
E6Extraversion.093.175.432.065.190
Table S2: Factor loadings of the 50 International Personality Item Pool Big-Five Factor Marker items in the five-factor semantic solution (mean-centered Pearson encoding of the Qwen3-Embedding-8B embeddings, minimum residual extraction, oblimin rotation)
ItemDomainMR4MR1MR3MR2MR5
E1Extraversion.712.042−- .010−- .015−- .012
E2Extraversion.713.086.053−- .026.008
E3Extraversion.623−- .189.179.061−- .068
E4Extraversion.755−- .023−- .039.018−- .027
E5Extraversion.714.012.123.067.034
E6Extraversion.522.005.073.010.235
Table S3: Factor loadings of the 50 International Personality Item Pool Big-Five Factor Marker items in the five-factor solution of the human responses (correlations of reverse-scored responses, minimum residual extraction, oblimin rotation, N=874,434N=874{,}434 respondents after exclusions)
Figure S2: Signed natural language inference (NLI) similarity, the probability of entailment minus the probability of contradiction, among the 50 International Personality Item Pool Big-Five Factor Marker items, grouped by domain. The color scale runs from −1-1 (red) through 0 (white) to 1 (blue), and the diagonal is 1. Red cells are negative and mark pairs that the cross-encoder judged to contradict each other. The matrix is shown before the repair that made it positive definite for factoring. Every similarity in the cosine matrix of Figure 2 in the main text is positive. Here, by contrast, pairs of items of the same domain averaged .20 when keyed the same way and -.79 when keyed oppositely, which produces negative bands within the domain blocks. Many pairs from different domains are negative as well (71% of them, and 26% below −.5-.5 ). Item codes give the domain (O, C, E, A, N) and the item’s position in the inventory.
Figure S3: Comparison-data misfit profiles for the similarity matrices of three embedding models (Qwen3-Embedding-0.6B, -4B, and -8B) and for one random subsample of n=2,500n=2{,}500 respondents’ reverse-scored responses, drawn with the seed 42. The misfit at each candidate number of factors is the median, across 500 random samples from the comparison population, of the eigenvalue RMSR, the root mean square difference between a sample’s eigenvalues and the observed eigenvalues. The eigenvalue RMSR is a different quantity from the root mean square residual (RMSR) that sfa() reports as a fit diagnostic. The embedding profiles treat the embedding dimensions (1,024 to 4,096) as cases, and the human profile treats respondents as cases. The three model sizes are drawn with different line types (0.6B dotted, 4B dashed, 8B solid), and the human profile is drawn in red. Left: misfit divided by its value at one factor. Right: relative improvement, the percentage reduction in misfit from the preceding number of factors. The human profile shows a sharp elbow at five factors, where the improvement peaks at 57.2% and then drops. The embedding profiles decline gradually. Their largest improvement, 22%, is less than half the human improvement into five factors.
ScaleModelrpbr_{\text{pb}}95% intervalGapNMIARIConsensus
Big Five0.6B.398[.286, .515].077.497.3936
 4B.458[.372, .548].077.693.5746
 8B.416[.316, .535].079.527.4025
DASS0.6B.481[.344, .612].099.412.2715
 4B.505[.349, .649].106.700.6504
 8B.510[.343, .664].124.834.8014
Table S4: Separation of the theoretical scales and agreement of the factor solutions with them, by model size, for the Big Five inventory and the DASS
Figure S4: Mean-centered Pearson similarity matrices of the 42 items of the Depression Anxiety Stress Scales (DASS), grouped by scale in the order Depression, Anxiety, and Stress, for Qwen3-Embedding-0.6B (left), 4B (center), and 8B (right). The axes mark each scale’s block with its initial letter (D, A, S) at the middle of the block. The three panels share one color scale of mean-centered Pearson similarity, from white at 0.2, the lowest similarity in any of the three matrices rounded down to the nearest tenth, to dark blue at 1. The diagonal is 1 by definition. The Depression and Stress blocks stand out more with each larger model, and the Anxiety block less.
More in this topic

Psychometric Validation

Psychometric validation tests whether assessments reliably measure their intended traits or abilities across people, settings, and uses.

See all 67 papers

Explore similar work

CardsList
Oct 26, 2026

When Respondents Are Scarce: LLM-Derived Semantic Priors for Bayesian Confirmatory Factor Analysis

Across social and behavioral research, measuring latent constructs can be challenging when target populations are small or difficult to reach. Recruiting additional respondents may be costly or infeasible, leaving researchers with limited data for estimating questionnaire factor structures. Yet item wording contains structural expectations before data collection, and whether these can improve small-sample estimation without replacing empirical validation remains unclear. We introduce the Semantic-Empirical Updating Framework (SEUF), which converts large language model paraphrases and embeddings, aligned with hypothesized item-factor assignments, into prespecified priors for Bayesian confirmatory factor analysis. We evaluated SEUF using a 38-item survey of 617 industrial-park stakeholders and two public surveys, FEVS and RIASEC. Semantic structure persisted after wording controls across all three instruments. With 50 respondents, SEUF achieved loading precision comparable to conventional CFA with 70-82 respondents, equivalent to 29-39% fewer respondents at the specified precision benchmark. Results showed lower loading errors and improved credible-interval coverage relative to conventional CFA in both public surveys. Gains were greatest when respondent data were limited, highlighting the value of semantic priors under information scarcity. By harnessing structural information already embedded in questionnaires, SEUF advances a more resource-efficient approach to measurement and opens new directions for study design and empirical inference in hard-to-reach populations.

Shaohuai Feng, Mohd Wira Mohd Shafiei, Mohd Sukri Shafie +1

Centre for Global Sustainability Studies (CGSS), Universiti Sains Malaysia, Minden, Penang, Malaysia. · Centre for Education and Training in Renewable Energy, Energy Efficiency and Green Technology (CETREE), Universiti Sains Malaysia, Minden, Penang, Malaysia. · Centre for Instructional Technology and Multimedia, Universiti Sains Malaysia, Minden, Penang, Malaysia.

Latent Variable ModelsPsychometric ValidationBayesian Inference
AskPDF
Sep 25, 2026

Toward Embedding-Based Psychometrics: Structural Modeling of Assessment-Item Semantics With Contextual Scores

Contextual scores represent assessment items through their similarities to reference words in an external corpus. We examine the semantic structure of scores for 40 TIMSS mathematics scored units using a partially specified two-step factor procedure. A search across factor counts identifies a persistent seven-group structure under the featured construction. Subsequent comparisons consistently favor a general dimension alongside group associations, although individual group memberships remain sensitive to some specification choices. Item examples distinguish recurring, cross-domain, sensitive, and imposed associations. Simpler and unrestricted references clarify the contribution and limits of the anchored representation: it improves on a single factor but does not achieve the lowest working Bayesian information criterion (BIC). A separate response benchmark compares three initial Q constructions and their Hull-PVAF revisions under higher-order and saturated attribute distributions. Among these diagnostic models, BIC favors the official content framework and the Akaike information criterion (AIC) favors its direct four-factor augmentation, but a matched unidimensional two-parameter logistic model has lower AIC and BIC than all twelve conditions. These findings support a conditional semantic representation while limiting direct diagnostic interpretation. We discuss learned text-assisted response calibration as a prospective application requiring a larger calibrated item bank and independent evaluation.

Jinsong Chen, Shi-Ting Chen

Faculty of Education, The University of Hong Kong

Educational AssessmentPsychometric ValidationLatent Variable ModelsItem Response Theory
AskPDF
May 26, 2026

Psychological Constructs in Shared Semantic Space

Psychological constructs are often measured in separate instruments, datasets, and research traditions, which makes direct comparison difficult. This paper proposes a framework for making such constructs semantically commensurate by representing and comparing them as directions in a shared word-embedding space. Using Supervised Semantic Differential, we estimate construct-specific semantic gradients from text-outcome associations and project them onto theoretically motivated reference axes. As an initial test case, we use Valence, Arousal, and Dominance (VAD) as an affective coordinate system. First, we recover interpretable VAD directions from English word-level affective norms. Second, we project semantic gradients for 27 GoEmotions categories into this space and recover the expected organization of emotions, especially along valence and arousal. Third, we apply the same procedure to Big Five personality domains and facets derived from IPIP-NEO-300 item-factor associations. Domain-level placements are broadly coherent, while facet-level results are more exploratory because they rely on sparse questionnaire text. The results suggest that embedding spaces can support construct-level comparison across otherwise incommensurable psychological measurements, provided that semantic placements are assessed for stability and interpretability.

Hubert Plisiecki

IDEAS Research Institute

Semantic Mapping
AskPDF