Toward Embedding-Based Psychometrics: Structural Modeling of Assessment-Item Semantics With Contextual Scores

Jinsong Chen; Shi-Ting Chen

cs.CLSep 25, 2026

Toward Embedding-Based Psychometrics: Structural Modeling of Assessment-Item Semantics With Contextual Scores

Authors: Jinsong Chen, Shi-Ting Chen

Organizations: Faculty of Education, The University of Hong Kong

Abstract

Contextual scores represent assessment items through their similarities to reference words in an external corpus. We examine the semantic structure of scores for 40 TIMSS mathematics scored units using a partially specified two-step factor procedure. A search across factor counts identifies a persistent seven-group structure under the featured construction. Subsequent comparisons consistently favor a general dimension alongside group associations, although individual group memberships remain sensitive to some specification choices. Item examples distinguish recurring, cross-domain, sensitive, and imposed associations. Simpler and unrestricted references clarify the contribution and limits of the anchored representation: it improves on a single factor but does not achieve the lowest working Bayesian information criterion (BIC). A separate response benchmark compares three initial Q constructions and their Hull-PVAF revisions under higher-order and saturated attribute distributions. Among these diagnostic models, BIC favors the official content framework and the Akaike information criterion (AIC) favors its direct four-factor augmentation, but a matched unidimensional two-parameter logistic model has lower AIC and BIC than all twelve conditions. These findings support a conditional semantic representation while limiting direct diagnostic interpretation. We discuss learned text-assisted response calibration as a prospective application requiring a larger calibrated item bank and independent evaluation.

Figures & tables

Feature	Contextual-score matrix	Student-response matrix
Row unit	Frequent-word semantic profile across 40 items	Student responses across administered items
Origin	Computed conditional on encoder, corpus, template, and vocabulary	Sampled behavior under the TIMSS booklet design
Row relation	Word profiles share corpus and encoder geometry; no i.i.d. sampling claim is made	Students are observational units under a complex booklet design; item coverage is sparse
Primary outputs	Semantic loadings, posterior inclusion support, and descriptive item-factor associations	Item parameters and residual dependence
Role here	Constructs and perturbs the text-side representation	Evaluates limited behavioral bridges only

Table 1: Construction and evaluation matrices.

Figure 1: Analytical flow. Contextual scores support the graded semantic map as the primary structural output. Three response-free binary constructions enter the response benchmark. Hull-PVAF uses student responses to revise separate Q copies; all six mappings are evaluated under higher-order and saturated attribute distributions, alongside a matched 2PL reference. Diagnostic eligibility is assessed separately from the existence of the graded representation.

Layer	Count evidence	Structural evidence	Reporting outcome
L1	$K_{p}\in\mathcal{K}_{20\%}$	Persistent	Persistent structure with count support
L2	$K_{p}\notin\mathcal{K}_{20\%}$ and $\mathcal{K}_{20\%}$ is nonempty	Persistent	Persistent structure with count uncertainty
L3	Count paths may be usable or unusable	All candidate counts fail the persistence rule	No persistent structure identified
Unclassified	Any	Persistence unresolved, or persistent $K_{p}$ with no finite count reading	Report the evidence limitation; do not relabel it L3

Table 2: Logical layers for one persistence profile. Cross-profile agreement is reported separately.

Factor count $K$	$\phi(K,K+1)$	$\phi(K,K+2)$	$\phi(K,K+3)$
4	.986	.695	.647
5	.451	.342	.322
6	.928	.855	.878
7	.900	.926	.924
8	.742	.604	.385
9	.589	.325	.384

Table 3: Direct persistence horizons.

Depth	$\mathcal{K}_{20\%}$	Profile	Persistent counts	$K_{p}$ /layer	$g_{CP}$
2	8, 9	.85/one	5, 7	7/L2	1
2	8, 9	.80/two	5, 7	7/L2	1
2	8, 9	.70/three	5	5/L2	3
4	7, 9	.85/one	4, 6, 7	7/L1	0
4	7, 9	.80/two	6, 7	7/L1	0
4	7, 9	.70/three	6, 7	7/L1	0

Table 4: Depth-by-profile persistence.

Definition	Effective $p$ (O/B)	Oblique BIC	Bifactor BIC	$\Delta$ BIC	Verdict
Depth 4, globally disjoint	174/173	92,910.9	89,432.5	3,478.4	Bifactor
Depth 4, overlap permitted	175/173	92,646.2	90,182.8	2,463.3	Bifactor
Depth 2, overlap permitted	174/171	91,800.9	89,702.4	2,098.5	Bifactor
Depth 2, globally disjoint	174/180	91,800.8	90,094.9	1,705.9	Bifactor

Table 5: Step-2 representation fits.

Model	Factors	Nominal/rank $p$	Working AIC	Working BIC	Hard SRMR
One factor	1	80/80	125,201.8	125,644.2	.0712
Framework PCFA	4	133/133	103,758.2	104,493.7	.0622
Retained oblique	7	174/174	91,948.7	92,910.9	.0676
Retained bifactor	8	173/173	88,475.8	89,432.5	.0278
Unrestricted eight	8	360/332	83,570.3	85,406.2	.0574

Table 6: Text-model references on the common contextual-score matrix.

Factor	SSL share	Relative effect $D/D_{G}$	Mapped	Three largest loadings	Composition or stem concentration
G	75.8%	1.000	40	23 (+.929), 6 (+.924), 10 (+.919)	Positive global elevation
S1	6.4%	.175	19	29 (+.607), 27 (+.569), 19 (+.442)	Algebra 11; number 7; geometry 1
S2	3.9%	.152	13	35 (+.720), 40 (+.640), 15 (+.327)	Data 6; algebra 4; number 3
S3	3.6%	.185	10	31 (+.717), 34 (+.661), 32 (+.270)	Geometry 4; data 3; number 3
S4	1.3%	.056	11	11 (+.338), 6 (+.271), 5 (+.188)	Number 8; algebra 3
S5	3.0%	.086	18	25 (+.472), 24 (+.361), 14 (+.332)	Broad/sign mixed; M042198 share = 39.9%

Table 7: Factor composition and effects.

Item and task	Association	Interpretation
M042245: identify the equation fitting two coordinate pairs	Adopted S1 loading .607, PIP 1.000; free and selected in all four definitions	The algebra-centered group captures a recognizably algebraic task.
M042186: extend a subtraction pattern to a negative subtrahend	Official number domain; PCFA algebra addition .923, PIP 1.000	Contextual scores connect an arithmetic item with algebraic pattern extension. The initial PCFA Q retains its official number membership.
M032047: express the sum of three consecutive integers when the middle is $2n$	Adopted S6 loading .100, PIP .662; free and selected in two of four definitions	A formally algebraic expression can have a weak, definition-sensitive secondary association.
M042169C: compare effects on the mean and median after increasing the largest staff count	Adopted S6 loading .615; imposed marker	Shared context and statistical content both contribute plausible interpretations. This imposed cell has no estimated PIP.

Table 8: Substantive readings of recurring, cross-domain, sensitive, and imposed associations.

Comparison	Numerical result	Interpretation
General profiles across definitions	Centered $r=.9141$ –.9978; $\phi\geq.9987$	Broad common dimension with some relative-profile variation
Matched group profiles	Minimum $\phi=.5986$ –.8797 across definition pairs	Less uniform group patterns
237 always-free cells	33 selected throughout; 145 never selected; 59 change status	Estimated-cell recurrence, excluding imposed markers
Frequency cutoff 35	1,330 words; minimum group $\phi=.9947$	Close agreement conditional on the adopted design
Frequency cutoff 50	1,080 words; minimum group $\phi=.9919$	Close agreement conditional on the adopted design
Dot-product scores	Minimum group $\phi=.7655$ ; centered G $r=.8631$	Metric choice changes some associations

Table 9: Stability and sensitivity of the continuous representation.

Construction	Attributes	Initial memberships	Hull additions/removals	Revised memberships
Official content framework ( $Q_{\mathrm{i}1}$ , $Q_{\mathrm{r}1}$ )	4	40	3 / 1	42
Semantic group projection ( $Q_{\mathrm{i}2}$ , $Q_{\mathrm{r}2}$ )	7	93	0 / 2	91
Framework-guided PCFA ( $Q_{\mathrm{i}3}$ , $Q_{\mathrm{r}3}$ )	4	87	2 / 3	86

Table 10: Initial Q constructions and empirical revision inventory.

Panel A. CDM comparisons
Q/model	Dist.	$p$	log $L$	AIC	BIC
$Q_{\mathrm{i}1}$	Sat.	95	-30,896.98	61,983.96	62,619.30
$Q_{\mathrm{i}1}$	HO	88	-30,933.93	62,043.87	62,632.39
$Q_{\mathrm{r}1}$	Sat.	99	-30,873.91	61,945.82	62,607.91
$Q_{\mathrm{r}1}$	HO	92	-30,908.08	62,000.16	62,615.43
$Q_{\mathrm{i}2}$	Sat.	417	-30,486.56	61,807.13	64,595.93

Table 11: Response-model comparisons on the common sample.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Panel A. Item-column marginals
Data object	Skewness, median (range)	Excess kurtosis, median (range)
Cosine contextual scores	$.14\;(-.14,\ 1.03)$	$.69\;(.34,\ 4.29)$
Dot-product contextual scores	$.37\;(-.11,\ 1.22)$	$.83\;(.24,\ 4.11)$
Binary item responses	$-.45\;(-2.74,\ 1.97)$	$-1.39\;(-2.00,\ 5.53)$

Appendix

Table 13: Marginal shape and row-profile concentration.

Stage	Setting
Step 1	vbpm 0.9.0; R 4.5.1; seed 2026; maximum 10,000 iterations; convergence tolerance $10^{-4}$
Step-1 prior path	$v_{0}=.01,.005,.002,.001$ ; raw PIP threshold .50
Candidate design	24 specifications and 276 fits; content $K=4$ –14 and cognitive $K=3$ –14
Count and persistence	20% BIC/ELBO paths; direct minimum congruence .85/one, .80/two, and .70/three
Step 2	vbpm 0.9.1; four fixed definitions; oblique and bifactor within each; BIC tie band 2
Text references	vbpm 0.9.1; same prior path; 10,000 iterations per stage, with one 20,000-iteration escalation if needed; tolerance $10^{-4}$

Appendix

Table 14: Estimation and decision settings.

Depth	Backbone	Corpus	Spec.	$K_{\mathrm{BIC},20\%}$	$K_{\mathrm{ELBO},20\%}$	.85/one	.80/two	.70/three
2	Cogn.	E	AZ	7	9	–/L3	–/L3	–/L3
2	Cogn.	E	AO	7	9	3/L2	3/L2	3/L2
2	Cogn.	M	AZ	8	8	6/L2	–/L3	–/L3
2	Cogn.	M	AO	7	9	5/L2	3/L2	3/L2
2	Cont.	E	AZ	10	10	7/L2	–/L3	–/L3
2	Cont.	E	AO	8	9	5/L2	–/L3	–/L3

Appendix

Table 15: Complete Step-1 persistence results.

Step	ELBO gain (%)	BIC gain (%)	Adjacent $\phi_{\min}$	RMSD max	Collision
$4\rightarrow 5$	100.0	100.0	.986	.096	No
$5\rightarrow 6$	56.3	56.9	.451	.227	No
$6\rightarrow 7$	48.9	36.4	.928	.160	No
$7\rightarrow 8$	29.7	1.6	.900	.181	No
$8\rightarrow 9$	20.7	30.6	.742	.228	No
$9\rightarrow 10$	14.7	-17.9	.589	.213	No

Appendix

Table 16: Featured Step-1 count path and adjacent comparisons.

Framework and domain	Depth 2	Depth 4	Depth 6
Content: algebra	22, 28	20, 22, 27, 28	19, 20, 22, 25, 27, 28
Content: data and chance	35, 40	35, 36, 39, 40	35, 36, 37, 38, 39, 40
Content: geometry	31, 32	31, 32, 33, 34	31, 32, 33, 34 (cap)
Content: number	2, 6	2, 4, 5, 6	2, 3, 4, 5, 6, 12
Cognitive: applying	7, 31	7, 21, 31, 39	7, 9, 21, 31, 33, 39
Cognitive: knowing	13, 19	4, 13, 19, 36	4, 13, 19, 30, 35, 36

Appendix

Table 17: Complete inherited anchor assignments.

Definition	Backbone markers per S1–S4	S5 markers	S6 markers	S7 markers
Depth-4 overlap	4	9, 15, 35, 40	15, 36, 37, 38	7, 9, 13, 33
Depth-4 disjoint	4	9, 16, 17, 23	1, 15, 37, 38	7, 8, 13, 18
Depth-2 overlap	2	36, 38	13, 33	35, 40
Depth-2 disjoint	2	36, 38	13, 33	9, 15

Appendix

Table 18: Added-column markers.

Off-diagonal residual summary	Hard-selected covariance	Continuous covariance
Root mean square	.0284	.0150
Mean absolute	.0219	.0105
Maximum absolute	.1066	.0824

Appendix

Table 19: Text-covariance approximation residuals.

Panel A. Nine-factor EFA projection
Threshold	Memberships	Density	Unassigned	Multiple
$\|\lambda\|>.30$	44	.1222	0	4
Panel B. Raw-embedding cluster alignment with content
Method	Clusters	Cramér’s $V$	ARI
$k$ -means	4	.5566	.2327
$k$ -means	8	.8373	.3170

Appendix

Table 20: Descriptive summaries of alternative item representations.

No.	G	S1	S2	S3	S4	S5	S6	S7
1	$\mathbf{.821}$	$-.052$	$.063$	$-.152$	$-.014$	$-.196$	$.159$	$.004$
2	$\mathbf{.918}$	$.060$	$-.041$	$-.076$	$.146$	$-.105$	$-.009$	$.044$
3	$\mathbf{.902}$	$.121$	$-.060$	$.080$	$.159$	$-.009$	$.027$	$.056$
4	$\mathbf{.846}$	$.229$	$.102$	$.146$	$.124$	$.112$	$.044$	$-.037$
5	$\mathbf{.892}$	$-.021$	$-.045$	$.066$	$.188$	$-.057$	$-.037$	$.194$
6	$\mathbf{.924}$	$-.031$	$.056$	$.020$	$.271$	$-.029$	$-.037$	$-.002$

Appendix

Table 21: Adopted bifactor loadings.

No.	G	S1	S2	S3	S4	S5	S6	S7
1	$1.000^{*}$	$.047$	$.088$	$\mathbf{.999}$	$.014$	$\mathbf{1.000}$	$1.000^{*}$	$.013$
2	$1.000^{*}$	$.071$	$.029$	$.185$	$1.000^{*}$	$\mathbf{.756}$	$.013$	$.032$
3	$1.000^{*}$	$\mathbf{.950}$	$.070$	$.230$	$\mathbf{1.000}$	$.013$	$.018$	$.058$
4	$1.000^{*}$	$\mathbf{1.000}$	$\mathbf{.695}$	$\mathbf{.998}$	$1.000^{*}$	$\mathbf{.867}$	$.032$	$.025$
5	$1.000^{*}$	$.015$	$.033$	$.102$	$1.000^{*}$	$.060$	$.024$	$\mathbf{1.000}$
6	$1.000^{*}$	$.020$	$.056$	$.015$	$1.000^{*}$	$.019$	$.024$	$.012$

Appendix

Table 22: Graded semantic item–factor map.

No.	Item	Alg.	Data	Geom.	Num.	No.	Item	Alg.	Data	Num.
1	M032064	0	$\mathbf{1}$	$\mathbf{1}^{\dagger}$	$1^{*}$	21	M042086	$1^{*}$	0	$\mathbf{1}^{\dagger}$
2	M032094	$\mathbf{1}$	0	$\mathbf{1}^{\dagger}$	$1^{*}$	22	M042103	$1^{*}$	0	0
3	M032166	$\mathbf{1}$	0	$\mathbf{1}$	$1^{*}$	23	M042198A	$1^{*}$	$\mathbf{1}$	$\mathbf{1}$
4	M032626	$\mathbf{1}$	$\mathbf{1}$	$\mathbf{1}$	$1^{*}$	24	M042198B	$1^{*}$	$\mathbf{1}$	$\mathbf{1}$
5	M032725	0	0	0	$1^{*}$	25	M042198C	$1^{*}$	$\mathbf{1}$	$\mathbf{1}$
6	M042032	0	0	0	$1^{*}$	26	M042226	$1^{*}$	$\mathbf{1}$	0

Appendix

Table 23: Initial framework-guided PCFA Q-matrix, with imposed and selected cells distinguished.

Revision	Item	Attribute	Edit
${Q_{\mathrm{i}1}\rightarrow Q_{\mathrm{r}1}}$	M032132	algebra	Addition
${Q_{\mathrm{i}1}\rightarrow Q_{\mathrm{r}1}}$	M052216	data and chance	Addition
${Q_{\mathrm{i}1}\rightarrow Q_{\mathrm{r}1}}$	M042198A	data and chance	Addition
${Q_{\mathrm{i}1}\rightarrow Q_{\mathrm{r}1}}$	M032132	data and chance	Removal
${Q_{\mathrm{i}2}\rightarrow Q_{\mathrm{r}2}}$	M052214	S4	Removal
${Q_{\mathrm{i}2}\rightarrow Q_{\mathrm{r}2}}$	M032295	S6	Removal

Appendix

Table 24: All Hull-PVAF changes relative to the three initial Q matrices.

Definition	Reduced groups	Reduced zero rows	Reduced identity witness	Full identity witness	Joint pass
Depth-4 disjoint	5	2	No	No	No
Depth-4 overlap	5	2	Yes	No	No
Depth-2 overlap	5	4	No	No	No
Depth-2 disjoint	6	0	No	No	No

Appendix

Table 25: Diagnostic-feasibility screen.

Panel A. Diagnostics for the twelve CDM conditions
Q	Dist.	Boundary items/cells	Nonmonotone items/decreases	Pair RMS
$Q_{\mathrm{i}1}$	Sat.	1/1	0/0	.0201
$Q_{\mathrm{i}1}$	HO	1/1	0/0	.0206
$Q_{\mathrm{r}1}$	Sat.	3/3	2/3	.0198
$Q_{\mathrm{r}1}$	HO	3/3	2/2	.0204
$Q_{\mathrm{i}2}$	Sat.	22/112	23/155	.0190

Appendix

Table 26: Response-model diagnostics and additional model comparisons.

Panel A. Four text-based projections
Definition	$p$	log $L$	AIC	BIC
Depth-4 disjoint	297	-30,872.48	62,338.96	64,325.23
Depth-4 overlap	263	-30,860.22	62,246.43	64,005.32
Depth-2 overlap, escalated	293	-30,891.72	62,369.44	64,328.96
Depth-2 disjoint	325	-30,792.76	62,235.52	64,409.05
Panel B. Density-matched placement null

Appendix

Table 27: Higher-order G-DINA mapping sensitivities.

Item set	Item pairs	Pearson $r$	Spearman $\rho$	Null-calibrated $z$
All 40 items	316	.035	.031	1.93
Exclude M042169	262	.041	.029	2.63
Exclude M042198	262	.079	.074	2.46
Exclude both bundles	217	.093	.071	2.99

Appendix

Table 28: Residual-geometry associations.

Representation	Target	Mean $R^{2}_{\mathrm{pred}}$ (SD)	Range	Positive repeats	Leave-one-group-out $R^{2}_{\mathrm{pred}}$
Contextual scores	2PL $b$	.025 (.032)	-.074 to .058	17/20	.030
Raw item embedding	2PL $b$	-.094 (.039)	-.195 to -.021	0/20	-.076
Contextual scores	2PL $\log(a)$	-.094 (.054)	-.255 to -.026	0/20	-.071
Raw item embedding	2PL $\log(a)$	-.141 (.061)	-.261 to -.059	0/20	-.170

Appendix

Table 29: Embedding-to-parameter prediction.

Explore similar work

Oct 5, 2026

COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal

Language models can score psychiatric questionnaires from speech, but agreement with self-report may reflect the questionnaire rather than the person. We introduce psychometric representational similarity analysis, a framework for comparing the structure of speech-derived scores, self-report, item wording and theory, and implement it alongside person-level construct scoring in COMPASS 2.0. We show how similarly worded items induce covariance without psychological signal. In pre-registered discovery and confirmation analyses of clinical interviews from 275 participants, language-derived symptom geometry resembled wording more than self-report, with no structure beyond wording detected by the registered tests. Geometric agreement with self-report survived assigning participants someone else's answers, whereas person-paired scores captured distress more than specific symptoms. Complementary analyses examined counselling quality and wording structure across 34 instruments and the Research Domain Criteria (RDoC) framework. These findings distinguish agreement about psychological structure from evidence that language-derived assessments track individual people.

Baihan Lin

Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA +2

Representational Similarity Analysis Psychometric Validation Clinical Language Model Evaluation

Ask PDF

Jul 13, 2026

Evaluating Nonuniform Dependability Across Response Conditions: A Conditional Generalizability Framework Illustrated in Automated Essay Scoring

Aggregate reliability estimates can obscure heterogeneity in measurement-design burden across response conditions, so a single G- or D-study may mischaracterize a design's adequacy for particular strata. This study introduces a conditional generalizability framework with three components. First, automated scoring configurations -- the encoder architectures and scoring-head families admissible within a fixed pipeline -- are treated as a universe of admissible measurement conditions rather than incidental modeling choices. Second, analytical D-study projections are compared with empirical configuration sweeps over a finite scoring pool, yielding two estimands of design adequacy whose agreement or divergence diagnoses the realized configuration universe. Third, evidence is conditioned on entropy-defined response strata, treating entropy as an operational stratification variable, not a construct claim about writing quality. Whereas recent generalizability-theory extensions address AI-generated item variants on the response side, this framework addresses the analogous scoring-side problem: AI-mediated scoring configurations. Demonstrated with automated essay scoring of timed L2 writing, the realized design was dependable in aggregate (Phi approx 0.76). Re-estimated within entropy strata, dependability stayed high but declined modestly and robustly (Phi = 0.88, 0.87, 0.84) -- a gradient implying different decision-study requirements, the highest-entropy stratum requiring the most crossed conditions. The framework offers a portable workflow for evaluating nonuniform dependability.

Yi Gui

Measurement Incorporated

Automated Essay Scoring Psychometric Validation

Ask PDF

Mar 26, 2026

Measuring What Matters -- or What's Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors

Automated systems have been widely adopted across the educational testing industry for open-response assessment and essay scoring. These systems commonly achieve performance levels comparable to or superior than trained human raters, but have frequently been demonstrated to be vulnerable to the influence of construct-irrelevant factors (i.e., features of responses that are unrelated to the construct assessed) and adversarial conditions. Given the rising usage of large language models in automated scoring systems, there is a renewed focus on ``hallucinations'' and the robustness of these LLM-based automated scoring approaches to construct-irrelevant factors. This study investigates the effects of construct-irrelevant factors on a dual-architecture LLM-based scoring system designed to score short essay-like open-response items in a situational judgment test. It was found that the scoring system was generally robust to padding responses with meaningless text, spelling errors, and writing sophistication. Duplicating large passages of text resulted in lower scores predicted by the system, on average, contradicting results from previous studies of non-LLM-based scoring systems, while off-topic responses were heavily penalized by the scoring system. These results provide encouraging support for the robustness of future LLM-based scoring systems when designed with construct relevance in mind.

Cole Walsh, Rodica Ivan

Acuity Insights Inc., Toronto, ON, Canada

Language Model Robustness Educational Assessment Automated Essay Scoring Spurious Correlation Robustness

Ask PDF