The evaluation of large language models (LLMs) relies heavily on human-designed assessments, implicitly assuming that AI and humans employ similar underlying cognitive constructs. Challenging this assumption, we investigate whether the latent factors governing LLM performance carry the same substantive, human-interpretable meaning as the cognitive constructs governing human learners. Using responses from humans and six LLMs across quantitative reasoning and chemistry assessments, we conducted Exploratory Factor Analysis (EFA) separately for both groups. Subject-Matter Experts (SMEs) then blindly evaluated the resulting factor graphs to ascribe pedagogical meaning to the emerged constructs. SMEs successfully interpreted most of the human-derived factors. Conversely, they could not ascribe meaning to any LLM-derived factors in quantitative reasoning and interpreted only half of the LLM factors in chemistry. By combining data-driven EFA with blind expert interpretation, this framework shows that LLMs frequently operate on statistically opaque mechanisms distinct from human reasoning.
Figures & tables
Figure 1: Methodological Flowchart. Human and LLM response data for two assessments are processed to extract the underlying factor structure. The blinded and shuffled factor structure results are passed on to subject matter experts for analysis.
Figure 2: EFA item-loading patterns for the chemistry assessment. Human and LLM factors and loadings are shown in blue and red for visual clarity; however, factor structures were originally extracted independently for human and LLM respondents before being combined, shuffled and blinded into a monochromatic version of this figure in preparation for SME evaluation. ∣loadings∣≥0.5 are labeled; 0.3≤∣loadings∣<0.5 are unlabeled; ∣loadings∣<0.3 not shown.
Figure 3: EFA item-loading patterns for the quantitative reasoning assessment. Factor structures independently derived for each group. Visualization and blinding procedures for SME evaluation follow the identical monochromatic protocol utilized for chemistry assessment.
Factors Retained
EFA Fit Results
Human
LLM
Instrument
Human
LLM
RMSEA
Confidence
RMSEA
Confidence
Chemistry
4
4
0.019
>0.90
0.045
>0.90
Quantitative reasoning
5
5
0.011
>0.90
0.092
>0.90
Table 1: Factor Retention and EFA Fit for Human and LLM Samples