Large-scale factor analysis shows machine intelligence is only partially interpretable

Faiz Ghifari Haznitrama; Afrizal Hasbi Azizy; Faeyza Rishad Ardi

cs.CLSep 29, 2026

Large-scale factor analysis shows machine intelligence is only partially interpretable

Authors: Faiz Ghifari Haznitrama, Afrizal Hasbi Azizy, Faeyza Rishad Ardi

Organizations: School of Computing, KAIST · Independent Researcher

Abstract

A common assumption in language model development is that cognitive abilities are organized around a general, domain-free intelligence factor, like fluid intelligence in humans. This assumption is rarely tested directly, and prior attempts have done so only at a much smaller scale. We take a latent variable approach to intelligence in language models, similar to how psychometricians study psychological constructs. Performance in every specific problem set is influenced by a domain-specific and a domain-agnostic latent factor. Using factor analysis as a dimension-reduction technique, we analyzed 13,251 published evaluation scores covering 1,618 language models across 456 different text-only benchmarks. Due to the super-sparse nature of the dataset, we triangulate our analysis across different data densifiers and imputation methods. A robust pattern across different modes of bias is that 1. A general intelligence factor accounts for 70.8% of variance in model performance at our most generous estimate, and far less than that in most of our solutions, 2. Content-similar benchmarks do not necessarily cluster together, and 3. The $g$ factor is not dominated by any common theme, and there is a lack of evidence that it is well-proxied by standard "intelligence" benchmarks. Our findings go against current endeavors of defining, identifying, and targeting general intelligence as a tangible construct in language model development. This leaves the strategy of targeting a single conceptual ability without support, since the first-order abilities it would have to reach are often partially idiosyncratic and not identifiable in practice.

Figures & tables

Dataset	Imputer	$k$	AVE	$\omega_{h}$	$\phi_{\text{avg}}$	$R^{2}$
C Std.	Mean fill	14	5.5%	0.708	0.142	0.224
C Std.	missForest	4	22.1%	0.695	0.398	0.399
S Std.	SoftImpute (corr.)	5	9.3%	0.676	0.306	0.378
C Std.	Zero fill	14	5.4%	0.621	0.093	0.286
C Aggr.	missForest	4	19.3%	0.521	0.112	0.241
C Std.	k-NN	7	10.8%	0.516	0.209	0.288

Table 3. : Point summaries of factor analysis results. AVE = average variance explained per factor,

k

= number of factors extracted,

\phi_{\text{avg}}

= average inter-factor correlation.

Label	Median A	Significant
code	+0.264	6/18
math	+0.031	0/18
reasoning	-0.008	0/18

Table 4. : Cohesion of subject labels. Full cohesion results for every label can be seen in Appendix M .

Figure 1. : UMAP plot of benchmark distances from S, SoftImpute. Domain-similar benchmarks are not guaranteed to cluster together.

No	Benchmark	$\rho_{\epsilon}$	$\rho$	95% CI	Best	Worst	$N$
1	bhasa	-0.346	0.141	[-0.005, 0.287]	0.024	0.318	5
2	mtrag	-0.341	0.147	[-0.051, 0.346]	0.021	0.394	5
3	creativityprism	-0.339	0.156	[-0.019, 0.332]	0.026	0.367	5
4	eqbench	-0.332	0.156	[-0.060, 0.372]	0.017	0.451	5
5	mceval	-0.331	0.156	[0.024, 0.288]	0.051	0.333	5
6	pwc_svamp	-0.329	0.169	[-0.033, 0.371]	0.058	0.298	4

Table 5. : Benchmarks and their average normalized rank order (

\rho

) of their

g

factor loadings. Sorted by frequency-residualized rank (

\rho_{\epsilon}

\rho

ranges from 0 to 1, where 0 = ranked first, 1 = ranked last.

N

= number of EFA estimations with that benchmark. CI and Best/Worst refer to

\rho

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Correlation evidence	Decision	Rows removed
LiveCodeBench release windows v1–v6 (Kaggle)	mean pairwise $r=0.995$ , worst pair $0.987$ , over 45 shared models	Keep the aggregate, drop 6 per-version identifiers	270
TwitterAAE dialect splits (African-American and White English)	$r=0.993$ – $0.999$ with each other and the parent, over 32 shared models	Keep the parent, drop both dialect splits	64
GPQA variants (few/zero-shot $\times$ diamond/main, Kaggle)	mean $r=0.944$ , worst pair $0.915$ , over 46–47 shared models, with the better-populated canonical GPQA and GPQA Diamond already present	Drop all 4 Kaggle variants	185
MultiLoKo per-language splits (31 languages, Kaggle)	mean pairwise $r=0.82$ , near-duplicate for well-resourced pairs (Simplified/Traditional Mandarin $0.989$ , Italian/Swedish $0.983$ ) and noisy for low-resource pairs on small overlap	Keep the paper-sourced MultiLoKo across-language aggregate, drop all 31 per-language identifiers	1,523
A Kaggle re-import of MMLU	Cross-source re-import (44 rows) of the canonical MMLU column (468 rows)	Drop the re-import	44
A Kaggle re-import of SciCode	Not a third metric. Per-model value matching shows it splices main-problem scores for 30 of 46 models and sub-problem scores for the other 13, which is a scraping artifact	Drop as a data-integrity fix. The 4 explicit split variants ( $r=0.71$ – $0.96$ ) are kept , as their correlations are not uniform enough to treat as duplicates	46

Appendix

Table A1. : Benchmark families audited for score redundancy, with the correlation evidence and decision for each.

Method	Description	Package	Configuration
SoftImpute ( Mazumder et al., 2010 )	Nuclear-norm-penalized low-rank completion by iterative soft-thresholded SVD, assuming a low-rank signal plus noise. Primary cell-level method.	softImpute	sweeps rank: 1…10 (capped at $\min(n,p)-1$ ), and at each rank a 30-point geometric $\lambda$ grid from $\lambda_{0}$ down to $\lambda_{0}/100$ , ALS with warm starts
k-NN	Each missing cell filled from the $k$ most similar models, an assumption-light baseline with no low-rank, linearity, or normality assumption.	VIM	sweeps $k$ : 1…10 (capped below $n$ ), Gower distance over benchmarks, weighted-mean aggregation
missForest ( Stekhoven & Bühlmann, 2012 )	Iterative random-forest imputation, nonparametric, able to capture nonlinear dependence the low-rank methods cannot represent.	missForest	sweeps number of trees: {50, 100, 200, 400}, at most 10 iterations

Appendix

Table A2. : Estimators of the missing dataset entries.

Estimator	Description	Package	Configuration
One-sided matrix completion ( Cao et al., 2023 )	Recovers the right-singular vectors of a reduced matrix of a large, super-sparse dataset. Originally demonstrated to work with simply 2 observations per row.	Custom Julia code	Sweeps rank 1…10, selecting the rank with the best $R^{2}$ .
SoftImpute ( Mazumder et al., 2010 )	Applies SoftImpute’s low-rank completion to the observed pairwise correlation matrix rather than the data matrix, whose missing entries are exactly the benchmark pairs never co-observed.	softImpute	sweeps rank 1…10 with the same nested $\lambda$ grid as the cell-level methods above
USVT ( Chatterjee, 2015 )	Universal singular value thresholding: completes the correlation matrix by hard-thresholding its singular values.	filling	fixed singular-value threshold $\eta=0.01$ , no sweep

Appendix

Table A3. : Estimators of the missing correlation entries.

Densifier	Benchmarks	$\leq 2022$	2023	2024	$\geq 2025$
C	78	87/20%/10	196/14%/6	294/12%/21	93/12%/10
S	124	86/14%/42	196/9%/30	292/9%/26	94/10%/16

Appendix

Table A4. : Size and coverage of each release-year group. Each cell gives the number of models, the share of observed cells, and the number of benchmarks with no observed score in that group.

Figure A1. : $\omega_{h}$ of each release-year group (dots) against the 5th to 95th percentile of 50 random subsets of the same size (bars), per imputer, on the C (left) and S (right) matrices.

Densifier	Imputer	$\leq 2022$	2023	2024	$\geq 2025$
C	SoftImpute	-1.33	-0.29	0.59	0.00
C	missForest	-1.02	-0.40	0.22	1.11
C	k-NN	-1.36	-0.34	0.25	1.22
S	SoftImpute	-1.47	-0.21	0.37	0.67
S	missForest	0.43	-0.32	-0.19	0.86
S	k-NN	-0.52	0.06	-0.27	1.20

Appendix

Table A5. : Mean general-factor score of each release-year group, with mean zero over all models.

Densifier	Imputer	$\rho$	Era $z$	Year gap ( $p$ )	Co-obs.
C	SoftImpute	-0.16	-6.1	0.07 (0.022)	0.39
C	missForest	0.21	-4.2	0.10 (0.054)	0.31
C	k-NN	-0.34	-8.2	0.06 (0.082)	0.43
C	OneSidedMC	0.10	-12.6	0.15 (0.000)	0.54
C	SoftImpute (corr.)	-0.06	-6.1	0.11 (0.018)	0.25
S	SoftImpute	-0.03	-6.9	0.09 (0.004)	0.20

Appendix

Table A6. : Benchmark release year against the pooled solutions.

\rho

is the Spearman correlation between release year and absolute general-factor loading. Era

z

compares within-era loading distance with permuted eras (negative means same-era benchmarks sit closer). The last two columns are standardized rank-regression coefficients of a pair’s loading distance on its release-year gap (with permutation

p

) and on the Jaccard distance between its sets of observed models (co-obs.).

Method	Dataset	$k$	$r$	$N$
Mean fill	C Std.	2	+0.4192	78
Mean fill	C Std.	14	+0.4192	78
k-NN	C Std.	2	+0.4065	78
k-NN	C Std.	7	+0.4065	78
k-NN	S Std.	2	+0.2335	124
k-NN	S Std.	11	+0.2335	124

Appendix

Table A7. : Correlations between benchmark frequency and their

g

loadings.

k

= number of factors extracted.

Method	Dataset	$N$	$r$
Mean fill	C Std.	78	-0.429
k-NN	C Std.	78	-0.429
k-NN	S Std.	124	+0.020
missForest	C Aggr.	102	-0.342
missForest	C Std.	78	-0.106
missForest	S Std.	124	+0.261

Appendix

Table A8. : Correlations between benchmark frequency and their

g

rankings. Mean

r=-0.173

, mean

\lvert r\rvert=0.273

; 13 negative and 7 positive cells.

$N$ cells	$N$ benchmarks	$r$ raw	$r$ adjusted
2	54	-0.023	-0.021
3	26	-0.024	-0.049
4	24	-0.214	-0.232
5	141	+0.061	+0.046
8	10	+0.040	+0.038
10	33	+0.518	+0.516

Appendix

Table A9. : Pooled correlation between mean frequency and mean rank.

Label	Median A	Significant	Median n
conversation	+0.283	3/6	7
likelihood_probe	+0.153	6/10	19
short_qa	+0.098	2/18	15
classification	+0.095	2/18	15
interactive	+0.078	1/10	16
long_reasoning	+0.066	0/18	9

Appendix

Table A10. : Cohesion by task-format label, sorted by descending median A.

Label	Median A	Significant	Median n
monolingual_non_english	+0.291	16/18	19
multilingual	+0.204	6/14	9
crosslingual	+0.137	1/10	12
english	-0.034	0/18	85

Appendix

Table A11. : Cohesion by language-coverage label, sorted by descending median A.

Label	Median A	Significant	Median n
finance	+0.870	10/10	4
professional_writing	+0.632	4/6	5
fact_verification	+0.300	0/4	5
commonsense	+0.287	5/18	11
code	+0.264	6/18	11
translation	+0.249	6/10	16

Appendix

Table A12. : Cohesion by subject-matter label, sorted by descending median A.

Dataset	Imputer	RMSE	$R^{2}$	Configuration
S Std.	SoftImpute	0.6338	0.504	rank=5 (swept 1..10)
C Std.	SoftImpute	0.6575	0.493	rank=9 (swept 1..10)
S Std.	missForest	0.6798	0.471	ntree=400 (swept [50,100,200,400])
C Std.	missForest	0.7321	0.399	ntree=50 (swept [50,100,200,400])
S Std.	SoftImpute (corr.)	0.7637	0.378	rank=6 (swept 1..10)
S Std.	OneSidedMC	0.7554	0.365	r=2 (swept 1..10)

Appendix

Table A13. : Results of all imputation runs, sorted by

R^{2}

Method	Dataset	$k$	$r$	$N$
Mean fill	C Std.	2	+0.0701	78
Mean fill	C Std.	14	+0.1316	78
k-NN	C Std.	2	+0.0268	78
k-NN	C Std.	7	+0.3205	78
k-NN	S Std.	2	+0.1765	124
k-NN	S Std.	11	+0.2069	124

Appendix

Table A14. : Correlations between benchmark observation and their

\Delta\omega_{h}

k

= number of factors extracted.

No	Benchmark	$\rho_{\epsilon}$	$\rho$	95% CI	Best	Worst	$N$	Reference
1	bhasa	-.346	.141	[-.005, .287]	.024	.318	5	Leong et al.,2023
2	mtrag	-.341	.147	[-.051, .346]	.021	.394	5	Katsis et al.,2025
3	creativityprism	-.339	.156	[-.019, .332]	.026	.367	5	Hou et al.,2026
4	eqbench	-.332	.156	[-.060, .372]	.017	.451	5	Paech,2023
5	mceval	-.331	.156	[.024, .288]	.051	.333	5	Chai et al.,2025
6	pwc_svamp	-.329	.169	[-.033, .371]	.058	.298	4	Patel et al.,2021

Appendix

Table A15. : All used 380 benchmarks, sorted by their average normalized rank order (ANR) of their

g

factor loadings, residualized (RANR) against their frequency. The normalized rank order ranges from 0 to 1. 0 = ranked first, 1 = ranked last.

N

cells = number of EFA solutions with that benchmark. CI and Best/Worst refers to ANR. Leading zeros are omitted. Reference is the paper introducing the benchmark; † marks benchmarks without one, cited by the source of their scores.

Figure A2. : C dataset, aggregated across 7 imputations.

Figure A3. : C dataset, imputed with mean fill.

Figure A4. : C dataset, imputed with k-NN.

Figure A5. : C dataset, imputed with missForest.

Figure A6. : C dataset, imputed with OneSidedMC.

Figure A7. : C dataset, imputed with SoftImpute.

Figure A8. : C dataset, imputed with SoftImpute (corr.).

Figure A9. : C dataset, imputed with zero fill.

Figure A10. : S dataset, aggregated across 5 imputations.

Figure A11. : S dataset, imputed with k-NN.

Figure A12. : S dataset, imputed with missForest.

Figure A13. : S dataset, imputed with OneSidedMC.

Figure A14. : S dataset, imputed with SoftImpute.

Figure A15. : S dataset, imputed with SoftImpute (corr.).

Figure A16. : R dataset, imputed with SoftImpute.

Figure A17. : Raw dataset, imputed with SoftImpute.

Explore similar work

May 27, 2026cs.CL

DEPART: DEcomposing PARiTy across Multilingual LLMs

Multilingual Large Language Models (mLLMs) leaderboards report per-language accuracy but rarely explain why disparities emerge, leaving systemic biases unattributed and offering practitioners no actionable levers. We first establish that these gaps are systematic rather than artifacts of sampling noise via distribution-free Friedman and Kruskal--Wallis tests, then introduce a two-step Bayesian hierarchical framework that decomposes multilingual performance variance into interpretable components. First, isolating the variance attributable to language identity, we show that observable language features (script, family, typological distance) explain

R^2_{\text{ling}} = 79\%

of this variance on understanding tasks and

92\%

on reasoning, with a model's internal representational similarity to English emerging as the dominant predictor across both task buckets. Second, decomposing the full (model

\times

benchmark

\times

language) cube, we find that NLU and reasoning have fundamentally divergent variance profiles: model identity dominates understanding (

66.7\%

of variance), whereas the benchmark

\times

model interaction dominates reasoning (

46.3\%

). Together these results recast multilingual evaluation from passive performance mapping into an explainable, diagnostic framework with concrete levers for targeting the root drivers of language disparity.

Manan Uppadhyay, Prashant Kodali, Pranjal Chitale +3

Microsoft Research India · IIT Gandhinagar

Topic pending

Ask PDF

Sep 28, 2026cs.CL

AwarenessBench: Assessing Cognitive Capabilities of Language Models

As language models (LMs) exhibit increasingly consciousness-like behaviors, evaluating their cognitive abilities becomes essential. We introduce AwarenessBench, the first comprehensive benchmark for assessing the cognitive abilities of LMs in four dimensions: metacognition, self-awareness, social awareness, and situational awareness, covering 15 cognitive functions and 14,381 samples. Evaluating 18 state-of-the-art LMs, we find that all consistently surpass random baselines, with more advanced models performing better. We further compare LMs with human performance across three demographic groups, where the best-performing model surpasses human averages overall, but most still fall markedly short in metacognition and self-awareness. Finally, we show that awareness is a distinct capability: progress in language modeling or reasoning does not necessarily translate into improved cognition.

Xiaojian Li, Rongwu Xu, Tianyun Zhang +9

Tsinghua University · Shanghai Qi Zhi Institute · Fangcun AI +5

Cognitive Science Metacognition

Ask PDF

Sep 30, 2026cs.AI

ArchitectureIQ: On the Measure of Training Intuition

Top researchers have good intuition, but do language models have as good intuition about model training as top AI researchers? To measure model intuition of LLMs and humans, we introduce the ArchitectureIQ benchmark. Each question presents a synthetic dataset and several training recipes, and the test-taker is asked to predict the recipe yielding the best test metric. Overall, we find that LLMs' model intuition is good but has four limitations: (1) The intuition is imperfect, or even sub-human in some cases. Frontier models achieve around 76% accuracy (random choice 33%) vs best human researcher (66.0%), yet remain far from perfect. For architecture-only questions, best human achieves 65% while GPT-6 Astra only has 38%. (2) The intuition is empirical, not structured, supported by the fact that more CoT compute does not lead to substantial improvement. Unlike math, we still lack a "Science of AI" language that enables structured reasoning on AI. (3) The intuition is not maximally condensed, and can be further compressed into a knoledge base. Our constructed knowledge base with only 20 items yields large gains for weak models: GPT-4o equipped with the accumulated knowledge almost matches the performance of Claude Opus 5. (4) The intuition is insensitive to dataset properties, but the best model should in general depend on data properties. This suggests that data is the real "dark matter" in AI -- LLMs (so do human researchers) understand too little about data, even less than model architectures.

Zirui Ren, Shaoyang Guo, Chencheng Tang +9

MetaCircle (元环智能) · Tsinghua University · Peking University +4

Intuition Artificial Intelligence Scientists Measures

Ask PDF