ArchitectureIQ: On the Measure of Training Intuition

Zirui Ren; Shaoyang Guo; Chencheng Tang; Jinxin Wang; Chengyu Xiong; Shanbin Yu; Peihang Li; Yidi Wu

cs.AISep 30, 2026

ArchitectureIQ: On the Measure of Training Intuition

Authors: Zirui Ren, Shaoyang Guo, Chencheng Tang, Jinxin Wang, Chengyu Xiong, Shanbin Yu, Peihang Li, Yidi Wu, +4 more

Organizations: MetaCircle (元环智能) · Tsinghua University · Peking University · University of California, Berkeley · Fudan University · University of Science and Technology of China · Shanghai Qizhi Institute

Abstract

Top researchers have good intuition, but do language models have as good intuition about model training as top AI researchers? To measure model intuition of LLMs and humans, we introduce the ArchitectureIQ benchmark. Each question presents a synthetic dataset and several training recipes, and the test-taker is asked to predict the recipe yielding the best test metric. Overall, we find that LLMs' model intuition is good but has four limitations: (1) The intuition is imperfect, or even sub-human in some cases. Frontier models achieve around 76% accuracy (random choice 33%) vs best human researcher (66.0%), yet remain far from perfect. For architecture-only questions, best human achieves 65% while GPT-6 Astra only has 38%. (2) The intuition is empirical, not structured, supported by the fact that more CoT compute does not lead to substantial improvement. Unlike math, we still lack a "Science of AI" language that enables structured reasoning on AI. (3) The intuition is not maximally condensed, and can be further compressed into a knoledge base. Our constructed knowledge base with only 20 items yields large gains for weak models: GPT-4o equipped with the accumulated knowledge almost matches the performance of Claude Opus 5. (4) The intuition is insensitive to dataset properties, but the best model should in general depend on data properties. This suggests that data is the real "dark matter" in AI -- LLMs (so do human researchers) understand too little about data, even less than model architectures.

Figures & tables

Figure 1: Accuracy on the 500-question, three-choice ArchitectureIQ benchmark (chance

33.3\%

). Printed numbers are overall accuracy; the light cap marks optimizer-only accuracy and the horizontal rule architecture-only accuracy. The best human ( † ) is measured on a fixed 50-question subset. Complete per-type results are given in Table 4 (Appendix D ).

Figure 2: ArchitectureIQ overview. A synthetic dataset and several candidate training recipes are rendered as executable programs. Running those programs across random seeds establishes the ground-truth winner. The evaluated model receives the dataset and recipe descriptions, and predicts which candidate will achieve the best test metric.

Model	Zero-shot	10-shot
Claude Opus 5	70.0%	70.0%
GPT-5.6 Luna	64.0%	60.0%
DeepSeek V4 Flash	62.0%	65.0%
DeepSeek V4 Pro	62.0%	60.0%

Table 1: Comparison of original zero-shot and 10-shot performance

Figure 5: ArchitectureIQ accuracy of rule-based and supervised structured predictors. The analytical decision policy in Figure 8 is evaluated separately.

Figure 6: Six entries from the final knowledge-base snapshot.

Figure 7: Accuracy with no injected propositions (KB0) and with successive knowledge-base snapshots (KB1–KB8), injecting at most the 20 highest-scoring propositions from each snapshot. Evaluation is carried out on a 50-question subset of the full benchmark.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Questions	Input / output	Generating structure	Train / test	Metric
Univariate regression	100	Scalar / scalar	Sampled symbolic function	256 / 256	MSE
Multivariate regression	100	$d$ -vector / scalar	Multivariate symbolic function	256 / 256	MSE
Bigram language modeling	100	Token sequence / next tokens	First-order Markov chain	800 / 200	CE
General tabular classification	100	$d$ -vector / binary label	Additive, interaction, or piecewise rule	1,024 / 2,048	CE
XOR classification	50	$d$ -vector / binary label	Sign interaction of two coordinates	1,024 / 2,048	CE
Spiral classification	50	Two-dimensional point / binary label	Two interleaved spiral arms	1,024 / 2,048	CE

Appendix

Table 2: Dataset families in the frozen v1.5 release. Split sizes count examples, except for bigram language modeling, where they count sequences. MSE denotes mean squared error and CE denotes cross-entropy.

Question type	Varying components	Additional fixed components	Count
Architecture-only	Model configuration	Complete optimizer configuration	170
Optimizer-only	Optimizer configuration	Complete model configuration	168
Mixed	Model and optimizer configurations	None beyond the shared controls	162

Appendix

Table 3: Question types in the frozen release. “Configuration” includes both the component type and its hyperparameters. Dataset, loss, and training budget are fixed within every question.

Figure 8 : Analytical decision tree summarizing an update-regime policy. The underlying development-stage policy reaches 78.2% dataset-grouped out-of-fold top-1 accuracy (391/500). The simplified flowchart was not independently evaluated as a frozen predictor.

Model	Overall	Architecture	Optimizer	Mixed
GPT-5.6 Sol	76.4	58	87	85
Claude Opus 5	76.0	61	86	81
Claude Fable 5	72.2	48	87	83
DeepSeek V4 Flash	70.6	48	85	80
DeepSeek V4 Pro	68.4	44	83	79
GPT-5.6 Luna	67.8	42	85	77

Appendix

Table 4 : Complete 500-question benchmark results (%). Shading runs from white at chance to dark at

90\%

; darker is better.

Metric	Claude Opus 5	GPT-5.6 Sol	GPT-5.6 Luna	DeepSeek V4 Flash

Pairs	52	52	52	52
Side A accuracy	0.538	0.519	0.635	0.615
Side B accuracy	0.423	0.404	0.365	0.327
Combined accuracy	0.481	0.462	0.500	0.471
Answer-change rate	0.231	0.058	0.096	0.059

Appendix

Table 5 : Performance and answer consistency on paired, label-flipped questions

Model	Acc.	$J$	Arch.	Opt.	Mixed
GPT-6 Astra (max)	52.0	0.280	60.0	50.0	50.0
Claude Fable 5	48.0	0.220	50.0	50.0	46.9
Claude Opus 5	48.0	0.220	50.0	50.0	46.9
DeepSeek V4 Flash	48.0	0.220	50.0	50.0	46.9
DeepSeek V4 Pro	48.0	0.220	50.0	50.0	46.9
Gemini 3.1 Pro (high)	48.0	0.220	60.0	37.5	46.9

Appendix

Table 6 : AIQ-Hard50 results (%).

J=(3a-1)/2

is the judgment score (chance

=0

). Arch./Opt./Mixed are per-type accuracies (10, 8 and 32 questions respectively). † 36 of 50 answers valid (repeated reasoning-budget exhaustion at max effort); ‡ 47 of 50 valid.

Figure 9 : AIQ-Hard50 accuracy. Dashed line: decision tree trained on the main benchmark (

48.0\%

). Chance

33.3\%

; data-blind bound

50\%

Model	Same pick (%)	Both right (%)	Exactly one (%)	Neither (%)
GPT-6 Astra (max)	84.0	12.0	80.0	8.0
Claude Fable 5	96.0	0.0	96.0	4.0
Claude Opus 5	100.0	0.0	96.0	4.0
DeepSeek V4 Flash	88.0	4.0	88.0	8.0
DeepSeek V4 Pro	84.0	8.0	80.0	12.0
Gemini 3.1 Pro (high)	76.0	12.0	72.0	16.0

Appendix

Table 7 : Per-pair outcomes on AIQ-Hard50 (25 flip pairs). Same pick : the model chose the same candidate on both sides despite the dataset change. Both / One / Neither : sides answered correctly. † 15 and ‡ 23 pairs with both sides answered (see Table 6 ).

Model	Low	Medium	High	Max
GPT-6 Astra	44.0	44.9 ‡	48.0	52.0
Claude Opus 5	46.0	48.0	48.0	48.0
GPT-5.6 Luna	42.0	44.0	40.0	48.0
GPT-5.6 Sol †	48.0	32.0	44.0	46.0

Appendix

Table 8 : Test-time scaling on AIQ-Hard50: accuracy (%) by reasoning-effort tier. † Sol does not support max; its top tier is xhigh. ‡ Astra medium has 49 of 50 valid answers (one persistent transport failure).

Figure 10 : Example AIQ-Hard50 questions. Consecutive panels are the two sides of one pair: (a,b), (c,d), (e,f), (g,h), (i,j). Left: dataset and candidates (differing fields in bold). Right: executed ground truth over ten seeds (median bold; winner bold in the legend).

Figure 10 : Example AIQ-Hard50 questions (continued).

Panel of Figure 10 — Figure 10 : Example AIQ-Hard50 questions (continued).

Figure 10 : Example AIQ-Hard50 questions (continued).

Explore similar work

ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Models
May 20, 2026Qirui Shen, Wenda Wang, Jiachen Lu +5Stable Spatial Understanding Cognitive Science
School of Architecture, Tsinghua University
SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models
Jun 28, 2026Tiziano Santilli, Francesco Daghero, Mayhar Tourchi MoghaddamLarge Language Model Benchmarks Multiple-Choice Questions
University of Southern Denmark, Odense 5300, DK
Large-scale factor analysis shows machine intelligence is only partially interpretable
Sep 29, 2026Faiz Ghifari Haznitrama, Afrizal Hasbi Azizy, Faeyza Rishad ArdiExploratory Factor Analysis Intelligence
School of Computing, KAIST · Independent Researcher

May 20, 2026, cs.CVJ/K move · Enter open · S save

ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Models

Qirui Shen, Wenda Wang, Jiachen Lu, Zilong Huang+4

School of Architecture, Tsinghua University

Stable Spatial Understanding Cognitive Science

Open paper AskPDF arXiv

Architectural spatial intelligence, the ability to recognize and infer architectural space, is fundamental to tasks such as robot navigation, embodied interaction, and 3D scene understanding and generation. Although extensive research has evaluated the basic spatial skills of Vision-Language Models (VLMs) such as relative orientation, distance comparison, and object counting, these tasks cover only the most elementary levels of spatial cognition and largely overlook higher-level cognition of architectural space, including layout understanding, circulation patterns, and functional zoning. In this work, we present ArchSIBench, a Benchmark for Architectural Spatial Intelligence based on the perspectives from architecture, cognitive science, and psychology. ArchSIBench covers five core dimensions: perception, reasoning, navigation, transformation, and configuration, comprising 17 fine-grained subtasks. Through careful manual annotation by experts with architectural backgrounds, we construct 3,000 question-answer pairs to enable comprehensive evaluation of architectural spatial intelligence. Based on ArchSIBench, we evaluate various VLMs and find that the architectural spatial intelligence of most models shows significant differences from human baselines; additionally, models exhibit substantial variability across capability dimensions. Some state-of-the-art models can approach the level of human evaluators without architectural training. However, a clear gap remains compared to human evaluators with architectural training, particularly in spatial transformation and configuration reasoning. We believe that ArchSIBench will provide important insights and systematic resources for measuring and advancing the architectural spatial intelligence of VLMs. The dataset and code are available at https://huggingface.co/datasets/ArchSIBench/ArchSIBench.

Abstract

Explore similar work

ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Models

SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

Large-scale factor analysis shows machine intelligence is only partially interpretable