ArchSpace: Recording, Reviewing, and Referencing LLM Architecture Research

Jie Wang; Tong Zhu; JiaHao Kuang; Zhe Xu; Yunkai Chai; Dehong Zheng; Yu Cheng; Tao Ji

cs.CLNov 12, 2026

ArchSpace: Recording, Reviewing, and Referencing LLM Architecture Research

Authors: Jie Wang, Tong Zhu, JiaHao Kuang, Zhe Xu, Yunkai Chai, Dehong Zheng, Yu Cheng, Tao Ji,

Organizations: Shanghai AI Laboratory · East China Normal University · Shanghai Jiao Tong University · Fudan University

Abstract

Model architecture shapes the scaling behavior, training efficiency, and inference cost of LLMs, making it a central focus of both industry and academia. In industry, teams such as those behind DeepSeek, Qwen, and Kimi routinely introduce architectural innovations, yet public reports rarely document which ideas succeeded, which were abandoned, and why. In academia, limited compute and inconsistent baselines often restrict evaluations to small-scale, single-point comparisons, making it difficult to establish whether improvements hold at larger scales or translate into broader impact. ArchSpace addresses both gaps as a public platform for Recording, Reviewing, and Referencing Architecture Researc}: each proposal is recorded with its experiments and reasoning, reviewed by the community, and referenced by later work. To evaluate ideas before costly large-scale training, ArchSpace leverages scaling laws to predict model loss at larger scales from smaller experiments, a practice widely adopted in industry. We provide initial references that make the platform usable: Qwen3.5 and OLMo3 baselines, fitted scaling laws, and a reusable training recipe. Ideas validated through this review can in turn become new baselines, and ArchSpace thereby aims to broaden the range of ideas that shape the next generation of models.

Figures & tables

Figure 1: Academic architecture research receives a disproportionately small citation share, motivating ArchSpace’s goal of broader visibility and recognition.

Panel of Figure 2 — Figure 2: The ArchSpace contribution workflow. Phase I compares the proposed architecture with its baseline across the scaling ladder; the three settings differ in their reuse of model size, token count, learning rate (LR), and batch size (BS). Phase II tests whether gains persist through longer training and evaluates downstream capabilities. Reviewed records support future reference. Here, $\widehat{L}$ denotes predicted loss; $C$ is training FLOPs, $M$ training FLOPs per token, $D$ the training token count, $\eta$ LR, and $B$ BS.

Baseline	$L_{\infty}$	$\alpha_{M}$	$\gamma_{M}$	$\alpha_{D}$	$\gamma_{D}$	$\gamma_{MD}$
Qwen3.5	1.722	$6.77\times 10^{4}$	0.520	$1.90\times 10^{4}$	0.453	0.505
OLMo3	1.488	$3.96\times 10^{5}$	0.576	$5.99\times 10^{4}$	0.475	0.402

Table 1: Coefficients of the base surface

L^{*}

(Equation 3 ), fitted to the lowest measured loss of each model–budget group.

M

is in FLOPs per token and

D

in tokens.

Parameter	Qwen3.5	OLMo3
$\eta^{*}$	$421M^{-0.408}D^{-0.152}$	$269M^{-0.389}D^{-0.154}$
$B^{*}$	$36.8M^{-0.047}D^{+0.485}$	$47.3M^{-0.036}D^{+0.457}$
$h$	$17.9M^{-0.075}D^{-0.242}$	$14.0M^{-0.097}D^{-0.203}$
$(\gamma_{\eta,B},\rho)$	$(0.5,1.16)$	$(0.5,1.15)$

Table 2: Power laws and shape parameters of the LR–BS term (Equations 4 and 5 ). The exponent

\gamma_{\eta,B}

is fixed at

1/2

M

is in FLOPs per token;

D

and

B^{*}

are in tokens.

Figure 3: Coupled and additive base surfaces for each baseline ladder. Circles mark the lowest measured loss per model–budget group; curves show the surfaces

L^{*}(M,D)

fitted to these losses. Table 1 gives the coupled-surface coefficients, Table 20 compares fit errors, and Table 16 lists the measured minima.

Baseline	Best-point MAPE	All-point MAPE	P95
Qwen3.5	0.22%	0.25%	0.80%
OLMo3	0.17%	0.27%	0.69%

Table 3: Loss-prediction errors of the full fixed-slope scaling laws on the swept runs. Mean absolute percentage error (MAPE) is reported at group minima (Best-point) and over all runs (All-point); P95 is the 95th percentile of absolute percentage error over all runs.

Figure 4: LR–BS dependence of loss in the Qwen3.5 baseline ladder. Colors show the measured loss above each group minimum, and contours show the fitted LR–BS term $\Delta L$ at intervals of 0.02. Crosses mark predicted optima and stars the best measured configurations; empty panels are outside the sweep. Table 2 gives the fitted laws.

Figure 5: Measured extrapolation losses compared with the compute-optimal loss

\min_{MD=C}L^{*}(M,D)

. Labels give the relative prediction errors

\widehat{L}/L-1

of the full scaling law at each run’s LR and BS, with the coefficients held fixed (Table 4 ).

Baseline	Model	FLOPs	TPP	$L$	$\widehat{L}$	Error
Qwen3.5	1.79B	$2.00\times 10^{20}$	9.1	2.6290	2.6337	$+0.18\%$
Qwen3.5	2.64B †	$4.43\times 10^{20}$	9.4	2.5418	2.5498	$+0.31\%$
OLMo3	1.91B	$2.98\times 10^{20}$	11.8	2.6131	2.6053	$-0.30\%$
OLMo3	3.36B †	$9.80\times 10^{20}$	12.9	2.4888	2.4766	$-0.49\%$
Mean absolute percentage error						0.32%

Table 4: Extrapolation runs beyond the largest swept budget,

1\times 10^{20}

FLOPs. Predictions use the frozen coefficients and each run’s LR and BS. Relative error is

\widehat{L}/L-1

. Daggers mark model sizes beyond the swept range. Table 18 gives the recipes, and Figure 5 compares the measured losses with the compute-optimal curve.

Figure 6: Model scale and token counts in the baseline ladders, extrapolation runs (Section 4.3 ), and Phase II pretraining (Section 5 ). Solid curves show the fitted compute-optimal paths

D^{*}(M)

; dotted lines indicate approximately 10 and 100 tokens per parameter (TPP).

Study	Data	$\gamma_{ND}$	Check	Coupled	Additive
Skaling [ 27 ]	Farseer grid	0.41	Larger $N$ , $D$ ; MAPE	2.31%	2.46%
Skaling [ 27 ]	SK grid	0.31	Larger $N$ , $D$ ; MAPE	0.70%	5.17%
OpenEuroLLM [ 1 ]	OpenEuroLLM ladder	0.41	Largest model; RMSE	0.0056	0.0174
This report	Qwen3.5	0.51	Group minima; MAPE	0.13%	0.48%
This report	OLMo3	0.40	Group minima; MAPE	0.11%	0.62%

Table 5: Within-study comparisons of the coupled form (Equation 3 ) and the additive Chinchilla form [ 9 ] .

\gamma_{ND}

is each study’s symbol for the outer exponent, written

\gamma_{MD}

in this report; RMSE denotes root mean square error. Prediction errors are evaluated on runs excluded from fitting in prior studies and at the swept group minima in this report.

Figure 7: Baseline scaling laws for optimal LR and BS compared with laws from prior work. Panels (a, b) compare the laws at 20 TPP with OpenEuroLLM [ 1 ] , StepLaw [ 14 ] , and Porian Law [ 23 ] ; panels (c, d) compare the laws with those of DeepSeek LLM [ 5 ] at compute-optimal model sizes and training token counts. Baseline curves span the swept sizes and budgets; curves for prior work use the reported coefficients under each study’s parameter-count convention. Table 6 gives the exponents.

Path	Law	$\eta^{*}$ exponent	$B^{*}$ exponent
Fixed model, varying $D$	Shulgin et al. [25] (proxy)	$-0.250$	$+0.500$
	OpenEuroLLM [ 1 ]	$+0.086$	$+0.460$
	StepLaw [ 14 ]	$+0.307$	$+0.571$
	This report, Qwen3.5	$-0.152$	$+0.485$
	This report, OLMo3	$-0.154$	$+0.457$
20 TPP, varying $N$	OpenEuroLLM [ 1 ]	$-0.331$	$+0.605$

Table 6: Scaling exponents of optimal LR and BS, measured as log–log slopes. The row blocks show exponents for a fixed model with varying data, for models of different sizes at 20 TPP, and for compute-optimal combinations of model size and training token count, respectively. When

N

varies, we evaluate the baseline laws at the swept model sizes and the laws from prior work at 0.1B–2B parameters under each study’s parameter-count convention; when

C

varies, slopes are measured over

1\times 10^{18}

–

1\times 10^{20}

FLOPs. The proxy row is theoretical; the other rows describe fitted laws. Figure 7 plots the laws as

N

C

varies.

Baseline	Model	TPP	$\eta^{*}$	$B^{*}$	$\eta^{*}(\cdot\mid B{=}\mathrm{4M})$	$L$	$\widehat{L}$	Error
Qwen3.5	1.21B	129.8	7.49e-4	3.29M	8.26e-4	2.4878	2.5256	$+1.52\%$
OLMo3	1.00B	156.9	7.5e-4	2.61M	9.28e-4	2.5374	2.5516	$+0.56\%$

Table 7: Predicted optima and losses for Phase II pretraining from the frozen scaling laws.

\eta^{*}

and

B^{*}

are the predicted optimal LR and BS; the conditional LR is evaluated at BS 4M (Equation 12 ).

L

and

\widehat{L}

are the measured and predicted losses for runs using the baseline recipe in Table 23 ; relative error is

\widehat{L}/L-1

$\eta$	$B$	$L$	$\widehat{L}$	Error	Score	Math	MC-STEM	MC-NonSTEM	GenQA
					Base Main
8e-4	2M	2.5004	2.5159	$+0.62\%$	27.3	2.6	26.0	28.8	51.8
1e-3	2M	2.5088	2.5171	$+0.33\%$	26.9	2.6	24.2	29.0	52.0
2e-3	2M	2.4958	2.5260	$+1.21\%$	27.6	2.9	25.6	29.5	52.3
8e-4	4M	2.5120	2.5138	$+0.07\%$	27.1	2.5	26.7	28.6	50.5
1e-3	4M	2.5055	2.5140	$+0.34\%$	27.1	2.6	26.4	28.0	51.6

Table 8: Qwen3.5 1B pretraining sweep on about 150B tokens: measured loss and Base Main scores, with predictions

\widehat{L}

from the frozen scaling law. Error is

\widehat{L}/L-1

. MC denotes multiple choice; GenQA denotes generative question answering.

Figure 8: Qwen3.5 long-context mid-training sweep: RULER (a) and HELMET (b) scores at each evaluation length, and the Base Main score against the peak LR (c). Base Main input limits are 8K for the pretrained checkpoint and 64K after mid-training. Tables 25 and 26 give the full results.

Checkpoint	RULER $\uparrow$	HELMET $\uparrow$
Qwen3.5
Pretrained checkpoint	17.5	23.3
Mid-trained at the final pretraining LR (3e-4)	51.9	37.8
Mid-trained at the selected LR (1e-3)	57.0	39.1
Highest HELMET (1.5e-3)	56.0	40.0
OLMo3

Table 9: 64K RULER and HELMET scores before and after long-context mid-training at BS 4M. The Qwen3.5 rows compare the selected LR (1e-3), the final pretraining LR (3e-4), and the LR with the highest HELMET score (1.5e-3). OLMo3 reuses the baseline recipe without tuning. Tables 25 and 28 give the full sweep and the OLMo3 scores after each stage, respectively.

	Qwen3.5		OLMo3
Task	Pretrained	Mid-trained	Pretrained	Mid-trained
Base Main score	27.9	46.3	26.2	40.5
Math	3.2	18.6	2.1	16.4
GSM8K	5.3	32.0	3.1	29.1
GSM Symbolic	1.7	12.3	1.1	9.9
MATH	2.5	11.4	2.1	10.2

Table 10: Base Main task and family scores for both baselines before and after long-context mid-training. Evaluation uses input limits of 8K for the pretrained checkpoints and 64K after mid-training.

		RULER					HELMET
Baseline	Checkpoint	4K	8K	16K	32K	64K	8K	16K	32K	64K
Qwen3.5	Pretrained	65.1	58.4	34.0	23.8	17.5	35.9	24.1	22.9	23.3
Qwen3.5	Mid-trained	81.2	74.4	71.2	65.3	57.0	46.3	42.8	40.9	39.1
OLMo3	Pretrained	69.4	51.9	7.9	4.1	2.7	34.6	8.1	9.1	7.4
OLMo3	Mid-trained	75.3	63.2	52.4	43.3	36.1	42.1	37.1	34.7	29.6

Table 11: Base Long-Context suite results [ 19 ] : RULER and HELMET before and after long-context mid-training, at each evaluation length. Tables 26 and 27 give the full sweep and the 64K category scores, respectively.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Baseline	Model	Layers	$d$	$d_{\mathrm{ff}}$	$N$	$N_{\mathrm{total}}$	$M$
Qwen3.5	93M	8	512	2048	92,907,872	144,288,096	676,930,560
	202M	12	768	2560	202,411,608	279,481,944	1,483,310,592
	407M	16	1024	3584	407,475,072	510,235,520	2,922,792,960
	736M	20	1280	4608	735,791,320	864,241,880	5,161,536,000
	1.21B	24	1536	5632	1,212,956,256	1,367,096,928	8,353,115,136
	1.79B	28	1792	6144	1,794,704,408	1,974,535,192	12,231,937,536

Appendix

Table 12: Model shapes, parameter counts, and per-token training FLOPs for the Qwen3.5 and OLMo3 baselines.

d

and

d_{\mathrm{ff}}

are model and feed-forward widths;

N

excludes the input embedding and

N_{\mathrm{total}}

includes it.

M

is training FLOPs per token and includes the cost of attention at an 8K-token context length. Daggers mark models used only in the extrapolation runs of Section 4.3 .

Parameters	Optimizer	Weight decay
Input embedding matrix	AdamW	0
Output head matrix	AdamW	0.1
Normalization weights	AdamW	0
GDN decay and time-step parameters	AdamW	0
GDN depthwise convolution weights	AdamW	0.1
All other 2D matrix parameters	Muon	0.1

Appendix

Table 13: Optimizer assignments and weight decay for Qwen3.5 and OLMo3. Gated DeltaNet (GDN) rows apply only to Qwen3.5.

Layers	$\eta$ grid
8	2e-3, 5e-3, 7e-3, 1e-2
12	1e-3, 2e-3, 5e-3, 7e-3
16	1e-3, 2e-3, 3e-3, 5e-3
20	8e-4, 1e-3, 2e-3, 3e-3
24 and 28	5e-4, 7e-4, 1e-3, 2e-3

Appendix

Table 14: Peak LR values tested in the Qwen3.5 and OLMo3 baseline ladders, grouped by model depth. Each LR value is tested at BS 128K, 256K, 512K, 1M, and 2M tokens where feasible.

Baseline	FLOPs	Model	$D$	$\eta$	$B$	$L$
Qwen3.5	$1\times 10^{18}$	93M	1.48B	1e-2	512K	3.4743
Qwen3.5	$3\times 10^{18}$	202M	2.02B	2e-3	512K	3.2553
Qwen3.5	$1\times 10^{19}$	407M	3.42B	2e-3	512K	3.0346
Qwen3.5	$3\times 10^{19}$	736M	5.81B	3e-3	512K	2.8768
Qwen3.5	$1\times 10^{20}$	1.21B	12.0B	2e-3	1M	2.7127
OLMo3	$1\times 10^{18}$	176M	746M	5e-3	256K	3.5170

Appendix

Table 15: Best measured recipe of each baseline at each compute budget.

FLOPs	Model	$D$	$\eta$	$B$	$L$	Model	$D$	$\eta$	$B$	$L$
	Qwen3.5					OLMo3
$1\times 10^{18}$	93M	1.48B	1e-2	512K	3.4743	85M	1.57B	5e-3	512K	3.5361
	202M	674M	5e-3	256K	3.4804	176M	746M	5e-3	256K	3.5170
	407M	342M	2e-3	128K	3.5826	346M	388M	5e-3	128K	3.6323
	736M	194M	3e-3	128K	3.7343	614M	224M	3e-3	128K	3.7505
$3\times 10^{18}$	93M	4.43B	1e-2	1M	3.3180	85M	4.72B	5e-3	512K	3.3793

Appendix

Table 16: Best measured recipe in each model–budget group.

Figure 9: LR–BS dependence of loss in the OLMo3 baseline ladder. Colors show the measured loss above each group minimum, and contours show the fitted LR–BS term $\Delta L$ at intervals of 0.02. Crosses mark predicted optima and stars the best measured configurations; empty panels are outside the sweep. Table 2 gives the fitted laws.

Baseline	FLOPs	$M^{*}$ (GFLOPs/token)	$D^{*}$	$\eta^{*}$	$B^{*}$	$L^{*}$
Qwen3.5	$1\times 10^{18}$	1.02	978M	3.82e-3	315K	3.4649
Qwen3.5	$3\times 10^{18}$	1.70	1.76B	2.84e-3	409K	3.2457
Qwen3.5	$1\times 10^{19}$	2.99	3.35B	2.05e-3	545K	3.0370
Qwen3.5	$3\times 10^{19}$	4.98	6.02B	1.52e-3	707K	2.8716
Qwen3.5	$1\times 10^{20}$	8.73	11.5B	1.10e-3	942K	2.7142
Qwen3.5	$1\times 10^{21}$	25.5 †	39.2B	5.87e-4	1.59M	2.4706

Appendix

Table 17: Compute-optimal model sizes and training token counts, together with predicted optima of the adopted fixed-slope scaling laws. Daggers mark model scales beyond the swept range. The predictions at

1\times 10^{21}

FLOPs are extrapolations beyond the swept budgets.

Baseline	Model	$\eta$	$B$
Qwen3.5	1.79B	8e-4	1M
Qwen3.5	2.64B	6e-4	1M
OLMo3	1.91B	8e-4	1M
OLMo3	3.36B	6e-4	2M

Appendix

Table 18: LR and BS used for the extrapolation runs whose measured and predicted losses are compared in Table 4 .

Component	Alternatives	Selected
Base surface	Additive form of Chinchilla [ 9 ] ( $\gamma_{MD}=1$ ); coupled form of Skaling and OpenEuroLLM [ 27 , 1 ]	Coupled form, Equation 3 (Appendix D.1 )
Fitting objective	Squared log-loss residuals; Huber loss on log-loss residuals, as in Skaling and OpenEuroLLM	Huber loss with $\delta=0.001$ when fitting $L^{}$ and $\Delta L$ , Equations 17 and 18 ; best-point MAPE of $L^{}$ is 0.13% (Qwen3.5) and 0.11% (OLMo3), compared with 0.13% and 0.12% for the squared loss
Fitting order	$L^{}$ and $\Delta L$ fitted together; $L^{}$ fitted first and held fixed while $\Delta L$ is fitted	Two-stage fit (Appendix D.2 )
Estimating the optima	Quadratic surfaces fitted to each group’s losses, then power laws fitted to the minima of those surfaces, following OpenEuroLLM; one LR–BS term fitted to losses from all runs	One shared LR–BS term (Appendix D.4 )
Form of $\Delta L$	Proxy shape of Shulgin et al. [ 25 ] with fitted power laws for its optima and amplitude; log-quadratic form based on the proxy’s second-order expansion, with fitted slope and curvature ratio; cosh form with a fitted slope; cosh form with the proxy’s slope $\gamma_{\eta,B}=1/2$	Cosh form with $\gamma_{\eta,B}=1/2$ , Equation 5 (Appendix D.3 )
Optimal BS	$B^{}$ depending on $D$ only, as in StepLaw [ 14 ] ; $B^{}$ depending on $M$ and $D$ , as in OpenEuroLLM	Depends on $M$ and $D$ for all baseline ladders

Appendix

Table 19: Design choices of the scaling law for the baseline ladders.

Baseline	Form	$L_{\infty}$	$\alpha_{M}$	$\gamma_{M}$	$\alpha_{D}$	$\gamma_{D}$	$\gamma_{MD}$	MAPE
Qwen3.5	Coupled	1.722	$6.77\times 10^{4}$	0.520	$1.90\times 10^{4}$	0.453	0.505	0.13%
Qwen3.5	Additive	2.300	$3.61\times 10^{4}$	0.537	$1.28\times 10^{3}$	0.364	1.000	0.48%
OLMo3	Coupled	1.488	$3.96\times 10^{5}$	0.576	$5.99\times 10^{4}$	0.475	0.402	0.11%
OLMo3	Additive	2.348	$8.52\times 10^{4}$	0.580	$1.37\times 10^{3}$	0.365	1.000	0.62%

Appendix

Table 20: Coupled and additive base surfaces fitted to the same group minima. MAPE is evaluated on those minima;

M

is in FLOPs per token and

D

in tokens. Figure 3 compares their IsoFLOP curves with measurements.

	MAPE
	$L^{*}$ , best-point		$\widehat{L}$ , best-point		$\widehat{L}$ , all-point
Baseline	Two-stage	Simultaneous	Two-stage	Simultaneous	Two-stage	Simultaneous
Swept runs
Qwen3.5	0.13%	0.16%	0.21%	0.21%	0.28%	0.27%
OLMo3	0.11%	0.24%	0.19%	0.18%	0.31%	0.29%
Runs excluded from fitting

Appendix

Table 21: Fitting-order ablation with a log-quadratic LR–BS term whose slope and curvature ratio are fitted. Two-stage fitting estimates

L^{*}

from group minima and holds it fixed while fitting

\Delta L

; simultaneous fitting estimates both from the losses of all swept runs.

		Qwen3.5: MAPE				OLMo3: MAPE
Form of $\Delta L$	Parameters	Best- point	All- point	$\eta^{*}$	$B^{*}$	Best- point	All- point	$\eta^{*}$	$B^{*}$
Proxy shape	15	0.29%	0.46%	40.57%	73.85%	0.26%	0.53%	28.90%	75.58%
Log-quadratic	17	0.21%	0.28%	34.49%	23.19%	0.19%	0.31%	22.47%	30.41%
Cosh, fitted slope	17	0.21%	0.25%	50.04%	24.33%	0.17%	0.27%	23.19%	26.59%
Cosh, $\gamma_{\eta,B}=1/2$	16	0.22%	0.25%	35.10%	21.42%	0.17%	0.27%	23.11%	26.68%

Appendix

Table 22: Each alternative LR–BS form is fitted using the same base surface and losses from all swept runs. The Best-point and All-point columns report loss MAPE on group minima and all runs; the

\eta^{*}

and

B^{*}

columns report MAPE against the best measured configuration of each group. Parameter counts include the base surface. Bold marks the adopted form.

Stage	Context	Tokens	$\eta$	$B$	Warmup	Final $\eta$
Pretraining	8K	157.4B	3e-3	4M	2,000 updates	3e-4
Long-context mid-training	64K	10.7B	1e-3	4M	200 updates	0

Appendix

Table 23: Phase II baseline recipe, selected for Qwen3.5 and reused for OLMo3 without tuning (Section 5 ).

Metric	3e-3/4M	2e-3/4M	2e-3/2M
Validation loss $L$	2.4878	2.4933	2.4958
ARC-Challenge accuracy $\uparrow$	24.8%	25.5%	27.2%
MMLU humanities accuracy $\uparrow$	25.4%	24.6%	25.9%
MMLU social sciences accuracy $\uparrow$	26.4%	26.6%	30.0%
MMLU STEM accuracy $\uparrow$	25.1%	27.0%	27.8%
Basic-skills coding accuracy $\uparrow$	60.7%	64.3%	63.8%

Appendix

Table 24: Qwen3.5 benchmark results at the end of pretraining for the selected recipe and alternatives from Table 8 . MMLU uses length-normalized accuracy; bits per byte is measured on the reference answer. These evaluations are separate from the Base Main suite.

Figure 10: Loss measured before decay (a) and reduction during decay (b) in the Qwen3.5 pretraining sweep. Subtracting the reduction from the loss before decay gives the final loss in Table 8 , showing how recipe rankings change during decay.

		Base Main					64K
$\eta$	$B$	Score	Math	MC-STEM	MC-NonSTEM	GenQA	RULER	HELMET
Pretrained checkpoint		27.9	3.2	27.0	28.8	52.6	17.5	23.3
3e-4	4M	44.5	15.5	49.8	56.4	56.2	51.9	37.8
5e-4	4M	45.5	17.3	50.6	57.9	56.3	52.6	38.4
1e-3	4M	46.3	18.6	51.3	59.3	56.0	57.0	39.1
1.5e-3	4M	46.3	19.8	50.4	59.2	55.8	56.0	40.0

Appendix

Table 25: Qwen3.5 long-context mid-training sweep: Base Main scores and 64K RULER and HELMET scores for every run. Long-context scores for the pretrained checkpoint are reference measurements. Base Main input limits are 8K for the pretrained checkpoint and 64K for the mid-trained checkpoints. Figure 8 plots the Base Main and long-context scores, and Table 26 gives long-context results at each evaluation length.

Qwen3.5
		RULER					HELMET
$\eta$	$B$	4K	8K	16K	32K	64K	8K	16K	32K	64K
Pretrained checkpoint		65.1	58.4	34.0	23.8	17.5	35.9	24.1	22.9	23.3
3e-4	4M	77.2	71.9	69.1	61.3	51.9	45.5	42.5	39.9	37.8
5e-4	4M	78.8	74.3	70.1	61.6	52.6	45.7	41.7	39.1	38.4
1e-3	4M	81.2	74.4	71.2	65.3	57.0	46.3	42.8	40.9	39.1

Appendix

Table 26: Long-context scores for the Qwen3.5 mid-training sweep and the OLMo3 baseline checkpoints; scores for the pretrained checkpoints are reference measurements. Figure 8 a,b plots the sweep results; Table 27 breaks down the 64K scores by category.

		RULER				HELMET
$\eta$	$B$	Retr.	Multi-hop	Aggr.	QA	Recall	RAG	Rerank	ICL	LongQA
Pretrained checkpoint		19.8	9.2	13.7	16.5	6.9	24.5	0.1	69.0	15.8
3e-4	4M	67.0	16.4	21.4	40.0	38.1	44.5	0.2	76.4	30.0
5e-4	4M	68.7	17.0	21.6	37.0	40.9	44.8	0.4	73.8	32.1
1e-3	4M	73.5	18.4	28.4	39.0	44.1	44.5	2.2	75.8	29.0
1.5e-3	4M	71.5	18.2	27.3	42.0	41.2	45.7	2.4	78.0	32.9

Appendix

Table 27: Qwen3.5 category scores at 64K; scores for the pretrained checkpoint are reference measurements. RULER categories are needle-in-a-haystack retrieval (Retr.), multi-hop tracing (Multi-hop), aggregation (Aggr.), and question answering (QA). HELMET categories are synthetic recall (Recall), retrieval-augmented generation (RAG), passage re-ranking (Rerank), many-shot in-context learning (ICL), and long-document QA (LongQA).

	Base Main					64K
Checkpoint	Score	Math	MC-STEM	MC-NonSTEM	GenQA	RULER	HELMET
Pretrained checkpoint	26.2	2.1	25.2	27.7	49.8	2.7	7.4
Mid-trained (1e-3/4M)	40.5	16.4	43.8	49.2	52.4	36.1	29.6

Appendix

Table 28: OLMo3 results with the Qwen3.5 recipe reused without tuning. Base Main input limits change from 8K after pretraining to 64K after mid-training. Long-context scores for the pretrained checkpoint are reference measurements. Tables 10 and 26 give task-level Base Main results and long-context scores at each evaluation length, respectively.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,502 papers

Sep 30, 2026

ArchitectureIQ: On the Measure of Training Intuition

Top researchers have good intuition, but do language models have as good intuition about model training as top AI researchers? To measure model intuition of LLMs and humans, we introduce the ArchitectureIQ benchmark. Each question presents a synthetic dataset and several training recipes, and the test-taker is asked to predict the recipe yielding the best test metric. Overall, we find that LLMs' model intuition is good but has four limitations: (1) The intuition is imperfect, or even sub-human in some cases. Frontier models achieve around 76% accuracy (random choice 33%) vs best human researcher (66.0%), yet remain far from perfect. For architecture-only questions, best human achieves 65% while GPT-6 Astra only has 38%. (2) The intuition is empirical, not structured, supported by the fact that more CoT compute does not lead to substantial improvement. Unlike math, we still lack a "Science of AI" language that enables structured reasoning on AI. (3) The intuition is not maximally condensed, and can be further compressed into a knoledge base. Our constructed knowledge base with only 20 items yields large gains for weak models: GPT-4o equipped with the accumulated knowledge almost matches the performance of Claude Opus 5. (4) The intuition is insensitive to dataset properties, but the best model should in general depend on data properties. This suggests that data is the real "dark matter" in AI -- LLMs (so do human researchers) understand too little about data, even less than model architectures.

Zirui Ren, Shaoyang Guo, Chencheng Tang +9

MetaCircle (元环智能) · Tsinghua University · Peking University +4

Model Selection LLM Evaluation

Ask PDF

Aug 20, 2026

Write Once, Run Everywhere: The Axon DSL for Shape-Safe and Framework-Agnostic LLM Architectures

The entire ecosystem of open-source language models effectively relies on a single platform. What if this platform was forced to shut down tomorrow? Implementing and maintaining efficient model definitions and translating them between different training and inference regimes is a resource-heavy task that severely limits model efficiency and portability, hindering both scaling and deployment. Here, we present Axon, a strongly typed domain-specific language with Haskell-like syntax, that enables a write-once, run everywhere paradigm for LLM architectures. By basing collaboration on a language specification rather than a specific framework's vision, Axon fosters open cooperation and empowers researchers to implement highly specialized architectures without giving up optimization infrastructure or accepting deployment lock-in. Axon allows for concise, auditable specifications that can be automatically compiled to standalone implementations for leading frameworks: PyTorch, PyTorch with Triton, JAX, MLX and vLLM. In 467 inference benchmarking experiments on models ranging from 135M to 32B parameters, we demonstrate median speedups of 7% on PyTorch, 12% on PyTorch with Triton, 91% on JAX, and 107% on MLX, compared to the reference implementations from Transformers. When deployed as native vLLM architectures with PagedAttention and KV-cache, Axon models achieve a 58% median speedup over Transformers implementations.

Jacob Nielsen, Danial Namazifard, Lukas Galke Poech +1

University of Southern Denmark Campusvej 55, 5230 Odense, Denmark

Efficient Language Model Inference LLM Inference Acceleration

Ask PDF

Apr 7, 2026

CAKE: Cloud Architecture Knowledge Evaluation of Large Language Models

In today's software architecture, large language models (LLMs) serve as software architecture co-pilots. However, no benchmark currently exists to evaluate large language models' actual understanding of cloud-native software architecture. For this reason we present a benchmark called CAKE, which consists of 188 expert-validated questions covering four cognitive levels of Bloom's revised taxonomy -- recall, analyze, design, and implement -- and five cloud-native topics. Evaluation is conducted on 22 model configurations (0.5B--70B parameters) across four LLM families, using three-run majority voting for multiple-choice questions (MCQs) and LLM-as-a-judge scoring for free-responses (FR). Based on this evaluation, four notable findings were identified. First, MCQ accuracy plateaus above 3B parameters, with the best model reaching 99.2%. Second, free-response scores scale steadily across all cognitive levels. Third, the two formats capture different facets of knowledge, as the MCQ accuracy approaches a ceiling while free-responses continue to differentiate models. Finally, reasoning augmentation (+think) improves free-response quality, while tool augmentation (+tool) degrades performance for small models. These results suggest that the evaluation format fundamentally shapes how we measure architectural knowledge in LLMs.

Tim Lukas Adam, Phongsakon Mark Konrad, Riccardo Terrenzi +4

Centre for Industrial Software, University of Southern Denmark, Alsion 2, Sønderborg, 6400, Denmark

Software Architecture Design LLM Evaluation Software Engineering Cloud Computing Language Model Generation Evaluation

Ask PDF

Abstract

LLM Evaluation

Explore similar work

ArchitectureIQ: On the Measure of Training Intuition

Write Once, Run Everywhere: The Axon DSL for Shape-Safe and Framework-Agnostic LLM Architectures

CAKE: Cloud Architecture Knowledge Evaluation of Large Language Models