Sometimes less is more: Investigating inductive biases of MLPs and Transformers on noisy k-parity features

Alasdair Paren

cs.LGOct 27, 2026

Sometimes less is more: Investigating inductive biases of MLPs and Transformers on noisy k-parity features

Authors: Alasdair Paren

Organizations: Chalmers University of Technology

Abstract

We empirically investigate how MLPs and Transformers differ when trained on data sets containing one or more features encoded by noisy k-parity. Noisy k-parity gives us hard synthetic data sets in which we can control the predictivity and availability of every feature. Across a range of data distributions, data scales and model sizes, we find large differences in the training dynamics and inductive biases of the two architectures: Transformers are more robust to noisy features but struggle to learn complex, low-availability features, whereas MLPs show the opposite trend and improve steadily with more data. We also find that, under a fixed training budget, increasing the number of unique training examples can slow Transformer optimisation enough to reduce final accuracy, so that a smaller data set performs better. Training-accuracy results show this is an optimisation-speed effect; it shrinks, but does not vanish, when the budget is scaled with the data set size. By giving precise control over feature predictivity and availability, this testbed offers a clear starting point for understanding when and why Transformers and MLPs learn differently, and for designing data and training budgets that exploit those differences.

Figures & tables

Figure 1: Test accuracy vs. k-parity size $d$ . Red : $m=10$ , $N=8000$ , $T=10^{4}$ . Blue : $m=20$ , $N=240000$ , $T=2\times 10^{4}$ . Scaling up all three gives worse results (Section 4.2 ).

Panel of Figure 2 — Figure 2: Test accuracy on a noiseless k-parity feature with a fixed training budget ( $T=10^{4}$ , finite training sets and $N=\infty$ ). Left two panels: MLP. Right two panels: GPT. For each model the first panel shows accuracy against feature size $d$ and the second against training set size $N$ . Corresponding training accuracies are in Appendix E .

	Fixed, $d=18$		Scaled, $d=18$		Scaled, $d=22$
Training set size $N$	train	test	train	test	train	test
2,400	0.993	0.537	0.817	0.494	0.831	0.495
8,000	0.889	0.757	0.867	0.705	0.822	0.506
24,000	0.594	0.549	0.792	0.610	0.904	0.597
80,000	0.506	0.501	1.000	1.000	0.876	0.600
240,000	–	–	1.000	1.000	0.771	0.702

Table 1: GPT train and test accuracy on a noiseless k-parity feature under the fixed budget (

T=10^{4}

) and the scaled budget (

T=1.25N

). Rows give the training set size

N

. Under the fixed budget, large data sets are not fitted even on the training set (undertraining), and under the scaled budget the training accuracy at

d=22

is lower for the largest data set than for

N=24{,}000

despite ten times more iterations. Full tables are in Appendix E .

Panel of Figure 7 — Figure 7: Train (first and third panels) and test (second and fourth panels) accuracy on the tiered data set with a scaled iteration budget ( $T=1.25N$ ). Left two panels: MLP. Right two panels: GPT.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

$N\backslash d$	8	10	12	14	16	18	20	22
2,400	1.000	1.000	1.000	1.000	0.997	0.993	0.992	0.992
8,000	1.000	1.000	1.000	0.999	0.999	0.889	0.859	0.825
24,000	1.000	1.000	0.999	0.999	0.977	0.594	0.593	0.603
80,000	1.000	1.000	0.950	0.900	0.684	0.506	0.507	0.507
$\infty$ (DGP)	1.000	0.752	0.500	0.499	0.500	0.500	0.500	0.500

Appendix

Table 2: GPT train accuracy, fixed budget (

T=10^{4}

$N\backslash d$	8	10	12	14	16	18	20	22
2,400	1.000	1.000	1.000	1.000	0.902	0.537	0.502	0.494
8,000	1.000	1.000	1.000	1.000	1.000	0.757	0.597	0.510
24,000	1.000	1.000	1.000	1.000	0.996	0.549	0.551	0.550
80,000	1.000	1.000	0.975	0.900	0.736	0.501	0.499	0.499
$\infty$ (DGP)	1.000	0.785	0.499	0.500	0.501	0.500	0.500	0.501

Appendix

Table 3: GPT test accuracy, fixed budget (

T=10^{4}

$N\backslash d$	8	10	12	14	16	18	20	22	24	26	28	30
2,400	1.000	0.999	0.999	0.997	0.950	0.817	0.821	0.831	0.831	0.819	0.863	0.811
8,000	1.000	1.000	1.000	0.999	0.999	0.867	0.853	0.822	0.863	0.839	0.807	0.808
24,000	1.000	1.000	1.000	1.000	1.000	0.792	0.801	0.904	0.851	0.846	0.857	0.867
80,000	1.000	1.000	1.000	1.000	1.000	1.000	0.846	0.876	0.793	0.843	0.914	0.853
240,000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	0.771	0.669	0.712	0.747	0.757

Appendix

Table 4: GPT train accuracy, scaled budget (

T=1.25N

$N\backslash d$	8	10	12	14	16	18	20	22	24	26	28	30
2,400	1.000	1.000	0.999	0.999	0.896	0.494	0.511	0.495	0.513	0.479	0.496	0.478
8,000	1.000	1.000	1.000	1.000	1.000	0.705	0.598	0.506	0.484	0.493	0.503	0.505
24,000	1.000	1.000	1.000	1.000	1.000	0.610	0.503	0.597	0.594	0.498	0.501	0.500
80,000	1.000	1.000	1.000	1.000	1.000	1.000	0.704	0.600	0.499	0.500	0.500	0.499
240,000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	0.702	0.499	0.499	0.499	0.502

Appendix

Table 5: GPT test accuracy, scaled budget (

T=1.25N

Setting	Training examples $N$	Iterations $T$	Expected passes per example	Total examples processed
Fixed	2,400	10,000	1067	$2.56\times 10^{6}$
Fixed	8,000	10,000	320	$2.56\times 10^{6}$
Fixed	24,000	10,000	107	$2.56\times 10^{6}$
Fixed	80,000	10,000	32	$2.56\times 10^{6}$
Fixed	$\infty$	10,000	0 (no repeats)	$2.56\times 10^{6}$
Scaled	2,400	3,000	320	$7.7\times 10^{5}$

Appendix

Table 6: Expected number of passes per training example and total examples processed (batch size 256) under the fixed and scaled budgets.

Feature name	Bits	RA	Predictivity ( $\alpha_{i}$ )
$f_{1}$	4	1/6	60%
$f_{2}$	6	1/20	70%
$f_{3}$	8	1/70	80%
$f_{4}$	10	1/252	90%
$f_{5}$	12	1/924	100%

Appendix

Table 7: Data Set with 5 features with linearly scaling size. In order to help quantify the relative availability (RA) of different features in a given data set we also report the inverse of the number of unique

{\bm{h}}

vectors for a feature of that many bits.

Panel of Figure 15 — Figure 15: Plots of accuracies on noiseless k-parity when the training iterations is scaled with the data set size $(T=1.25N)$ Left: Test Accuracy for GPT model. Right: Train Accuracy for GPT model

Panel of Figure 17 — Figure 17: GPT accuracy on a noisy k-parity feature, $\alpha_{1}=0.8$ , with the iteration budget scaled with the data set size ( $T=1.25N$ ). Left two panels: test accuracy. Right two panels: train accuracy.

Panel of Figure 18 — Figure 18: Accuracy on a noisy k-parity feature, $\alpha_{1}=0.8$ , under a fixed budget ( $T=10^{4}$ ). Left: MLP. Right: GPT.

Panel of Figure 22 — Figure 22: Test accuracy for Tiered Data Set with a fixed training budget Top: MLP. Bottom: GPT.

Panel of Figure 23 — Figure 23: Test accuracy for Tiered Data Set with a scaling training budget Top: MLP. Bottom: GPT.

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 544 papers

Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
Jul 20, 2026Damien Teney, Liangze Jiang, Hemanth Saratchandran +1Neural Network Activation Functions Inductive Bias
Idiap Research Institute · Adelaide University · EPFL
Putting Transformer Theories to the Test
Oct 23, 2026Awni Altabaa, Tobias Schnabel, Kiran TomlinsonTransformer Transformer Expressivity
Yale University · Microsoft Research
Transformers Can Learn Rules They've Never Seen: Proof of Computation Beyond Interpolation
Mar 17, 2026Andy GrayTransformer Neural Network Generalization
Kortical

Jul 20, 2026J/K move · Enter open · S save

Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

Damien Teney, Liangze Jiang, Hemanth Saratchandran, Simon Lucey

Idiap Research Institute · Adelaide University · EPFL

Neural Network Activation Functions Inductive Bias Neural Architecture Search

Open paper AskPDF arXiv

Transformers are remarkably versatile and their design is largely consistent across a variety of applications. But are they optimal for any given task or dataset? The answer may be key for pushing AI beyond merely scaling current designs. Method. We present a method to optimize a transformer architecture for a given dataset, which we use as a tool to study optimal task-specific inductive biases. This method replaces the most important non-linearities (GeLUs,;softmax) with functions learned on held-out data. We then train the resulting architectures on other datasets, as a way to evaluate the compatibility between pairs of tasks. Findings. On algorithmic toy tasks, we identify new architectures with dramatic improvements in learning speed, in- and out-of-distribution generalization, and stability across seeds. The new designs prove very task-specific however, and indicate that these tasks require inductive biases very different from those of standard transformers. On code and language modeling datasets, we also find architectures with consistent, yet smaller improvements. These designs transfer much better across datasets and domains (English & computer code). Implications. Our results show that standard transformers are rarely a local optimum in the space of architectures. Simple alternatives can perform much better but sacrifice universality. This suggests that there may be room for improved architectures that better support multiple capabilities simultaneously, such as fluency and robust reasoning.

Abstract

Transformer

Explore similar work

Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

Putting Transformer Theories to the Test

Transformers Can Learn Rules They've Never Seen: Proof of Computation Beyond Interpolation