Rationally Linearized MLPs

Arijit Sehanobish; Byeongchan Kim; Avinava Dubey; Min-hwan Oh; Krzysztof Choromanski

cs.LGOct 9, 2026

Rationally Linearized MLPs

Authors: Arijit Sehanobish, Byeongchan Kim, Avinava Dubey, Min-hwan Oh, Krzysztof Choromanski

Organizations: Kensho Technologies · Seoul National University · Google Research · Google DeepMind ·

Abstract

We present Rationally Linearized Multilayer Perceptrons(RaLiMPs), a new approach to efficiently linearizing MLPs using low-degree rational approximations of standard activation functions. Inspired by polynomial-based MLP linearization, we show that rational approximations can provide accurate representations at substantially lower degree. RaLiMPs exploit these approximations to construct kernel representations with narrow bottlenecks, enabling efficient composition across MLP layers. We provide theoretical guarantees for the resulting approximation and evaluate RaLiMPs across applications ranging from implicit 3D models to large language models. Our experiments demonstrate favorable accuracy--efficiency tradeoffs, with RaLiMPs achieving competitive or improved accuracy while substantially reducing inference cost.

Figures & tables

Figure 1: Compression–quality trade-off across diverse model families. Each point represents a model family compressed using RaLiMPs.

Figure 2: Synthetic Experiments. Approximation for various Feedforward layers with different activations. RaLiMPs beat all other published methods. Error for all methods go down as the number of random features increase. All experiments are run using 10 seeds.

Figure 3: ViT Results. Top Row : Classification on CiIFAR-100, ImageNet and Places365. Bottom Row : Summarized results on VTAB across different image categories. RaLiMP outperforms all variants. Low Rank and EUGen are consistently weak across all methods at high levels of compression. As more layers are replaced, we get up to 57 % reduction in inference parameters.

	Competition math (avg@16 / pass@16)			Reasoning
Method	AMC23	AIME24	AIME25	GSM8K	MATH500	Minerva	Olympiad	GPQA	MMLU
$k{=}4$ (last four FFNs replaced)
SNNK	76.2 88.0	53.8 68.8	47.5 70.8	92.6	89.9	40.4	66.3	48.2 80.4	71.4
EUGen	79.1 90.6	57.2 72.4	49.4 72.0	93.0	90.2	43.1	65.0	49.8 80.8	72.0
RaLiMP (ours)	82.2 96.4	61.0 86.0	52.3 77.5	95.5	92.8	45.3	70.4	55.6 85.2	72.7
$k{=}8$ (last eight FFNs replaced)

Table 1: Qwen3-8B accuracy (%) at matched inference budget with the top-

k

feed-forward layers replaced. For the high-variance benchmarks (AMC23

n{=}83

, AIME24/25

n{=}30

, GPQA

n{=}198

) each cell reports avg@16 with pass@16 beneath it, at

t{=}0.6

; the remaining benchmarks are single-sample at a

16

k-token thinking-mode budget, and MMLU is the 57-subject accuracy. Within each compression level the best value is in bold .

Figure 4: Casual Language Modeling with GPT2-L (770M) on OpenWebText. Left : Perplexity (Lower is better) RaLiMP shows the best validation perplexity among all the efficient methods considered with matched inference parameters. Middle : Memory usage shrinks to almost 50 % as more layers are replaced. Right : Throughput increases as well.

Figure 5: RaLiMP in PiNNs, NeRF and SIREN. RaLiMP matches the performance of the baseline PiNN ( Left ) and outperforms NeRF ( Middle ). RaLiMP also matches the SIREN network outperforming the other variants ( Right ). In all cases RaLiMPs reduce model size by 30 % – 50 % while maintaining performance.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Left: the spectral density

\pi e^{-\xi}

1/(1+t^{2})

is continuous, so no finite set of frequencies is exact and the continuum must be discretized. Right: max-abs reconstruction error on

|t|\leq 1

from

n

frequencies, Gauss–Laguerre nodes against

s\sim\mathrm{Exp}(1)

(mean of 200 draws, shaded 10–90th percentile), both with least-squares-fitted coefficients.

Dataset	train / eval	epochs	batch	lr / wd	augmentation
CIFAR-100	50k / 10k	12–20	256	$5\!\times\!10^{-5}$ / $0.05$	resize, h-flip
Places365	1.71M / 90k	15	128	$5\!\times\!10^{-5}$ / $0.05$	random-resized crop, h-flip
ImageNet-1k	1.28M / 50k	25–35	128–256	$5\!\times\!10^{-5}$ / $0.05$	random-resized crop, h-flip
VTAB-1k ( $19$ tasks)	1k / full test	100	64	$2.5\!\times\!10^{-4}$ / $10^{-3}$	resize, random-resized crop, h-flip

Appendix

Table 4: ViT-B/16 fine-tuning settings. VTAB-1k uses random-resized crop only when the source resolution exceeds

100

pixels.

Hyperparameter	Values
learning rate	$1.9,\,2.1,\,2.5\ (\times 10^{-4})$ ; up to $10^{-2}$ on the hardest Structured tasks
weight decay	$10^{-3}$ , $5\!\times\!10^{-2}$
scale multiplier $c$ in $\tau$	$4,\,8,\,10,\,12$
cap on $\tau$	none, $10$ , $12$ , $16$ , $20$ , $21$
label smoothing / dropout	$0,\,0.05,\,0.1,\,0.15$ / $0,\,0.1$

Appendix

Table 5: Hyperparameters swept on VTAB-1k, around the default recipe of Table 4 . Deep layers of a pretrained ViT have much larger pre-activations than early ones, so capping

\tau

is the one lever specific to our layer.

Figure 7: RaLiMPs in ViT. Replacing more layers by RaLiMPs reduce memory overhead and speed up inference.

Configuration	Value
PTv3 backbone
Attention	Serialized softmax attention (FlashAttention)
Serialization patterns	Z + TZ + H + TH
Patch interaction	Shift Order + Shuffle Order
Positional encoding	xCPE
Encoder depths	$[2,2,2,6,2]$

Appendix

Table 7: PTv3 backbone and RaLiMP configurations for semantic segmentation.

C

denotes the channel dimension of the corresponding block.

\tau_{0}

and

\lambda_{\mathrm{gate},0}

denote the initial values of the learnable rational input scale and gate, respectively.

k

denotes the number of replaced MLP modules, with replacement starting from the top MLP modules within each selected stage.

Figure 8: SIREN Networks. RaLiMP matches the dense baseline while replacing all layers and outperform all variants.

Panel of Figure 9 — Figure 9: Top Row : Reference solution of the viscous Burgers’ equation. Left: the ground-truth field $u(t,x)$ over $(t,x)\in[0,1]\times[-1,1]$ . Right: snapshots at $t\in\{0,0.25,0.5,1\}$ , showing the initial sinusoidal profile steepening into a sharp shock at $x{=}0$ . Bottom Row : Predicted Burgers’ fields. Solution $u(t,x)$ recovered by each model against the reference (leftmost). The dense tanh network and RaLiMP sharply reproduce the shock at $x{=}0$ , whereas EUGen and SNNK smear it and lose the discontinuity, at matched inference budget. Tanh refer to the baseline PiNN model with tanh activation.

	$k=8$	$k=4$
Blocks replaced	28–35	32–35
Linear (random-projection) group dimension	192	192
Quadratic (EUGen squared-dot) group dimension	192	192
Rational quadrature frequencies	6	6
Rational random features per frequency	48	48
Rational group width ( $2\times 3\times 48$ )	288	288

Appendix

Table 8: Architecture of the replaced feed-forward layers. Both settings replace the top

k

blocks of Qwen3-8B and share all other hyperparameters.

		$k{=}4$		$k{=}8$
Benchmark	$L$	pre	$+$ dist	pre	$+$ dist	Base
GSM8K	1,685	94.5	95.5	92.3	95.2	93.5
MATH500	3,528	87.4	92.8	85.2	94.5	90.0
Minerva	4,892	35.3	45.3	28.7	43.4	45.6
AMC23	7,820	75.2	82.2	71.8	78.8	83.0
OlympiadBench	8,244	65.4	70.4	63.8	69.0	69.9

Appendix

Table 9: Effect of the on-policy distillation stage (accuracy, %), benchmarks ordered by base median generation length

L

(tokens). For each compression level, “pre” is the Stage-1 supervised model and “

+

dist” is the same model after Stage 3; “Base” is uncompressed Qwen3-8B. AMC/AIME are avg@16, the rest single-sample; maximum generation length 16,384 tokens.

Figure 10: Accuracy deficit vs. generation length

L

at matched decoding. Points are the benchmarks; the dashed curve is the one-parameter fit

1-e^{-\delta L}

(

\delta{=}3.2\times 10^{-5}

, fit on the unsaturated benchmarks). Swapping more layers (

k{=}8

vs.

k{=}4

) raises the deficit at fixed

L

$r$	AMC23	AIME24	AIME25	GSM8K	MATH500	Minerva	GPQA
$256$	74.7	48.1	41.5	92.9	87.2	37.9	44.9
$512$	75.2	51.0	43.3	94.5	87.4	35.3	41.4

Appendix

Table 10:

k{=}4

at Hadamard rank

r{=}256

vs.

r{=}512

(accuracy after stage 1 training, %; AMC/AIME are avg@16).

Model	Params removed	% of model	% of FFN	Throughput (tok/s)	Speedup vs. base
Uncompressed	—	—	—	763.3	—
$k=4$	0.59B	7.2%	10.8%	785.6	$+2.9\%$
$k=8$	1.17B	14.3%	21.5%	809.6	$+6.1\%$

Appendix

Table 11: Efficiency of replacing the top

k

feed-forward networks of Qwen3-8B (36 blocks, 8.19B parameters). Parameter reduction is relative to the full model / to the feed-forward parameters (which are

\sim

two-thirds of the model). Throughput is compiled decoding (bfloat16; 4096-token prompt, 1024 generated tokens, 16 concurrent sequences, single GPU).

Method	Chair	Drums	Ficus	Hotdog	Lego	Mater.	Mic	Ship	Avg
NeRF	33.70	25.46	26.40	35.81	31.31	29.00	33.11	28.79	30.45
EUGen	33.09	25.18	28.40	35.47	30.60	28.88	32.63	28.66	30.36
SNNK	30.66	24.76	28.21	35.72	30.00	28.64	30.44	28.41	29.61
RaLiMP (ours)	33.19	25.26	28.89	35.73	30.95	29.02	32.86	28.95	30.61

Appendix

Table 12: Synthetic-NeRF held-out test PSNR (dB). RaLiMP beats all other methods, at matched deployed parameters and even beating the dense NeRF baseline.

		$k$ (layers replaced)
Dataset	Method	4	7	9	11
CIFAR-100	Low Rank	86.6	78.3	74.3	69.3
	SNNK	87.3	84.0	80.3	75.9
	EUGen	88.6	86.5	81.3	74.6
	RaLiMP (ours)	89.8	88.1	84.6	80.4
	Baseline	92.0

Appendix

Table 13: ViT results on CIFAR-100, ImageNet, and Places365 (Top-1 Accuracy). RaLiMPs beat all other methods at matched deployed inference budget . Baseline is the dense ViT-B/16 full fine-tune.

	$k{=}4$				$k{=}7$				$k{=}9$				$k{=}11$
Task	R	S	E	L	R	S	E	L	R	S	E	L	R	S	E	L
Natural
CIFAR-100	58.2	52.9	38.2	4.8	57.5	43.3	26.7	3.3	51.0	36.0	13.8	2.5	40.3	29.1	3.6	2.8
Caltech101	85.2	82.1	74.3	25.6	84.4	79.0	63.6	4.2	76.0	75.9	49.1	9.0	69.4	69.2	7.0	3.8
DTD	69.0	66.2	59.9	18.2	67.9	63.5	53.4	8.5	59.1	59.0	44.2	10.5	50.0	49.7	22.8	9.7
Flowers102	95.6	95.4	90.3	10.2	94.3	94.0	76.9	6.0	92.3	91.8	73.6	4.3	85.2	86.1	31.2	6.7

Appendix

Table 14: VTAB-1k top-1 accuracy (%), matched inference budget, per

k

(layers replaced). R aLiMP (ours), S NNK, E UGen, L owrank. Bold = best method at each

(\text{task},k)

Table 22

	FFN layers replaced ( $k$ )
Method	2	6	11	18	24	27	30
Low Rank	16.83	17.57	18.79	19.06	20.91	22.66	25.31
EUGen	16.03	16.08	16.37	16.66	17.06	17.74	18.42
SNNK	14.18	14.32	14.71	15.02	15.69	16.05	16.44
RaLiMP (ours)	14.10	14.22	14.58	14.90	15.26	15.42	15.69
Dense (no replacement)	14.17

Appendix

Table 17: GPT-2 Large (774M, OpenWebText) validation perplexity (lower is better) at a fixed 30k-iteration budget. RaLiMP improves on the EUGen and SNNK baselines at every

k

Figure 11: GLUE results. MCC score is reported for CoLA, F1 score is reported for MRPC and QQP, Spearman correlation is reported for STSB, and accuracy scores are reported for the other tasks. RaLiMP is competitive with baseline while being almost 35% smaller. Results are an average of 5 seeds.

Multilayer Perceptrons

Multilayer perceptrons are layered feedforward neural networks studied for learning, representing, and efficiently computing functions from data.

See all 48 papers

Rational Neural Networks have Expressivity Advantages
Feb 12, 2026Maosen Tang, Alex TownsendNeural Network Activation Functions Neural Network Approximation Theory
Center for Applied Mathematics, Cornell University, United States · Department of Mathematics, Cornell University, United States
Activation-Adjacent Layer Normalization: Effects on Approximation Power
Oct 15, 2026Yunhao Ni, Yuxin Guo, Jie Luo +2Layer Normalization Neural Network Approximation Theory
State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China
All you need is SAMPAT
Jul 10, 2026Jayadeva, Madhur AswaniShallow Neural Networks Interpretable ML
Department of Electrical Engineering, Indian Institute of Technology, Delhi

Feb 12, 2026J/K move · Enter open · S save

Rational Neural Networks have Expressivity Advantages

Maosen Tang, Alex Townsend

Center for Applied Mathematics, Cornell University, United States · Department of Mathematics, Cornell University, United States

Neural Network Activation Functions Neural Network Approximation Theory Neural Network Expressivity

Open paper AskPDF arXiv

We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations such as ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax. For an error target of

\varepsilon>0

, we establish approximation-theoretic separations: Any network built from standard fixed activations can be uniformly approximated on compact domains by a rational-activation network with only

\mathrm{poly}(\log\log(1/\varepsilon))

overhead in size, while the converse provably requires

Ω(\log(1/\varepsilon))

parameters in the worst case. This exponential gap persists at the level of full networks and extends to gated activations and transformer-style nonlinearities. In practice, rational activations integrate seamlessly into standard architectures and training pipelines, allowing rationals to match or outperform fixed activations under identical architectures and optimizers.

Abstract

Multilayer Perceptrons

Explore similar work

Rational Neural Networks have Expressivity Advantages

Activation-Adjacent Layer Normalization: Effects on Approximation Power

All you need is SAMPAT