Polynomial Mixing for Efficient Self-supervised Speech Encoders

Eva Feillet; Ryan Whetten; David Picard; Alexandre Allauzen

cs.CLFeb 28, 2026

Polynomial Mixing for Efficient Self-supervised Speech Encoders

Authors: Eva Feillet, Ryan Whetten, David Picard, Alexandre Allauzen

Organizations: Universit´e Paris-Saclay, CNRS, Laboratoire Interdisciplinaire des Sciences du Num´erique · Miles team, Universit´e Paris-Dauphine-PSL · Laboratoire Informatique d’Avignon, Avignon Universit´e · LIGM, ´Ecole Nationale des Ponts et Chauss´ees

Abstract

State-of-the-art speech-to-text models typically employ Transformer-based encoders that model token dependencies via self-attention mechanisms. However, the quadratic complexity of self-attention in both memory and computation imposes significant constraints on scalability. In this work, we propose a novel token-mixing mechanism, the Polynomial Mixer (PoM), as a drop-in replacement for multi-head self-attention. PoM computes a polynomial representation of the input with linear complexity with respect to the input sequence length. We integrate PoM into a self-supervised speech representation learning framework based on BEST-RQ and evaluate its performance on downstream speech recognition tasks. Experimental results demonstrate that PoM achieves a competitive word error rate compared to full self-attention and other linear-complexity alternatives, offering an improved trade-off between performance and efficiency in time and memory.

Figures & tables

Figure 1: Principle of the Polynomial Mixer. The input sequence is projected through

k

polynomial branches, aggregated into a global representation

H(X)

, and combined with a token-wise selector

S

. The output is obtained by projecting the selected state back to the input space.

	Model	Clean	Clean + LM	Other	Other + LM
$\sim$ 95M param.	RelPosMHA	7.96 ( $\pm$ 0.32)	4.89 ( $\pm$ 0.25)	17.61 ( $\pm$ 0.54)	12.13 ( $\pm$ 0.44)
	RoPE MHA	8.06 ( $\pm$ 0.31)	4.90 ( $\pm$ 0.26)	17.53 ( $\pm$ 0.48)	11.98 ( $\pm$ 0.45)
	regular MHA	8.59 ( $\pm$ 0.32)	5.37 ( $\pm$ 0.25)	19.44 ( $\pm$ 0.54)	13.47 ( $\pm$ 0.46)
	PoM base	8.31 ( $\pm$ 0.31)	5.42 ( $\pm$ 0.27)	19.06 ( $\pm$ 0.53)	13.62 ( $\pm$ 0.48)
	SummaryMixing	9.79 ( $\pm$ 0.34)	5.93 ( $\pm$ 0.27)	22.80 ( $\pm$ 0.60)	15.84 ( $\pm$ 0.51)
	Mamba †	7.61	5.50 ( $\pm$ 0.28)	19.97	15.37

Table 1: WER on LibriSpeech test-clean and test-other . Models are pretrained on LibriSpeech-960h and fine-tuned on train-100 . Confidence intervals from 1000 bootstrap trials. Results with † reported from [ 7 ] . Lower is better. Best MHA variant in bold, best linear mixer underlined.

Figure 2: Inference time and peak memory usage of BEST-RQ models (

\sim 95

M params) with various token mixers. Input length is increased from 10 to 80 seconds. MHA requires significantly more time and VRAM as the input size increases in comparison with linear alternatives, including PoM.

mixer	ldrop	clean	clean+LM	other	other+LM
PoM base	1	$8.76\pm 0.28$	$5.51\pm 0.15$	$19.87\pm 0.51$	$14.02\pm 0.32$
PoM select	1	$8.74\pm 0.20$	$5.73\pm 0.18$	$20.56\pm 0.60$	$14.89\pm 0.68$
PoM 2ways	1	$8.75\pm 0.26$	$5.65\pm 0.11$	$20.28\pm 0.57$	$14.60\pm 0.31$
PoM 3ways	1	$8.77\pm 0.47$	$5.82\pm 0.11$	$20.86\pm 0.76$	$15.08\pm 0.42$

Table 2: WER averaged by PoM version, with standard deviation.

mixer	ldrop	clean	clean+LM	other	other+LM
PoM (all)	0	8.92	5.71 ( $-3.21$ )	19.86	14.12 ( $-5.74$ )
PoM (all)	1	8.61 ( $-0.31$ )	5.42 ( $-3.50$ )	19.81 ( $-0.05$ )	14.00 ( $-5.86$ )
MHA	0	8.39	5.30 ( $-3.09$ )	19.01	13.25 ( $-5.76$ )
MHA	1	8.20 ( $-0.19$ )	5.05 ( $-3.34$ )	18.19 ( $-0.82$ )	12.53 ( $-6.48$ )

Table 3: WER averaged by mixer type, depending on the use of a language model at inference (LM) and the use of layerdrop during training (ldrop). (0: without, 1: with). Values in parentheses are absolute deltas vs. the same mixer’s baseline (LM=0, ldrop=0). Grey cells indicate the mixer-specific reference values.

Speech Processing

Speech processing develops computational methods for recognizing, synthesizing, analyzing, and transforming spoken audio.

See all 412 papers

PoM: A Linear-Time Replacement for Attention with the Polynomial Mixer
Apr 7, 2026David Picard, Nicolas Dufour, Lucas Degeorge +14Self-Attention Transformer Inference
LIGM, CNRS, Univ Gustave Eiffel, ENPC, Institut Polytechnique de Paris, France · LIX, École Polytechnique, CNRS, IP Paris, France · AMIAD, Pole recherche +4
WNet: Discrete Wavelets Transform for Efficient Token Mixing
Oct 3, 2026Rana Aref Salama, Abdou Youssef, Mona DiabTransformer Self-Attention
George Washington University, Washington DC, USA · Faculty of Computers and Artificial Intelligence, Cairo University, Egypt · Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
Poly-attention: a general scheme for higher-order self-attention
Feb 2, 2026Sayak Chakrabarti, Toniann Pitassi, Josh AlmanSelf-Attention Compositional Reasoning
Computer Science, Columbia University, New York, NY 10027.

Apr 7, 2026J/K move · Enter open · S save

PoM: A Linear-Time Replacement for Attention with the Polynomial Mixer

David Picard, Nicolas Dufour, Lucas Degeorge, Arijit Ghosh+13

LIGM, CNRS, Univ Gustave Eiffel, ENPC, Institut Polytechnique de Paris, France · LIX, École Polytechnique, CNRS, IP Paris, France · AMIAD, Pole recherche+4

Self-Attention Transformer Inference Linear Attention

Open paper AskPDF arXiv

This paper introduces the Polynomial Mixer (PoM), a novel token mixing mechanism with linear complexity that serves as a drop-in replacement for self-attention. PoM aggregates input tokens into a compact representation through a learned polynomial function, from which each token retrieves contextual information. We prove that PoM satisfies the contextual mapping property, ensuring that transformers equipped with PoM remain universal sequence-to-sequence approximators. We replace standard self-attention with PoM across five diverse domains: text generation, handwritten text recognition, image generation, 3D modeling, and Earth observation. PoM matches the performance of attention-based models while drastically reducing computational cost when working with long sequences. The code is available at https://github.com/davidpicard/pom.

Abstract

Speech Processing

Explore similar work

PoM: A Linear-Time Replacement for Attention with the Polynomial Mixer

WNet: Discrete Wavelets Transform for Efficient Token Mixing

Poly-attention: a general scheme for higher-order self-attention