Learnable Pulse Accumulation for On-Device Speech Recognition: How Much Attention Do You Need?

Yakov Pyotr Shkolnikov

eess.ASMar 11, 2026

Learnable Pulse Accumulation for On-Device Speech Recognition: How Much Attention Do You Need?

Authors: Yakov Pyotr Shkolnikov

Abstract

Self-attention scales quadratically with sequence length, limiting transformer-based speech models on edge devices. We introduce the Learnable Pulse Accumulator (LPA), an O(n) replacement that substitutes key-query dot products with learned gating functions: content-dependent rectangular pulses, periodic windows, and position-dependent basis functions. An MSE diagnostic sweep determines per-layer replacement difficulty and ordering. Replacing 8 of 12 wav2vec2-base layers yields 10.61% word error rate (WER) on LibriSpeech test-clean, +7.24 percentage points (pp) over the 3.37% baseline, with 3.27x speedup at 120s audio on Apple M4 Pro via an optimized MLX inference path. Cross-domain validation on SepFormer speech enhancement shows all 16 intra-chunk attention layers can be replaced without collapse, suggesting the depth wall arises from linguistic computation rather than an LPA limitation. LPA's near-binary gates at inference enable dense GPU computation with no CPU-GPU synchronization, and all operations map to mobile neural accelerators.

Figures & tables

Figure 1: (a) Standard self-attention computes an

n{\times}n

matrix via

QK^{\top}

. (b) LPA replaces this with three learned gate types that define soft windows over the sequence. Gated accumulation produces per-pulse summaries at

O(nP)

cost. All operations are accelerator-compatible.

Configuration	WER	$\Delta$
Naïve replacement †	58.33	—
+ Selective init, unfreeze LN †	26.50	$-$ 31.8
+ Inter-stage alignment †	16.18	$-$ 10.3
+ FFN co-adapt (0.1 $\times$ lr) †	14.64	$-$ 1.5
+ Temperature curriculum †	11.72	$-$ 2.9
+ Periodic + positional gates	10.25	$-$ 1.5

Table 1: Cumulative ablation at 8/12 layers replaced (dev-clean WER %). Each row adds one technique to the previous configuration. Rows marked

\dagger

are estimated by correcting an early evaluation bug (

-

9.3 pp uniform offset, Appendix B ). The bottom three rows use the corrected evaluation directly.

Figure 2: WER (%) vs. number of replaced layers across four cumulative configurations (full data in Table 7 , Appendix). MSE-ordered replacement yields lower WER at every stage.

Layers	SI-SNRi	$\Delta$ from baseline
0 (baseline)	5.69	—
2/16	8.89	$+$ 3.20
4/16	8.92	$+$ 3.23
8/16	8.14	$+$ 2.45
11/16	7.68	$+$ 1.99
16/16	6.82	$+$ 1.13

Table 2: SepFormer SI-SNRi (dB) on held-out test-clean during progressive LPA replacement of intra-chunk attention. All 16 layers are replaced without collapse.

Audio	Base	Base	8/12	12/12	8/12	Speedup
	(fp32)	(fp16)	(fp32)	(fp32)	(MLX)	vs fp16
10s	47	45	59	65	42	1.07 $\times$
30s	183	186	157	135	125	1.49 $\times$
60s	500	522	350	246	244	2.14 $\times$
120s	1668	1767	895	479	540	3.27 $\times$

Table 3: Inference time (ms) on Apple M4 Pro, batch size 1. PyTorch : MPS backend. MLX : Metal GPU, FP16 with FP32 accumulation. FP16 attention is slower than FP32 on MPS (the

n{\times}n

matmul does not benefit from half precision on this hardware), confirming that the MLX speedup reflects LPA’s algorithmic advantage.

Figure 3: Speedup vs. FP16 attention baseline on Apple M4 Pro, batch 1. Top: inference time (ms, log scale). Bottom: speedup ratio. FP16 attention is slower than FP32 on MPS (Table 3 ).

	wav2vec2 WER (%)		SepFormer SI-SNRi (dB)
Layers	LPA	SM	LPA	SM
0 (base)	3.18	3.18	5.69	5.69
1	4.26	5.44	7.27	8.38
2	4.64	7.31	8.89	9.60
4	6.38	15.47	8.92	9.62
8	9.92	29.04	8.14	9.12

Table 4: LPA vs. SummaryMixing (SM) progressive replacement. wav2vec2: dev-clean WER (%,

\downarrow

). SepFormer: held-out test-clean SI-SNRi (dB,

\uparrow

, 16 intra-chunk layers). SepFormer uses identical training budgets; wav2vec2 SM uses fewer samples per epoch (see text).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Audio	Attn	Attn	LPA (PT)	LPA (MLX)	Speedup
	(fp32)	(fp16)	(fp32)	(fp16)	vs fp16
10 s	47.1 ms	45.3 ms	58.8 ms	42.4 ms	1.07 $\times$
30 s	182.7 ms	186.1 ms	156.9 ms	124.9 ms	1.49 $\times$
60 s	499.7 ms	521.6 ms	349.5 ms	244.2 ms	2.14 $\times$
120 s	1668.0 ms	1767.1 ms	894.8 ms	540.1 ms	3.27 $\times$

Appendix

Table 5: Inference latency on Apple M4 Pro (36 GB, 273 GB/s), batch=1. 8/12 LPA layers replaced. Speedup relative to attention baseline. FP16 attention is slower than FP32 on MPS, so the precision-fair comparison favors LPA even more.

Layer	MSE	Surviving / 144	Difficulty
L0	0.006	48	easy
L2	0.007	48	easy
L1	0.007	51	easy
L3	0.021	57	moderate
L6	0.030	74	moderate
L5	0.034	75	moderate

Appendix

Table 6: Per-layer MSE difficulty from the elastic net diagnostic sweep. Layers sorted by increasing difficulty. The sweep overprovisions each layer with 144 pulses (12

\times

3 types

\times

4 heads) and uses L1/L2 regularization to prune; the surviving count indicates how much capacity each layer demands. The MSE ranking determines replacement order (Sec. 5.2 ). The per-layer allocation suggested by the sweep has not been validated end-to-end (Appendix F ).

Layers	Naïve	+Recipe	+Architecture	+Order
0	3.34	3.34	3.34	3.34
1	4.59	4.88	4.38	4.25
2	6.21	5.17	4.71	4.67
3	9.94	7.17	6.55	4.77
4	10.43	6.93	6.35	5.31
5	13.20	7.73	7.24	5.26

Appendix

Table 7: Best WER (%) during progressive replacement. Columns show cumulative recipe improvements: training techniques (100h), architectural additions with 360h data, and deferred replacement order with MSE auxiliary loss. Naïve column estimated via uniform offset correction (Appendix B ).

Layers	dev-clean	test-clean	test-other
0	3.18	3.37	8.67
1	4.26	4.63	11.83
2	4.64	4.78	12.67
4	6.38	6.62	16.52
6	7.35	7.90	18.98
7	7.90	8.50	20.84

Appendix

Table 8: WER (%) across evaluation splits for the +Architecture configuration. All results use greedy CTC decoding without language model. Dev-clean baseline differs from Table 7 (3.18% vs. 3.34%) because this evaluation includes utterances up to 30s while training-time evaluation filters at 10s.

Audio	Base	Base	8/12	12/12	8/12	Speedup vs Base fp16
	(fp32)	(fp16)	(fp32)	(fp32)	(MLX)	12/12 PT	8/12 MLX
10s	47	45	59	65	42	0.69 $\times$	1.07 $\times$
30s	183	186	157	135	125	1.38 $\times$	1.49 $\times$
60s	500	522	350	246	244	2.12 $\times$	2.14 $\times$
120s	1668	1767	895	479	540	3.69 $\times$	3.27 $\times$

Appendix

Table 9: Inference time (ms) across configurations and durations on Apple M4 Pro, batch 1. PT = PyTorch/MPS. MLX = MLX/Metal (FP16). Speedup is relative to the attention baseline (FP16).

Audio	Frames	Attention	LPA (12p)	Ratio
10s	500	0.95 MB	23 KB	42 $\times$
30s	1,500	8.6 MB	70 KB	125 $\times$
60s	3,000	34.3 MB	140 KB	250 $\times$
120s	6,000	137.3 MB	281 KB	500 $\times$

Appendix

Table 10: Per-layer peak memory for gates/attention weights (float32).

Component	Attention	LPA ( $P{=}12$ )	LPA ( $P{=}36$ )
Linear projections	1,696	1,696	1,696
$QK^{\top}$ / gate prediction	3,311	797	807
Softmax / element-wise	2,110	201	201
$\text{Scores}\times V$ / accumulation	3,311	136	142
Total per layer	10,428	2,835	2,851
$\times$ 12 layers	125 ms	34.0 ms	34.2 ms

Appendix

Table 11: Roofline analysis: per-layer inference time (

\mu

s) on M4 Pro,

B{=}1

T{=}6000

(120s audio). Four

768{\times}768

linear projections dominate LPA time regardless of pulse count.

Figure 4: Per-layer memory for gates (LPA,

P{=}12

) vs. attention score matrix across audio durations (FP32). LPA memory grows linearly while attention grows quadratically.

Explore similar work

LAST: Looped Audio Spectrogram Transformer
Oct 1, 2026Haider Al-Tahan, Sean O'Brien, Anastasia Razdaibiedina +1Audio Understanding Recurrent Transformers
Georgia Institute of Technology · University of California San Diego · Google DeepMind
Test-Time Compute Scaling for ASR with Depth-Conditioned Looped Transformers
Jun 3, 2026Yacouba Kaloga, Shashi Kumar, Shakeel A. Sheikh +3Inference-Time Scaling Test-Time Scaling
Idiap Research Institute, Switzerland · EPFL, Switzerland · BUT, Czech Republic
GrainSpeech: Less Context, More Detail for Compact Speech Synthesis
Sep 16, 2026Zitao Liang, Chang GaoTTS Synthesis Speech Processing
Department of Microelectronics, Delft University of Technology, The Netherlands

Oct 1, 2026J/K move · Enter open · S save

LAST: Looped Audio Spectrogram Transformer

Haider Al-Tahan, Sean O'Brien, Anastasia Razdaibiedina, N. Apurva Ratan Murty

Georgia Institute of Technology · University of California San Diego · Google DeepMind

Audio Understanding Recurrent Transformers Audio Classification

Open paper AskPDF arXiv

Increasing depth of transformer models improves recognition, but it comes at a substantial cost. Each additional layer requires more parameters, which makes the process computationally inefficient. We ask whether additional processing can focus on integrating features already computed. Looped Audio Spectrogram Transformer (LAST) first processes all tokens, then reuses the same blocks to refine only the class token over fixed audio features, thereby making later passes inexpensive. On AudioSet, ten-pass LAST achieves 0.345 mean average precision, exceeding a twelve-layer sequential transformer by 2.1% relative with 49.4% fewer parameters, 42% fewer multiply-accumulate operations, and 9.8% higher measured throughput. Across separately trained models, increasing the pass count from two to ten improves accuracy while adding only 1.2% computation. Further evaluations show improved robustness to temporal masking and various other auditory augmentations, with better generalization on classification tasks with music, environmental, and event sounds.

Abstract

Explore similar work

LAST: Looped Audio Spectrogram Transformer

Test-Time Compute Scaling for ASR with Depth-Conditioned Looped Transformers

GrainSpeech: Less Context, More Detail for Compact Speech Synthesis