Real-time voice agents need a speech recognizer that never waits for future audio and whose memory per call does not grow with the length of the call. We present E-Gaborformer, a strictly causal encoder that combines Gated DeltaNet-2 recurrent layers with sliding-window attention in a 3:1 ratio, and builds every temporal convolution from a fixed dictionary of causal Gabor atoms. Its streaming state is constant, about 19 MB per call at production scale including the prediction network, where a causal-attention encoder of the same size accumulates over 1 GB after ten minutes. In a controlled causal study on LibriSpeech, E-Gaborformer leaves the initial plateau earlier, reaches a given accuracy up to 18% sooner, and matches or beats an E-Branchformer trained with the identical front end and recipe. At production scale, a 0.6B-parameter Hebrew and English streaming transducer built on E-Gaborformer has 28 to 31% lower word error rate (WER) than NVIDIA's Nemotron 3.5 ASR Streaming, a model of the same size, on English voice-agent traffic, and up to 20% lower Hebrew WER than our previous production system. The same weights serve strictly causal and 480 ms-lookahead decoding with no loss in accuracy, and a two-class formulation of the RNN-T lattice trains the transducer 2.6x faster with exact loss and gradients.
Figures & tables
Figure 1: One E-Gaborformer block (residual connections omitted). The global branch is a GDN-2 recurrence in three layers out of four and causal sliding-window attention in the fourth; both depthwise convolutions are Gabor-dictionary convolutions.
Figure 2: The 16 fixed causal Gabor atoms (kernel 31, lag −30…0 left to right; frequencies at 80 ms frames). Each channel’s kernel is a learned linear combination of them.
Figure 3: Standard (SiLU) and sigmoid GDN-2 output gate in the causal 1:1 hybrid (three seeds each, 40 ms front end). Left: one SiLU-gate seed never leaves the blank plateau. Right: output RMS of the GDN-2 and attention branches, logged at every evaluation.
study model (12 layers, d=256 )
production encoder (18 layers, d=1024 )
GDN-2 states
9×4 heads: 0.6 MB
14×16 heads: 3.7 MB
attention key/value windows
3 layers: 0.8 MB
4 layers: 4.2 MB
convolution caches
1.6 MB
9.4 MB
total, any stream length
≈ 3 MB
≈ 17 MB (+ ≈ 2 MB prediction network)
unbounded causal attention after 10 min
≈ 185 MB
≈ 1.1 GB
Table 1: Per-stream inference state (fp32, 80 ms frames). Attention windows are 128 frames (10.24 s).
Figure 4: Inference state per stream at production scale ( d=1024 , 18 layers, fp32).
Figure 5: Development WER of causal E-Gaborformer and E-Branchformer with identical front end and recipe (mean of two seeds, evaluated every 250 updates; greedy CTC, no language model). Right: difference between the two models; negative values favor E-Gaborformer.
3 epochs
10 epochs
model
params
final
average of last 3 checkpoints
final
E-Gaborformer (GDN-2 : attention = 3:1)
27.6M
26.2 / 44.5
25.2 / 43.0
20.5 / 39.2
E-Branchformer
26.8M
27.1 / 45.1
26.3 / 43.7
20.8 / 39.2
Table 2: Causal models with 80 ms frames and 128 mel bins. WER (%) on dev-clean / dev-other, greedy CTC, no language model. Three-epoch results are the mean of three seeds; ten-epoch results the mean of two.
epochs (updates)
arms
1 (2,250)
1.5 (3,500)
2 (4,750)
2.5 (5,750)
3 (6,981)
10 (23,270)
Gabor vs. unconstrained convolutions (1:1)
+0.4
−0.5
−0.7
−1.0
−1.2
—
E-Gaborformer vs. E-Branchformer
−1.8
−1.2
−1.0
−1.0
−0.9
−0.3
3:1 vs. 1:1 hybrid
−2.3
−0.7
−0.6
−0.3
0.0
—
Table 3: Development-clean WER difference between two arms during training (mean over seeds; negative favors the first arm), at the same evaluation steps as Table 4 . Unconstrained convolutions lead after one epoch; the Gabor lead then grows.
configuration
1 epoch
left plateau
3 epochs
Round 2: SiLU GDN-2 output gate; 40 ms frames, 80 mel bins
Conformer
85.3 / 89.4
1/3
30.0 / 48.0 (1)
FastConformer
74.5 / 81.3
3/3
—
E-Branchformer
42.1 / 58.2
3/3
28.5 / 46.2 (3)
E-Branchformer, Gabor gate conv
52.4 / 66.1
3/3
30.6 / 48.5 (3)
GDN-2 only, Gabor gate conv
66.4 / 76.4
2/3
29.8 / 47.4 (2)
Table 4: All causal configurations (LibriSpeech-100, greedy CTC). WER (%) on dev-clean / dev-other after one epoch (mean of three seeds) and three epochs (number of seeds in parentheses; in round 2 only seeds that left the plateau were continued). “Left plateau”: seeds below 90% dev-clean WER within the first epoch. 1:1 and 3:1 are GDN-2 / attention ratios; “Gabor” and “free” denote Gabor-dictionary and unconstrained depthwise convolutions.
configuration
1 epoch, raw dictionary
1 epoch, orthonormal basis
3 epochs
Conformer
46.4 / 60.6 (3)
—
—
E-Branchformer
32.6 / 48.7 (3)
—
20.5 / 37.1 (3)
E-Branchformer, Gabor gate conv
77.7 / 83.1 (1)
33.9 / 50.0 (3)
20.8 / 37.6 (1)
GDN-2 only, Gabor gate conv
79.5 / 85.0 (1)
65.9 / 75.6 (2)
23.4 / 40.8 (1)
1:1, Gabor gate conv
99.6 / 99.8 (0)
59.1 / 69.6 (2)
24.4 / 41.0 (1, orthonormal)
1:1, gate conv half Gabor, half free
52.4 / 64.1 (2)
54.6 / 66.3 (3)
20.8 / 37.6 (2)
Table 5: Bidirectional configurations (round 1; SiLU GDN-2 output gate, 40 ms frames, 80 mel bins). WER (%) on dev-clean / dev-other after one epoch (three seeds; seeds that left the plateau in parentheses) with the raw and the orthonormalized Gabor dictionary, and after three epochs (number of seeds).
Figure 6: Development WER of the main round-3 configurations (mean of three seeds, 40 ms front end).
test set
previous production model
current model
relative
Hebrew, general
12.3
10.9
−11%
Hebrew, hard
14.2
11.5
−19%
Hebrew, names
10.2
8.2
−20%
Hebrew, addresses, URLs and emails
14.8
15.8
+7%
English, general
9.5
9.6
+1%
English, hard
11.0
11.0
0%
Table 6: Production test sets, WER (%), greedy decoding, no external language model. Lookahead 0 ms unless stated. WhatsApp: 54 Hebrew voice messages (1.18 h, 8,941 words) decoded in 20 s segments.
model
params
causal
480 ms lookahead
Nemotron 3.5 ASR Streaming
0.6B
14.0
13.3
E-Gaborformer transducer
0.6B
9.6
9.6
Table 7: General English production set (24 recordings, 1,214 words), WER (%), greedy decoding, identical normalization.
Table 8: Training time per update (same batch and loss). Single-GPU rows replay the same eight real micro-batches (about 2,765 s of audio per update); the last rows are live training on three GPUs.
Real-time conversational agents require speaker diarization that streams and runs on a CPU. Most systems apply an utterance-level speaker encoder to short, heavily overlapping chunks, which wastes computation and leaves the model optimized for the wrong task. We instead turn a state-of-the-art speaker recognition architecture into a causal frame-level encoder that reads the stream once and emits one embedding every 80~ms from a bounded two-second window of past audio, and pair it with online clustering that gates every update on the self-similarity of the stream. Trained only by distillation from an utterance-level teacher on simulated and out-of-domain mixtures, and evaluated with one fixed set of hyperparameters, the system is the most accurate streaming diarizer on low-overlap benchmarks at sub-second latency, degrades far less than cache-based systems as the number of speakers grows, and runs five times faster than real time on a single CPU thread.
Nikita Torgashov, Okan Köpüklü
Department of Speech, Music and Hearing KTH Royal Institute of Technology Stockholm, Sweden · Microsoft Munich, Germany
Orukeet replaces half of an adapted Parakeet encoder's temporal filters with 12,288 fitted Gabor kernels, freezes these replacements, and trains the remaining parameters on multilingual and multi-accent data. Final adaptation and checkpoint selection use LibriSpeech test-other. Across 20,146 FLEURS recordings in 25 languages, pooled word error rate (WER) falls from Parakeet's 11.01% to Orukeet's 9.85%, a 10.6% relative reduction. Orukeet has lower WER on 23 of the 25 languages. Orukeet outperforms Parakeet on 61 out of 74 tested splits, including LibriSpeech test-clean (1.46% vs. 1.53% WER), test-other (2.86% vs. 3.14%), and FLEURS English (3.82% vs. 4.28%). All comparisons decode the same audio with matched NeMo settings. The fitted kernels are stored as ordinary convolution weights, retaining Parakeet's architecture and inference operators.
Self-attention scales quadratically with sequence length, limiting transformer-based speech models on edge devices. We introduce the Learnable Pulse Accumulator (LPA), an O(n) replacement that substitutes key-query dot products with learned gating functions: content-dependent rectangular pulses, periodic windows, and position-dependent basis functions. An MSE diagnostic sweep determines per-layer replacement difficulty and ordering. Replacing 8 of 12 wav2vec2-base layers yields 10.61% word error rate (WER) on LibriSpeech test-clean, +7.24 percentage points (pp) over the 3.37% baseline, with 3.27x speedup at 120s audio on Apple M4 Pro via an optimized MLX inference path. Cross-domain validation on SepFormer speech enhancement shows all 16 intra-chunk attention layers can be replaced without collapse, suggesting the depth wall arises from linguistic computation rather than an LPA limitation. LPA's near-binary gates at inference enable dense GPU computation with no CPU-GPU synchronization, and all operations map to mobile neural accelerators.