How Far Can the Gemma 4 Speech Encoder Go Without Changing Its Weights? Saudi and Gulf Arabic Speech Recognition with a Frozen and an Adapter-Augmented Encoder
We study the 305M-parameter audio encoder shared by Gemma 4 and EmbeddingGemma 2 (bit-identical to that of Gemma 4 E2B except for the final audio-embedding projection) for Saudi and Gulf Arabic speech recognition, without changing any of its pretrained weights; the adapters of our adapted system are additive and kept separate. A frozen system trains only a 22.6M-parameter CTC reader over a learned mix of four intermediate encoder layers and the encoder output; an adapted system adds 3.1M low-rank adapter parameters, a 41.5M-parameter hybrid CTC/attention reader and joint decoding with a sub-word language model. Against Cohere Transcribe Arabic (2.07B parameters), re-run on identical audio, the frozen system trails in word error rate (WER) but is not significantly different in character error rate (CER) on five of six human-labelled conditions, while running 3.4 times faster on CPU with greedy decoding. The adapted system is not significantly different in WER on a public Saudi customer-service call (19.20% against 19.27%) and on clean Emirati speech, is significantly better on telephone-coded Emirati speech, has a significantly lower CER on all six conditions of these sets, and beats a 0.6B recogniser fine-tuned on SADA in WER on all six; it still trails Cohere in WER on long-form Saudi speech. On a blind, human-corrected public call-centre sample it ranks 4 of 15 systems, behind three commercial APIs (significantly only one) and ahead of the other self-hostable systems by point estimate. An optimised joint decoder runs it at 139 times real time on one GPU. We also find, on our data, that references made by correcting machine drafts favour the drafting system. Code and public-set results are released.
Figures & tables
Figure 1: The two systems. The Gemma 4 audio encoder is never changed (grey). Both systems train a layer mix, a reader and a CTC head (blue); the adapted system adds low-rank adapters inside the encoder, an attention decoder with self-conditioned CTC and joint CTC/attention search (orange).
Clean 16 kHz
Telephone 8 kHz
WER
CER
WER
CER
Reader
greedy
+LM
+LM
greedy
+LM
+LM
Calls only
71.06
59.84
32.44
74.16
63.55
36.12
+ SADA, telephone-coded
59.98
48.55
24.16
63.33
51.58
26.85
+ SADA, clean
60.06
48.36
24.41
63.45
52.25
28.15
Table 1: SADA test set (4,971 utterances): WER and CER (%) on the original 16 kHz audio and after simulated telephone coding (8 kHz).
System
Clean
Telephone
Frozen (telephone-SADA reader)
48.55 / 24.16
51.58 / 26.85
Adapted
40.38 / 21.31
42.51 / 23.34
Nemotron 0.6B, fine-tuned on SADA
46.50 / 21.21
48.22 / 22.62
Cohere Transcribe Arabic 2B
46.85 / 27.59
47.88 / 29.00
Table 2: SADA test (4,971 utterances), WER / CER (%) of every system under the same normaliser. Our systems use CTC search with the word 4-gram.
Test set
n
Cohere
Ours + LM
Ours greedy
Saudi customer-service calls (clean)
48
19.27 / 10.05
29.62 / 11.91
34.50 / 11.07
Saudi customer-service calls (telephone)
48
20.53 / 10.92
30.63 / 12.72
37.04 / 12.13
Casablanca, Emirati (clean)
813
48.35 / 20.35
54.63 / 21.26
62.79 / 19.93
Casablanca, Emirati (telephone)
813
53.15 / 24.38
58.82 / 25.38
67.24 / 23.36
SaudiTalk long-form (clean)
133
56.75 / 29.51
65.29 / 29.60
70.17 / 28.61
SaudiTalk long-form (telephone)
133
55.80 / 28.40
66.56 / 31.36
72.46 / 30.47
Table 3: Additional Saudi and Gulf test sets: WER / CER (%) of Cohere Transcribe Arabic and of the frozen-encoder system (telephone-SADA reader) with and without the language model.
Test set
n
Δ WER
Δ CER
Corpus WER (Cohere / ours)
Saudi customer-service calls (clean)
48
+10.35 [+7.16, +13.50]
+1.85 [-0.44, +4.00]
20.77 / 30.12
Saudi customer-service calls (telephone)
48
+10.10 [+7.49, +12.79]
+1.81 [-0.06, +3.65]
21.87 / 31.91
Casablanca, Emirati (clean)
813
+6.28 [+4.52, +8.05]
+0.91 [-0.52, +2.30]
47.52 / 53.53
Casablanca, Emirati (telephone)
813
+5.67 [+3.74, +7.57]
+1.00 [-0.60, +2.59]
51.75 / 57.32
SaudiTalk long-form (clean)
133
+8.54 [+6.89, +10.20]
+0.09 [-1.58, +1.56]
52.69 / 61.35
SaudiTalk long-form (telephone)
133
+10.76 [+9.37, +12.14]
+2.96 [+1.96, +3.96]
51.89 / 63.07
Table 4: Paired difference, frozen-encoder system (+LM) minus Cohere, in percentage points, with 95% bootstrap intervals over utterances (10,000 resamples); the differences are bootstrap means of per-utterance differences. Negative values favour the frozen-encoder system. The last column gives corpus-level WER (total errors over total reference words).
Frozen
Adapted
Test set
Cohere
Nemotron
CTC + LM
CTC + LM
Joint + LM
Saudi customer-service calls (clean)
19.27 / 10.05
24.12 / 8.58
29.62 / 11.91
20.86 / 10.02
19.20 / 7.77
Saudi customer-service calls (telephone)
20.53 / 10.92
25.41 / 9.85
30.63 / 12.72
22.61 / 10.58
21.02 / 8.39
Casablanca, Emirati (clean)
48.35 / 20.35
53.27 / 17.09
54.63 / 21.26
50.66 / 20.77
46.98 / 14.89
Casablanca, Emirati (telephone)
53.15 / 24.38
56.56 / 19.81
58.82 / 25.38
53.50 / 23.53
50.01 / 17.40
SaudiTalk long-form (clean)
56.75 / 29.51
61.25 / 25.76
65.29 / 29.60
60.29 / 29.78
58.79 / 25.58
Table 5: Human-labelled test sets, WER / CER (%): Cohere Transcribe Arabic, the frozen system (telephone-SADA reader, word 4-gram) and the adapted system with CTC beam search and a word 4-gram, or with joint CTC/attention search (beam 16) and a sub-word 6-gram; Nemotron is the 0.6B model fine-tuned on SADA. A dash marks a run not made.
Figure 2: WER and CER on the human-labelled test sets for Cohere, the SADA-fine-tuned Nemotron model and our two systems (adapted: joint search; values in Table 5 ).
Test set
Search
Δ WER
Δ CER
Corpus WER
Saudi customer-service calls (clean)
Joint + LM
-0.07 [-3.90, +3.48]
-2.28 [-4.45, -0.21]
20.77 / 19.81
Saudi customer-service calls (telephone)
Joint + LM
+0.49 [-2.51, +3.46]
-2.53 [-4.23, -0.84]
21.87 / 21.60
Casablanca, Emirati (clean)
Joint + LM
-1.37 [-3.11, +0.43]
-5.47 [-6.88, -4.10]
47.52 / 47.38
Casablanca, Emirati (telephone)
Joint + LM
-3.14 [-5.01, -1.32]
-6.98 [-8.63, -5.40]
51.75 / 50.06
SaudiTalk long-form (clean)
Joint + LM
+2.04 [+0.64, +3.41]
-3.93 [-5.59, -2.49]
52.69 / 54.34
SaudiTalk long-form (telephone)
Joint + LM
+4.64 [+3.41, +5.86]
-1.76 [-2.85, -0.73]
51.89 / 56.20
Table 6: Paired difference, adapted system minus Cohere, in percentage points, with 95% bootstrap intervals over utterances (10,000 resamples). Negative values favour the adapted system. The last column gives corpus-level WER (Cohere / adapted).
Test set
Search
Δ WER
Δ CER
Saudi customer-service calls (clean)
Joint + LM
-4.92 [-8.25, -1.68]
-0.81 [-2.05, +0.38]
Saudi customer-service calls (telephone)
Joint + LM
-4.39 [-7.81, -1.07]
-1.46 [-2.88, -0.12]
Casablanca, Emirati (clean)
Joint + LM
-6.28 [-7.70, -4.88]
-2.20 [-2.92, -1.50]
Casablanca, Emirati (telephone)
Joint + LM
-6.55 [-8.00, -5.13]
-2.41 [-3.17, -1.63]
SaudiTalk long-form (clean)
Joint + LM
-2.46 [-3.35, -1.59]
-0.19 [-0.59, +0.23]
SaudiTalk long-form (telephone)
Joint + LM
-2.18 [-3.02, -1.33]
-0.58 [-1.00, -0.15]
Table 7: Paired difference, adapted system minus Nemotron 0.6B (fine-tuned on SADA), in percentage points, with 95% bootstrap intervals over utterances. Negative values favour the adapted system.
System
Clean
Telephone
Adapted (ours)
29.89 / 11.30 (61)
31.51 / 11.92 (61)
Cohere Transcribe Arabic 2B
34.17 / 14.59 (62)
30.87 / 14.10 (62)
ElevenLabs Scribe v2
35.32 / 17.16 (61)
37.00 / 19.30 (61)
Table 8: Public Saudi call-centre dialogue sample with human-corrected references and a balanced blind prefill: each clip’s draft came from one of the three systems, hidden from the annotator, and each system is scored only on clips it did not draft (number of clips in brackets). WER / CER (%).
System
Self-hostable
n
Clean
Telephone
Δ WER vs ours
GPT-Transcribe [ 52 ]
no
61
18.84 / 8.14
19.21 / 9.13
+11.05 [+6.14, +16.17] ∗
Chirp 3 [ 53 ]
no
61
26.50 / 9.78
25.70 / 10.12
+3.39 [-1.48, +8.17]
MAI-Transcribe-2 [ 54 ]
no
61
27.28 / 9.12
24.32 / 9.11
+2.61 [-1.88, +6.63]
Adapted (ours)
yes
61
29.89 / 11.30
31.51 / 11.92
–
Grok STT 1.0 [ 59 ]
no
61
31.23 / 17.12
31.94 / 14.25
-1.34 [-8.10, +4.94]
Gemini 3.5 Transcribe [ 58 ]
no
61
31.33 / 13.05
29.73 / 11.62
-1.44 [-8.09, +4.66]
Table 9: Blind call-centre sample: 15 recognisers sorted by clean WER. Each system is scored on the clips that neither it nor our system drafted ( n ); commercial APIs drafted none. The last column is the paired WER difference, ours minus that system, on clean audio over clips drafted by neither, with a 95% bootstrap interval; ∗ marks intervals that exclude zero. WER / CER (%).
Figure 3: Clean-audio WER of the fifteen systems on the blind call-centre sample, sorted; self-hostable systems in blue. Values and intervals in Table 9 .
Figure 4: Outputs of seven systems for one clip of the blind call-centre sample, after the scoring normalisation. Red words differ from the reference; a grey placeholder marks a missing reference word.
System
Self-hostable
WER / CER (%)
Speed
Price per hour
GPT-Transcribe
no
18.84 / 8.14
cloud API
$0.27
Chirp 3
no
26.50 / 9.78
cloud API
$0.96
MAI-Transcribe-2
no
27.28 / 9.12
cloud API
$0.10
Adapted (ours)
yes
29.89 / 11.30
139 ×
self-hosted
Grok STT 1.0
no
31.23 / 17.12
cloud API
$0.10
Gemini 3.5 Transcribe
no
31.33 / 13.05
cloud API
$0.30
Table 10: Final comparison on the blind call-centre sample (clean audio, clips drafted by neither the system nor ours). Speed is end-to-end throughput measured on one Quadro RTX 6000 (Table 15 ; different audio and optimisation level; see Section 5.9 ); cloud APIs were not timed comparably. Price is the list price for batch transcription per audio hour, fetched on 2026-10-16 from each vendor’s pricing page (MAI-Transcribe-2 via OpenRouter, for which no vendor price is published); for open-weight models we accessed through an API, the API price is shown. Self-hosted systems pay only for their own hardware.
Greedy
+ 4-gram LM
System
WER
CER
WER
CER
Calls only
60.43
27.10
49.00
25.99
+ SADA, telephone-coded
58.69
25.61
47.20
25.32
+ SADA, clean
58.15
25.65
47.26
25.47
Nemotron 3.5 ASR 0.6B, zero-shot (beam 8)
–
51.33
25.83
Nemotron 3.5 ASR 0.6B, fine-tuned on SADA (beam 8)
–
46.71
21.47
Table 11: In-house call-centre recordings (72 windows, 8 kHz; machine-drafted references reviewed by one annotator, see Section 4 ): mean per-window WER and CER (%) with greedy CTC decoding and with the 4-gram language model. The reference systems use their own decoders, so their scores appear in the right-hand columns.
System
Cohere draft
Cloud ASR draft
Edited
Calls only
60.11
37.87
52.75
+ SADA, telephone-coded
58.38
36.28
50.01
+ SADA, clean
57.54
37.14
50.11
Nemotron 0.6B, fine-tuned
56.16
35.94
54.72
Cohere 2B
43.60
30.88
42.37
Table 12: Call-centre WER (%) split by where the reference came from: Cohere draft accepted unchanged (30 windows), commercial cloud recogniser’s draft accepted unchanged (33), and edited by the annotator (9).
System
Speech
Numbers
Calls only
49.25
48.63
+ SADA, telephone-coded
47.70
46.47
+ SADA, clean
47.16
47.41
Nemotron 0.6B, fine-tuned
47.41
45.67
Cohere 2B
37.36
38.00
Table 13: Call-centre WER (%) by window type: conversational speech (43 windows) and windows dominated by spoken numbers such as identity, phone and account numbers (29).
Figure 5: Learned layer-mix weights of the frozen telephone-SADA reader. The last two bars are two branches that both carry the projected encoder output (Section 3.1 ).
GPU
CPU (16 threads)
System
Params (M)
RTFx
Mem. (GiB)
GPU-h
RTFx
core-h
Frozen encoder + reader (greedy)
328
257
1.5
3.9
31.1
514
encoder only
305
348
–
2.9
34.4
–
Cohere Transcribe Arabic (greedy), batch 16
2,066
157
4.8
6.4
9.2
1,739
Cohere Transcribe Arabic (greedy), batch 32
2,066
222
5.2
4.5
–
–
Nemotron 0.6B fine-tuned (greedy)
638
–
–
–
22.6
708
Table 14: Speed and compute cost on identical audio with length-sorted batches. RTFx is seconds of audio processed per second of wall time, including feature extraction and greedy decoding. GPU: one NVIDIA Quadro RTX 6000 (24 GB), 56 minutes of audio (Casablanca Emirati test). CPU: 16 threads of an Intel Xeon Gold 6230, 13 minutes of audio, batch 8, 32-bit floats. Cost is given as GPU-hours or CPU core-hours per 1,000 hours of audio. The 4-gram beam search adds 12 CPU core-hours per 1,000 hours (one process, RTFx 84).
System
RTFx
CS-call tel. WER / CER
Optimised joint search, 1 process
68
20.32 / 8.34
Optimised joint search, 2 processes
106
20.48 / 8.38
Optimised joint search, 3 processes
133
20.48 / 8.38
Optimised joint search, 4 processes
139
20.48 / 8.38
Original joint search (beam 10)
1.7
Adapted, CTC greedy (batch 1 / 16)
174 / 141
Table 15: Throughput of the adapted system and the reference recognisers on one idle Quadro RTX 6000 (RTFx = seconds of audio per second of wall time, end to end). Upper part: the optimised joint CTC/attention pipeline with the sub-word LM (827 s of audio, batches of 16 utterances) and its WER/CER on the telephone customer-service call. Lower part: the original joint search and the other systems on the 1673 s of the in-house call windows.
Orukeet replaces half of an adapted Parakeet encoder's temporal filters with 12,288 fitted Gabor kernels, freezes these replacements, and trains the remaining parameters on multilingual and multi-accent data. Final adaptation and checkpoint selection use LibriSpeech test-other. Across 20,146 FLEURS recordings in 25 languages, pooled word error rate (WER) falls from Parakeet's 11.01% to Orukeet's 9.85%, a 10.6% relative reduction. Orukeet has lower WER on 23 of the 25 languages. Orukeet outperforms Parakeet on 61 out of 74 tested splits, including LibriSpeech test-clean (1.46% vs. 1.53% WER), test-other (2.86% vs. 3.14%), and FLEURS English (3.82% vs. 4.28%). All comparisons decode the same audio with matched NeMo settings. The fitted kernels are stored as ordinary convolution weights, retaining Parakeet's architecture and inference operators.
We describe the winning system for Task 2 of the KSAA-2026 Shared Task on Arabic Speech Dictation with Automatic Diacritization. The task requires producing fully diacritized Arabic text from speech audio and undiacritized transcripts, with only 2,327 training samples available and no external data permitted. Our system fine-tunes CATT-Whisper, a character-level multimodal model combining a pretrained CATT text encoder with a frozen Whisper speech encoder. The key to our approach is training regularization: R-Drop consistency regularization, Optuna-optimized hyperparameters with high weight decay, and Focal Loss. At inference, we average 200 stochastic forward passes across four model checkpoints using Monte Carlo Dropout at the softmax probability level. The system achieves 23.26% WER on the primary leaderboard metric (with case endings, including no-diacritic positions), placing 1st among all participants.
Meshal Alamr, Hassan Alqaeri, Abdullah Aldahlawi
Thaka, Advanced AI and Information Technology Riyadh, Saudi Arabia
Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time. We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models. A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock. The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.
Harsha Vardhan Khurdula, Abhinav Kumar Singh, Yoeven D Khemlani +1