How Far Can the Gemma 4 Speech Encoder Go Without Changing Its Weights? Saudi and Gulf Arabic Speech Recognition with a Frozen and an Adapter-Augmented Encoder

Amr Abdelkhalik

cs.CLOct 18, 2026

How Far Can the Gemma 4 Speech Encoder Go Without Changing Its Weights? Saudi and Gulf Arabic Speech Recognition with a Frozen and an Adapter-Augmented Encoder

Authors: Amr Abdelkhalik

Organizations: Independent Researcher, Egypt

Abstract

We study the 305M-parameter audio encoder shared by Gemma 4 and EmbeddingGemma 2 (bit-identical to that of Gemma 4 E2B except for the final audio-embedding projection) for Saudi and Gulf Arabic speech recognition, without changing any of its pretrained weights; the adapters of our adapted system are additive and kept separate. A frozen system trains only a 22.6M-parameter CTC reader over a learned mix of four intermediate encoder layers and the encoder output; an adapted system adds 3.1M low-rank adapter parameters, a 41.5M-parameter hybrid CTC/attention reader and joint decoding with a sub-word language model. Against Cohere Transcribe Arabic (2.07B parameters), re-run on identical audio, the frozen system trails in word error rate (WER) but is not significantly different in character error rate (CER) on five of six human-labelled conditions, while running 3.4 times faster on CPU with greedy decoding. The adapted system is not significantly different in WER on a public Saudi customer-service call (19.20% against 19.27%) and on clean Emirati speech, is significantly better on telephone-coded Emirati speech, has a significantly lower CER on all six conditions of these sets, and beats a 0.6B recogniser fine-tuned on SADA in WER on all six; it still trails Cohere in WER on long-form Saudi speech. On a blind, human-corrected public call-centre sample it ranks 4 of 15 systems, behind three commercial APIs (significantly only one) and ahead of the other self-hostable systems by point estimate. An optimised joint decoder runs it at 139 times real time on one GPU. We also find, on our data, that references made by correcting machine drafts favour the drafting system. Code and public-set results are released.

Figures & tables

Figure 1: The two systems. The Gemma 4 audio encoder is never changed (grey). Both systems train a layer mix, a reader and a CTC head (blue); the adapted system adds low-rank adapters inside the encoder, an attention decoder with self-conditioned CTC and joint CTC/attention search (orange).

Reader	greedy	+LM	+LM	greedy	+LM	+LM
	Clean 16 kHz			Telephone 8 kHz
	WER		CER	WER		CER
Calls only	71.06	59.84	32.44	74.16	63.55	36.12
+ SADA, telephone-coded	59.98	48.55	24.16	63.33	51.58	26.85
+ SADA, clean	60.06	48.36	24.41	63.45	52.25	28.15

Table 1: SADA test set (4,971 utterances): WER and CER (%) on the original 16 kHz audio and after simulated telephone coding (8 kHz).

System	Clean	Telephone
Frozen (telephone-SADA reader)	48.55 / 24.16	51.58 / 26.85
Adapted	40.38 / 21.31	42.51 / 23.34
Nemotron 0.6B, fine-tuned on SADA	46.50 / 21.21	48.22 / 22.62
Cohere Transcribe Arabic 2B	46.85 / 27.59	47.88 / 29.00

Table 2: SADA test (4,971 utterances), WER / CER (%) of every system under the same normaliser. Our systems use CTC search with the word 4-gram.

Test set	$n$	Cohere	Ours + LM	Ours greedy
Saudi customer-service calls (clean)	48	19.27 / 10.05	29.62 / 11.91	34.50 / 11.07
Saudi customer-service calls (telephone)	48	20.53 / 10.92	30.63 / 12.72	37.04 / 12.13
Casablanca, Emirati (clean)	813	48.35 / 20.35	54.63 / 21.26	62.79 / 19.93
Casablanca, Emirati (telephone)	813	53.15 / 24.38	58.82 / 25.38	67.24 / 23.36
SaudiTalk long-form (clean)	133	56.75 / 29.51	65.29 / 29.60	70.17 / 28.61
SaudiTalk long-form (telephone)	133	55.80 / 28.40	66.56 / 31.36	72.46 / 30.47

Table 3: Additional Saudi and Gulf test sets: WER / CER (%) of Cohere Transcribe Arabic and of the frozen-encoder system (telephone-SADA reader) with and without the language model.

Test set	$n$	$\Delta$ WER	$\Delta$ CER	Corpus WER (Cohere / ours)
Saudi customer-service calls (clean)	48	+10.35 [+7.16, +13.50]	+1.85 [-0.44, +4.00]	20.77 / 30.12
Saudi customer-service calls (telephone)	48	+10.10 [+7.49, +12.79]	+1.81 [-0.06, +3.65]	21.87 / 31.91
Casablanca, Emirati (clean)	813	+6.28 [+4.52, +8.05]	+0.91 [-0.52, +2.30]	47.52 / 53.53
Casablanca, Emirati (telephone)	813	+5.67 [+3.74, +7.57]	+1.00 [-0.60, +2.59]	51.75 / 57.32
SaudiTalk long-form (clean)	133	+8.54 [+6.89, +10.20]	+0.09 [-1.58, +1.56]	52.69 / 61.35
SaudiTalk long-form (telephone)	133	+10.76 [+9.37, +12.14]	+2.96 [+1.96, +3.96]	51.89 / 63.07

Table 4: Paired difference, frozen-encoder system (+LM) minus Cohere, in percentage points, with 95% bootstrap intervals over utterances (10,000 resamples); the differences are bootstrap means of per-utterance differences. Negative values favour the frozen-encoder system. The last column gives corpus-level WER (total errors over total reference words).

			Frozen	Adapted
Test set	Cohere	Nemotron	CTC + LM	CTC + LM	Joint + LM
Saudi customer-service calls (clean)	19.27 / 10.05	24.12 / 8.58	29.62 / 11.91	20.86 / 10.02	19.20 / 7.77
Saudi customer-service calls (telephone)	20.53 / 10.92	25.41 / 9.85	30.63 / 12.72	22.61 / 10.58	21.02 / 8.39
Casablanca, Emirati (clean)	48.35 / 20.35	53.27 / 17.09	54.63 / 21.26	50.66 / 20.77	46.98 / 14.89
Casablanca, Emirati (telephone)	53.15 / 24.38	56.56 / 19.81	58.82 / 25.38	53.50 / 23.53	50.01 / 17.40
SaudiTalk long-form (clean)	56.75 / 29.51	61.25 / 25.76	65.29 / 29.60	60.29 / 29.78	58.79 / 25.58

Table 5: Human-labelled test sets, WER / CER (%): Cohere Transcribe Arabic, the frozen system (telephone-SADA reader, word 4-gram) and the adapted system with CTC beam search and a word 4-gram, or with joint CTC/attention search (beam 16) and a sub-word 6-gram; Nemotron is the 0.6B model fine-tuned on SADA. A dash marks a run not made.

Figure 2: WER and CER on the human-labelled test sets for Cohere, the SADA-fine-tuned Nemotron model and our two systems (adapted: joint search; values in Table 5 ).

Test set	Search	$\Delta$ WER	$\Delta$ CER	Corpus WER
Saudi customer-service calls (clean)	Joint + LM	-0.07 [-3.90, +3.48]	-2.28 [-4.45, -0.21]	20.77 / 19.81
Saudi customer-service calls (telephone)	Joint + LM	+0.49 [-2.51, +3.46]	-2.53 [-4.23, -0.84]	21.87 / 21.60
Casablanca, Emirati (clean)	Joint + LM	-1.37 [-3.11, +0.43]	-5.47 [-6.88, -4.10]	47.52 / 47.38
Casablanca, Emirati (telephone)	Joint + LM	-3.14 [-5.01, -1.32]	-6.98 [-8.63, -5.40]	51.75 / 50.06
SaudiTalk long-form (clean)	Joint + LM	+2.04 [+0.64, +3.41]	-3.93 [-5.59, -2.49]	52.69 / 54.34
SaudiTalk long-form (telephone)	Joint + LM	+4.64 [+3.41, +5.86]	-1.76 [-2.85, -0.73]	51.89 / 56.20

Table 6: Paired difference, adapted system minus Cohere, in percentage points, with 95% bootstrap intervals over utterances (10,000 resamples). Negative values favour the adapted system. The last column gives corpus-level WER (Cohere / adapted).

Test set	Search	$\Delta$ WER	$\Delta$ CER
Saudi customer-service calls (clean)	Joint + LM	-4.92 [-8.25, -1.68]	-0.81 [-2.05, +0.38]
Saudi customer-service calls (telephone)	Joint + LM	-4.39 [-7.81, -1.07]	-1.46 [-2.88, -0.12]
Casablanca, Emirati (clean)	Joint + LM	-6.28 [-7.70, -4.88]	-2.20 [-2.92, -1.50]
Casablanca, Emirati (telephone)	Joint + LM	-6.55 [-8.00, -5.13]	-2.41 [-3.17, -1.63]
SaudiTalk long-form (clean)	Joint + LM	-2.46 [-3.35, -1.59]	-0.19 [-0.59, +0.23]
SaudiTalk long-form (telephone)	Joint + LM	-2.18 [-3.02, -1.33]	-0.58 [-1.00, -0.15]

Table 7: Paired difference, adapted system minus Nemotron 0.6B (fine-tuned on SADA), in percentage points, with 95% bootstrap intervals over utterances. Negative values favour the adapted system.

System	Clean	Telephone
Adapted (ours)	29.89 / 11.30 (61)	31.51 / 11.92 (61)
Cohere Transcribe Arabic 2B	34.17 / 14.59 (62)	30.87 / 14.10 (62)
ElevenLabs Scribe v2	35.32 / 17.16 (61)	37.00 / 19.30 (61)

Table 8: Public Saudi call-centre dialogue sample with human-corrected references and a balanced blind prefill: each clip’s draft came from one of the three systems, hidden from the annotator, and each system is scored only on clips it did not draft (number of clips in brackets). WER / CER (%).

System	Self-hostable	$n$	Clean	Telephone	$\Delta$ WER vs ours
GPT-Transcribe [ 52 ]	no	61	18.84 / 8.14	19.21 / 9.13	+11.05 [+6.14, +16.17] ∗
Chirp 3 [ 53 ]	no	61	26.50 / 9.78	25.70 / 10.12	+3.39 [-1.48, +8.17]
MAI-Transcribe-2 [ 54 ]	no	61	27.28 / 9.12	24.32 / 9.11	+2.61 [-1.88, +6.63]
Adapted (ours)	yes	61	29.89 / 11.30	31.51 / 11.92	–
Grok STT 1.0 [ 59 ]	no	61	31.23 / 17.12	31.94 / 14.25	-1.34 [-8.10, +4.94]
Gemini 3.5 Transcribe [ 58 ]	no	61	31.33 / 13.05	29.73 / 11.62	-1.44 [-8.09, +4.66]

Table 9: Blind call-centre sample: 15 recognisers sorted by clean WER. Each system is scored on the clips that neither it nor our system drafted (

n

); commercial APIs drafted none. The last column is the paired WER difference, ours minus that system, on clean audio over clips drafted by neither, with a 95% bootstrap interval; ∗ marks intervals that exclude zero. WER / CER (%).

Figure 3: Clean-audio WER of the fifteen systems on the blind call-centre sample, sorted; self-hostable systems in blue. Values and intervals in Table 9 .

Figure 4: Outputs of seven systems for one clip of the blind call-centre sample, after the scoring normalisation. Red words differ from the reference; a grey placeholder marks a missing reference word.

System	Self-hostable	WER / CER (%)	Speed	Price per hour
GPT-Transcribe	no	18.84 / 8.14	cloud API	$0.27
Chirp 3	no	26.50 / 9.78	cloud API	$0.96
MAI-Transcribe-2	no	27.28 / 9.12	cloud API	$0.10
Adapted (ours)	yes	29.89 / 11.30	139 $\times$	self-hosted
Grok STT 1.0	no	31.23 / 17.12	cloud API	$0.10
Gemini 3.5 Transcribe	no	31.33 / 13.05	cloud API	$0.30

Table 10: Final comparison on the blind call-centre sample (clean audio, clips drafted by neither the system nor ours). Speed is end-to-end throughput measured on one Quadro RTX 6000 (Table 15 ; different audio and optimisation level; see Section 5.9 ); cloud APIs were not timed comparably. Price is the list price for batch transcription per audio hour, fetched on 2026-10-16 from each vendor’s pricing page (MAI-Transcribe-2 via OpenRouter, for which no vendor price is published); for open-weight models we accessed through an API, the API price is shown. Self-hosted systems pay only for their own hardware.

	Greedy		+ 4-gram LM
System	WER	CER	WER	CER
Calls only	60.43	27.10	49.00	25.99
+ SADA, telephone-coded	58.69	25.61	47.20	25.32
+ SADA, clean	58.15	25.65	47.26	25.47
Nemotron 3.5 ASR 0.6B, zero-shot (beam 8)	–		51.33	25.83
Nemotron 3.5 ASR 0.6B, fine-tuned on SADA (beam 8)	–		46.71	21.47

Table 11: In-house call-centre recordings (72 windows, 8 kHz; machine-drafted references reviewed by one annotator, see Section 4 ): mean per-window WER and CER (%) with greedy CTC decoding and with the 4-gram language model. The reference systems use their own decoders, so their scores appear in the right-hand columns.

System	Cohere draft	Cloud ASR draft	Edited
Calls only	60.11	37.87	52.75
+ SADA, telephone-coded	58.38	36.28	50.01
+ SADA, clean	57.54	37.14	50.11
Nemotron 0.6B, fine-tuned	56.16	35.94	54.72
Cohere 2B	43.60	30.88	42.37

Table 12: Call-centre WER (%) split by where the reference came from: Cohere draft accepted unchanged (30 windows), commercial cloud recogniser’s draft accepted unchanged (33), and edited by the annotator (9).

System	Speech	Numbers
Calls only	49.25	48.63
+ SADA, telephone-coded	47.70	46.47
+ SADA, clean	47.16	47.41
Nemotron 0.6B, fine-tuned	47.41	45.67
Cohere 2B	37.36	38.00

Table 13: Call-centre WER (%) by window type: conversational speech (43 windows) and windows dominated by spoken numbers such as identity, phone and account numbers (29).

Figure 5: Learned layer-mix weights of the frozen telephone-SADA reader. The last two bars are two branches that both carry the projected encoder output (Section 3.1 ).

		GPU			CPU (16 threads)
System	Params (M)	RTFx	Mem. (GiB)	GPU-h	RTFx	core-h
Frozen encoder + reader (greedy)	328	257	1.5	3.9	31.1	514
encoder only	305	348	–	2.9	34.4	–
Cohere Transcribe Arabic (greedy), batch 16	2,066	157	4.8	6.4	9.2	1,739
Cohere Transcribe Arabic (greedy), batch 32	2,066	222	5.2	4.5	–	–
Nemotron 0.6B fine-tuned (greedy)	638	–	–	–	22.6	708

Table 14: Speed and compute cost on identical audio with length-sorted batches. RTFx is seconds of audio processed per second of wall time, including feature extraction and greedy decoding. GPU: one NVIDIA Quadro RTX 6000 (24 GB), 56 minutes of audio (Casablanca Emirati test). CPU: 16 threads of an Intel Xeon Gold 6230, 13 minutes of audio, batch 8, 32-bit floats. Cost is given as GPU-hours or CPU core-hours per 1,000 hours of audio. The 4-gram beam search adds 12 CPU core-hours per 1,000 hours (one process, RTFx 84).

System	RTFx	CS-call tel. WER / CER
Optimised joint search, 1 process	68	20.32 / 8.34
Optimised joint search, 2 processes	106	20.48 / 8.38
Optimised joint search, 3 processes	133	20.48 / 8.38
Optimised joint search, 4 processes	139	20.48 / 8.38
Original joint search (beam 10)	1.7
Adapted, CTC greedy (batch 1 / 16)	174 / 141

Table 15: Throughput of the adapted system and the reference recognisers on one idle Quadro RTX 6000 (RTFx = seconds of audio per second of wall time, end to end). Upper part: the optimised joint CTC/attention pipeline with the sub-word LM (827 s of audio, batches of 16 utterances) and its WER/CER on the telephone customer-service call. Lower part: the original joint search and the other systems on the 1673 s of the in-house call windows.

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 484 papers

Orukeet: Multilingual ASR with Frozen Gabor Kernels
Sep 9, 2026Nathan Roll, Irene Yi, Büşra Marşan +6Automatic Speech Recognition Multilingual ASR
Oruk AI · Stanford University · OpenWhispr +2
Thaka at KSAA-2026 Task 2: Regularized Fine-Tuning for Arabic Speech Diacritization
May 25, 2026Meshal Alamr, Hassan Alqaeri, Abdullah AldahlawiFine-Tuning Arabic NLP
Thaka, Advanced AI and Information Technology Riyadh, Saudi Arabia
Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model
Jul 14, 2026Harsha Vardhan Khurdula, Abhinav Kumar Singh, Yoeven D Khemlani +1Automatic Speech Recognition Diffusion Language Models
Interfaze AI · San Francisco, CA, USA

Sep 9, 2026J/K move · Enter open · S save

Orukeet: Multilingual ASR with Frozen Gabor Kernels

Nathan Roll, Irene Yi, Büşra Marşan, Vianney Grenez+5

Oruk AI · Stanford University · OpenWhispr+2

Automatic Speech Recognition Multilingual ASR

Open paper AskPDF arXiv

Orukeet replaces half of an adapted Parakeet encoder's temporal filters with 12,288 fitted Gabor kernels, freezes these replacements, and trains the remaining parameters on multilingual and multi-accent data. Final adaptation and checkpoint selection use LibriSpeech test-other. Across 20,146 FLEURS recordings in 25 languages, pooled word error rate (WER) falls from Parakeet's 11.01% to Orukeet's 9.85%, a 10.6% relative reduction. Orukeet has lower WER on 23 of the 25 languages. Orukeet outperforms Parakeet on 61 out of 74 tested splits, including LibriSpeech test-clean (1.46% vs. 1.53% WER), test-other (2.86% vs. 3.14%), and FLEURS English (3.82% vs. 4.28%). All comparisons decode the same audio with matched NeMo settings. The fitted kernels are stored as ordinary convolution weights, retaining Parakeet's architecture and inference operators.

Abstract

Automatic Speech Recognition

Explore similar work

Orukeet: Multilingual ASR with Frozen Gabor Kernels

Thaka at KSAA-2026 Task 2: Regularized Fine-Tuning for Arabic Speech Diacritization

Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model