Textual Echo Cancellation

Shaojin Ding; Ye Jia; Ke Hu; Quan Wang

eess.ASAug 13, 2020

Textual Echo Cancellation

Authors: Shaojin Ding, Ye Jia, Ke Hu, Quan Wang

Organizations: Google LLC, USA

Abstract

In this paper, we propose Textual Echo Cancellation (TEC) - a framework for cancelling the text-to-speech (TTS) playback echo from overlapping speech recordings. Such a system can largely improve speech recognition performance and user experience for intelligent devices such as smart speakers, as the user can talk to the device while the device is still playing the TTS signal responding to the previous query. We implement this system by using a novel sequence-to-sequence model with multi-source attention that takes both the microphone mixture signal and source text of the TTS playback as inputs, and predicts the enhanced audio. Experiments show that the textual information of the TTS playback is critical to enhancement performance. Besides, the text sequence is much smaller in size compared with the raw acoustic signal of the TTS playback, and can be immediately transmitted to the device or ASR server even before the playback is synthesized. Therefore, our proposed approach effectively reduces Internet communication and latency compared with alternative approaches such as acoustic echo cancellation (AEC).

Figures & tables

Figure 1: Acoustic echoes caused by TTS playback overlapping with user query.

Figure 2: Diagram of the textual echo cancellation framework.

Spectral analysis	frame length: 50 ms; frame shift: 12.5 ms;
	128 Mel-filterbanks
Audio encoder	Conv layers $\times$ 2	32 3 $\times$ 3 kernel with 2 $\times$ 2 stride;
		ReLU; batch norm
	Bi-CLSTM $\times$ 1	256 units per direction;
		1 $\times$ 3 kernel with 1 $\times$ 1 stride
	Bi-LSTM $\times$ 3	256 units per direction

Table 1: Hyperparameters of the TEC network.

Figure 3: Diagram of the decoder with multi-source attention.

Condition	Synthetic subset	User’s speech	Interfering speech
Single interfering voice	training	LibriTTS training	LJ Speech training
	test-clean	LibriTTS test-clean	LJ Speech test
	test-other	LibriTTS test-other	LJ Speech test
Multiple interfering voices	training	LibriTTS training	VCTK training
	test-clean	LibriTTS test-clean	VCTK test
	test-other	LibriTTS test-other	VCTK test

Table 2: Data configuration for our experiments. This table shows how the microphone signal mixtures were generated. For example, in single interfering voice condition, the synthetic training set was mixed using LibriTTS training set and LJ Speech training set.

Condition	Method	WER (%)		MCD		MOS		Side input (KB)	GFLOPS
		test-	test-	test-	test-	test-clean	test-other
		clean	other	clean	other	test-clean	test-other
Ground-truth LibriTTS	-	2.30	4.50	0.00	0.00	4.43 $\pm$ 0.04	3.82 $\pm$ 0.06	-	-
Single interfering voice	Microphone signal	89.9	120.5	18.83	21.44	-	-	-	-
	AEC-NLMS	48.6	60.1	12.26	12.57	1.95 $\pm$ 0.10	1.28 $\pm$ 0.09	310	0
	Vanilla-Seq2seq	25.4	54.0	7.85	8.84	1.99 $\pm$ 0.06	1.47 $\pm$ 0.05	0	6.32

Table 3: Word Error Rate (WER), Mel-Cepstral Distortion (MCD), and speech naturalness Mean Opinion Score (MOS) evaluation results of the single speaker interfering voice and multiple interfering voices conditions. The MOS is presented with 95% confidence intervals. We also include the size of the side input in kilobytes (the TTS playback echo or the TTS source text) and the floating point operations per second in Giga (GFLOPS).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Original Internal Implementation (Main Paper)	Open-Source Reproduction ( wq2012/tec )
Modeling Framework	Internal Google Lingvo / TensorFlow trained on $2\times 2$ Cloud TPU slices (global batch size 32, 50,000 steps).	Standalone open-source lingvo ( $\geq 0.12.7$ ) and tensorflow ( $\geq 2.13$ ) supporting CPU, GPU, and TFLite execution.
Acoustic Frontend	24 kHz sample rate, 50 ms frame length (1,200 samples), 12.5 ms frame shift (300 samples), 128 log-Mel filterbanks.	WaveformProcessor ( tec/waveform_processor.py ): 24 kHz, 50 ms Hann window, 12.5 ms hop, 2,048-point FFT, 128 log-Mel bins (20–12,000 Hz, natural-log compression with magnitude floor $10^{-3}$ ).
Audio Encoder	2 strided $3\times 3$ Conv2D layers ( $2\times 2$ stride, 32 channels), 1 Bi-CLSTM ( $1\times 3$ kernel, 256 units/dir), 3 Bi-LSTM layers (256 units/dir).	SpeechEncoderV1 ( tec/encoder_speech.py ): Subclasses lingvo.tasks.asr.encoder.AsrEncoder with identical $3\times 3$ Conv2D pyramid ( $4\times$ time reduction), configurable Bi-CLSTM, and 3 Bi-LSTM layers (512-dim output).
Text Tokenizer & Encoder	Internal TTS text normalization and pre-trained 512-dim phoneme embedding table, followed by 3 Conv1D layers ( $5\times 1$ , 512 channels, BatchNorm, ReLU) and 1 Bi-LSTM (256 units/dir).	CharTokenizer ( tec/tokenizer.py ) + TtsEncoderV2 ( tec/encoder_text.py ): Self-contained 96-symbol printable ASCII character vocabulary ( <pad>=0 , </s>=1 , <unk>=2 + 93 ASCII characters) with learned 512-dim embedding, 3 Conv1D layers ( $5\times 1$ , 512 channels, BatchNorm, ReLU, dropout 0.5), and 1 Bi-LSTM (256 units/dir, zoneout 0.1).
Multi-Source Attention	Independent 5-mixture Gaussian Mixture Model (GMM) monotonic attention [ 13 ] per encoder stream with context summation $\mathbf{c}^{t}=\mathbf{c}^{t}_{\mathbf{x}}+\mathbf{c}^{t}_{\mathbf{y}}$ .	MultiSourceAttentionStep ( tec/multi_source_attention_steps.py ): Wraps GmmMonotonicAttention (5 mixtures, 128-dim hidden state) or AdditiveAttention for each source stream ( source_0 , source_1 ) and sums context vectors.
Spectrogram Decoder	Tacotron-2 [ 43 ] Pre-Net ( $2\times 256$ ReLU, dropout 0.5), 2-layer LSTM (256 units), linear Mel and stop-token heads, and 5-layer Conv1D Post-Net ( $5\times 1$ , 512 channels, BatchNorm, $\tanh$ ).	MultiSourceFbeDecoderV1 and FbeDecoderV1 ( tec/decoder.py ) + PostEditConvNet ( tec/layers.py ): Identical layer dimensions with frame reduction factor $r=4$ , masked $L_{1}+L_{2}+\mathrm{BCE}$ loss, and step-aligned spectral residual skip connections for fast convergence.

Appendix

Table 4: Detailed comparison between the original internal implementation (Section 2 – 3 ) and the standalone open-source reproduction ( textual-echo-cancellation ).

File Path	Key Classes / Functions	Paper Reference & Functionality
tec/waveform_processor.py	WaveformProcessor , compute_fft_size	Section 2 .1 & Table 1 : 24 kHz STFT, 128-bin log-Mel filterbank extraction ( WaveformsToSpectrograms ), and Griffin-Lim / phase-preserving waveform synthesis ( SpectrogramsToWaveforms ).
tec/tokenizer.py	CharTokenizer	Section 2 .3: Maps TTS source text strings to 96-symbol ASCII character token IDs and padding masks ( text_to_ids , ids_to_text , batch_encode ).
tec/encoder_speech.py	SpeechEncoderV1	Section 2 .2, Eq. (1) & Table 1 : Encodes 128-bin log-Mel spectrograms $\mathbf{x}\in\mathbb{R}^{T_{x}\times 128}$ into 512-dim hidden sequences $\mathbf{h_{x}}$ at $4\times$ reduced frame rate.
tec/encoder_text.py	TtsEncoderV2	Section 2 .3, Eq. (2) & Table 1 : Encodes TTS character IDs into 512-dim hidden representations $\mathbf{h_{y}}$ via 512-dim embeddings, 3 Conv1D layers ( $5\times 1$ ), and 1 Bi-LSTM.
tec/multi_source_attention_steps.py	MultiSourceAttentionStep	Section 2.5 , Eq. (8)–(10): Computes per-source GMM monotonic attention contexts $\mathbf{c}_{\mathbf{x}}^{t}$ and $\mathbf{c}_{\mathbf{y}}^{t}$ and sums them into $\mathbf{c}^{t}=\mathbf{c}_{\mathbf{x}}^{t}+\mathbf{c}_{\mathbf{y}}^{t}$ .
tec/layers.py	PostEditConvNet	Section 2 .4, Eq. (7) & Table 1 : 5-layer Conv1D residual Post-Net ( $5\times 1$ kernel, 512 channels, BatchNorm, $\tanh$ on first 4 layers, linear final projection).

Appendix

Table 5: Module-by-module reference for the open-source textual-echo-cancellation package ( https://github.com/wq2012/tec ).

#	Clean User Query (LibriTTS test-clean )	Interfering TTS Source Text (LJ Speech test )	ASR on Original Mixture (Before TEC)	ASR on TEC Enhanced Audio (After TEC)
1	“I can’t see you at all, anywhere.”	“a table showing the figures for the year ending Michaelmas eighteen oh two.”	“I can’t see you at all anywhere for the year ending, Michael Miss 1802. ” ( 100.0% WER )	“ I can’t see you at all anywhere. ” ( 0.0% WER )
2	“Because the thing had been such a scare?”	“and to approve or disapprove the public policy written into these laws.”	“Because the thing had been such a scare. The public policy written into these laws. ” ( 87.5% WER )	“ because the thing had been such a scare. ” ( 0.0% WER )
3	“I must know about you.”	“was living in the city while the walls were still standing, though in a ruinous condition.”	“I must know about you. The yellow walls were still standing, though in a row it is conditioned. ” ( 260.0% WER )	“ I must know about you. ” ( 0.0% WER )
4	“I should much prefer that you called in the aid of the police.”	“a subsequent bullet, which was lethal, shattered the right side of his skull.”	“I should much prefer that you called in the aid of a police. Shattered the right side of his skull. ” ( 61.5% WER )	“ I should much prefer that you call them the aid of the police. ” ( 15.4% WER )

Appendix

Table 6: Side-by-side ASR recognition results ( Whisper base.en ) before and after Textual Echo Cancellation ( wq2012/tec_single_interfering ) on the four benchmark examples included in the Hugging Face Space demo ( https://huggingface.co/spaces/wq2012/tec ).

Condition	Split	Source Manifests (User + Interfering)	User Pool	Interfering Pool	Prepared TFRecord Shard
Single interfering voice	train	LibriTTS train-clean-100 + LJSpeech train (90%)	33,236	11,790	1,000 mixtures (1.3 GB)
	test-clean	LibriTTS test-clean + LJSpeech test (10%)	4,837	1,310	100 mixtures (131 MB)
	test-other	LibriTTS test-other + LJSpeech test (10%)	5,120	1,310	100 mixtures (126 MB)
Multiple interfering voices	train	LibriTTS train-clean-100 + VCTK train (90%/spk)	33,236	39,859	1,000 mixtures (1.1 GB)
	test-clean	LibriTTS test-clean + VCTK test (10%/spk)	4,837	4,424	100 mixtures (106 MB)
	test-other	LibriTTS test-other + VCTK test (10%/spk)	5,120	4,424	100 mixtures (105 MB)

Appendix

Table 7: Summary of deterministic dataset splits and prepared TFRecord shards in the open-source reproduction.

			Open-Source Reproduction ( wq2012/tec + Qwen3-ASR-0.6B )						Paper Ref. (Table 3 )
			WER (%) $\downarrow$		MCD (dB) $\downarrow$		Side Input (KB) $\downarrow$		WER (%)	MCD (dB)
Condition	Method	Hugging Face Repo ( wq2012/... )	clean	other	clean	other	clean	other	clean / other	clean / other
Single interfering voice (LibriTTS + LJ Speech)	GroundTruth	—	3.52 (7/199)	6.78 (12/177)	0.00	0.00	0.000	0.000	2.30 / 4.50	0.00 / 0.00
	MicrophoneSignal	—	90.45 (180/199)	114.12 (202/177)	12.86	14.61	0.000	0.000	89.9 / 120.5	18.83 / 21.44
	NlmsAec (AEC-NLMS)	—	88.44 (176/199)	107.34 (190/177)	12.80	14.48	243.465	209.085	48.6 / 60.1	12.26 / 12.57
	Vanilla-Seq2seq	vanilla_seq2seq_single_interfering	45.23 (90/199)	91.53 (162/177)	9.58	11.34	0.000	0.000	25.4 / 54.0	7.85 / 8.84

Appendix

Table 8: Open-source reproduction evaluation results across both conditions ( Single interfering voice : LibriTTS + LJ Speech; Multiple interfering voices : LibriTTS + VCTK) on test-clean and test-other , evaluated with Qwen3-ASR-0.6B-F16 ( audio.cpp ) and 13-MFCC DTW MCD. Word error counts are shown in parentheses as

(\mbox{errors}/\mbox{total words})

. Original paper reference numbers from Table 3 are listed in the right-hand columns for direct comparison.

Explore similar work

Streaming T5-based Text-to-Speech Synthesis with Limited Lookahead
Jun 20, 2026Muyang Du, Jason Roche, Junjie LaiText-To-Speech Synthesis F5-Tts
1NVIDIA, China · 2NVIDIA, USA
Taming Long-form Text-to-Speech
Sep 15, 2026Rongxiang Wang, Berkin Durmus, Aysegul Orhon +2Autoregressive Text-To-Speech Voice Conversion
Argmax, Inc. · University of Virginia · Bilkent University
DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech
Jul 5, 2026Junwon Moon, Seungbeom Kim, Yejin Lee +4Autoregressive Text-To-Speech Diffusion Language Models
1Sungkyunkwan University · University of Seoul

Jun 20, 2026, cs.SDJ/K move · Enter open · S save

Streaming T5-based Text-to-Speech Synthesis with Limited Lookahead

Muyang Du, Jason Roche, Junjie Lai

1NVIDIA, China · 2NVIDIA, USA

Text-To-Speech Synthesis F5-Tts Speaker Similarity Streaming Lookahead Probe-Logit Distillation

Open paper AskPDF arXiv

Streaming text-to-speech synthesis in cascaded LLM-TTS systems still faces latency challenges as most TTS models require full context before initiating generation. We present S5-TTS, a streaming variant of T5-TTS that enables low-latency, word-by-word incremental speech synthesis through encoder-decoder language modeling and monotonic alignment learning. S5-TTS begins generating speech immediately after receiving the first few words, substantially reducing end-to-end response latency. To maintain quality under limited lookahead, we introduce a lookahead-causal masking mechanism with Conv-based auxiliary attention that preserves intelligibility and speaker similarity, and employ interleaved multi-source distillation to further restore naturalness. Experiments show that S5-TTS achieves comparable quality to full-context T5-TTS, supports zero-shot synthesis with high speaker similarity, and significantly reduces end-to-end latency for practical conversational AI systems.

Abstract

Explore similar work

Streaming T5-based Text-to-Speech Synthesis with Limited Lookahead

Taming Long-form Text-to-Speech

DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech