Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment

Joanna Hong; Sanjeel Parekh; Honglie Chen; Jacob Donley; Ke Tan; Buye Xu; Anurag Kumar

cs.SDJan 30, 2025

Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment

Authors: Joanna Hong, Sanjeel Parekh, Honglie Chen, Jacob Donley, Ke Tan, Buye Xu, Anurag Kumar

Organizations: Meta

Abstract

Building reliable speech systems often requires combining multiple modalities, like audio and visual cues. While such multimodal solutions frequently lead to improvements in performance and may even be critical in certain cases, they come with several constraints such as increased sensory requirements, computational cost, and modality synchronization, to mention a few. These challenges constrain the direct uses of these multimodal solutions in real-world applications. In this work, we develop approaches where the learning happens with all available modalities but the deployment or inference is done with just one or reduced modalities. To do so, we propose a Multimodal Training and Unimodal Deployment (MUTUD) framework which includes a Temporally Aligned Modality feature Estimation (TAME) module that can estimate information from missing modality using modalities present during inference. This innovative approach facilitates the integration of information across different modalities, enhancing the overall inference process by leveraging the strengths of each modality to compensate for the absence of certain modalities during inference. We apply MUTUD to various audiovisual speech tasks and show that it can reduce the performance gap between the multimodal and corresponding unimodal models to a considerable extent. MUTUD can achieve this while reducing the model size and compute compared to multimodal models, in some cases by almost 80%.

Figures & tables

Figure 1 : (a) The left panel shows a comparison between conventional audiovisual speech processing and MUTUD. TAME module enables audiovisual learning without doing video processing during prediction. (b) The upper half in the right panel illustrates MUTUD for an AVSE model. After training the video encoder is discarded. (c) The bottom half in the right panel shows the estimation of video representations using TAME. The illustration is for $t=0$ in Eq 5 .

Method	STOI (%)					SISDR (dB)					PESQ
Method	5	0	-5	-10	-15	5	0	-5	-10	-15	5	0	-5	-10	-15
Noisy Audio	82.6	72.4	60.5	48.8	38.9	5.00	0.01	-5.02	-10.03	-15.07	1.24	1.12	1.07	1.07	1.07
Audio-only	92.7	88.1	80.1	67.5	51.5	13.64	10.55	7.08	2.88	-2.82	2.18	1.80	1.48	1.27	1.14
Audio-only w. similar # params	93.0	88.3	80.4	68.0	52.6	13.75	10.58	7.05	2.86	-2.62	2.30	1.87	1.53	1.30	1.16
MUTUD w.o. pretrained TAME	93.4	89.1	81.6	69.5	53.6	14.07	10.99	7.54	3.32	-2.24	2.37	1.93	1.58	1.33	1.17
MUTUD	93.5	89.2	81.8	69.8	54.0	14.11	11.02	7.56	3.38	-2.19	2.36	1.92	1.57	1.32	1.17

Table 1 : Speech Enhancement performance comparison of different models for 3-BN test condition.

Figure 2 : MUTUD bridges the gap between Audiovisual and Audio-only models. Performance (in %) of gain in STOI through different methods relative to the gain Audiovisual brings in average intelligibility (STOI) of noisy speech samples. MUTUD is able to get most of performance gains of the Audiovisual model across different SNRs. For example, at -5dB SNR Audio-only gives gives 86.0% of the gain of the Audioivisual model whereas MUTUD is at 93.4% of Audiovisual model.

Method	STOI (%)					SISDR (dB)					PESQ
Method	5	0	-5	-10	-15	5	0	-5	-10	-15	5	0	-5	-10	-15
Noisy Audio	81.7	70.8	58.2	46.0	36.3	5.00	0.00	-5.00	-10.00	-15.02	1.21	1.10	1.06	1.06	1.08
Audio-only	92.2	87.0	78.3	64.4	47.0	13.28	10.08	6.47	1.99	-4.17	2.16	1.74	1.43	1.23	1.11
MUTUD w.o. pretrained TAME	92.7	87.7	79.3	65.5	48.1	13.45	10.32	6.72	2.27	-3.88	2.24	1.8	1.47	1.25	1.12
MUTUD	92.8	88.0	79.6	65.6	48.0	13.60	10.43	6.85	2.33	-3.86	2.23	1.80	1.47	1.25	1.12
Audiovisual	92.9	88.6	81.6	71.3	58.9	13.43	10.37	6.95	2.90	-2.23	2.25	1.85	1.53	1.30	1.16

Table 2 : Speech Enhancement performance comparison of different models for 5-BN test condition.

Figure 3 : Spectrograms for noisy audio, Audio-only , MUTUD, Audiovisual , and clean audio (rows) at SNR = 5, 0, $-5$ , $-10$ , $-15$ dB (columns).

Method	STOI (%)					SISDR (dB)					PESQ
Method	5	0	-5	-10	-15	5	0	-5	-10	-15	5	0	-5	-10	-15
Noisy Audio	82.6	72.4	60.5	48.8	38.9	5.00	0.00	-5.0	-10.03	-15.06	1.24	1.12	1.07	1.07	1.07
Audio-only	91.9	86.6	77.7	64.5	49.3	12.98	9.68	5.99	1.58	-4.14	2.12	1.73	1.44	1.24	1.13
MUTUD w.o. pretrained TAME	92.3	87.3	78.9	66.0	50.3	13.31	10.07	6.45	2.08	-3.65	2.20	1.79	1.49	1.27	1.14
MUTUD	92.3	87.3	78.9	66.0	50.5	13.29	10.04	6.43	2.06	-3.66	2.22	1.81	1.49	1.27	1.14
Audiovisual	92.6	88.2	81.1	71.1	59.9	13.23	10.10	6.65	2.72	-2.01	2.15	1.80	1.51	1.31	1.18

Table 3 : Speech Enhancement performance comparison for Backbone 2. A smaller backbone, that is, the Audio-only and the corresponding audiovisual models used here are smaller and weaker.

Method	STOI (%)					SISDR (dB)					PESQ
Method	5	0	-5	-10	-15	5	0	-5	-10	-15	5	0	-5	-10	-15
Noisy Audio	82.6	72.4	60.5	48.8	38.9	5.00	0.00	-5.00	-10.03	-15.06	1.24	1.12	1.07	1.07	1.07
Audio-only	91.8	85.9	76.5	64.0	48.8	11.67	8.62	5.17	1.08	-4.90	2.05	1.60	1.32	1.17	1.09
MUTUD	93.5	89.0	82.0	71.3	56.5	12.72	9.68	6.53	2.98	-2.04	2.40	1.94	1.58	1.33	1.18
Audiovisual	94.0	90.1	84.1	75.4	64.3	12.92	9.94	6.90	3.54	-0.86	2.51	2.03	1.65	1.39	1.21

Table 4 : Speech Enhancement performance comparison for Backbone 3. VisualVoice as backbone for Audio-only, audiovisual, and MUTUD.

Method	PESQ	SI-SDR	STOI	# Params
Noisy	1.58	9.07	0.92	-
FullSubnet ( Hao et al., 2021 )	2.77	17.29	0.96	5.6M
CleanUNet ( Kong et al., 2022 )	3.15	-	0.96	46.07M
Demucs ( Defossez et al., 2020 )	2.66	-	0.97	33.53M
NSNet ( Xia et al., 2020 )	2.15	15.61	0.94	5.1M
Audio-only (Ours)	2.32	16.24	0.94	2.98M

Table 5 : Performance comparison on DNS Challenge evaluation (no-reverb) set.

	Audio-only	Audio-only w. matched params	Audiovisual	MUTUD
# of Param.	2.978M	3.627M	15.736M	3.635M
MACs	1.381G	1.821G	9.324G	1.593G
Inference Time (ms)	98.1 $\pm$ 2.87	100.2 $\pm$ 2.50	206.0 $\pm$ 8.1	108.0 $\pm$ 2.1

Table 6 : Number of Parameters and Multiply Accumulate Operations (MACs) for all models.

Figure 4 : Cosine similarity (red) and

\ell_{2}

distance (blue) between video features and estimated video features, video and audio features, and estimated video and audio features for different SNRs.

Figure 5 : TSNE visualization of the estimated video features $\hat{F}_{v}$ , the actual video features ${F}_{v}$ , and the audio features ${F}_{a}$ for SNRs ranging from 5dB to -15dB.

Figure 6 : Visualization of the learned audio and video codebooks ( $\bm{C}^{a}$ and $\bm{C}^{v}$ ). The plots show the mean of each code in all $K(=4)$ codebooks.

Figure 7 : Visualization of the probability distribution $q$ (for each $k$ ) for a sample noisy audio frame.

# of codes	STOI ( $\%$ )					SISDR (dB)					PESQ
# of codes	5	0	-5	-10	-15	5	0	-5	-10	-15	5	0	-5	-10	-15
8	93.3	88.7	80.9	68.6	53.0	13.91	10.71	7.20	3.02	-2.39	2.36	1.91	1.56	1.32	1.17
16	93.4	88.8	81.1	68.9	53.3	13.98	10.81	7.30	3.10	-2.41	2.35	1.92	1.57	1.32	1.17
32	93.5	89.2	81.8	69.8	54.0	14.11	11.02	7.56	3.38	-2.19	2.36	1.92	1.57	1.32	1.17
64	93.6	89.1	81.6	69.6	54.0	14.11	11.00	7.51	3.31	-2.21	2.38	1.95	1.59	1.34	1.18

Table 7 : Ablation on different numbers of codes,

N

, in each MSC of TAME.

Method	WER (%) $\downarrow$
Method	5	0	-5	-10	-15
Audio-only	12.24	17.836	31.37	60.64	93.32
Audiovisual	5.26	7.088	11.01	21.12	36.56
MUTUD	11.71	16.299	24.99	44.07	73.56

Table 8 : MUTUD effectiveness in Audiovisual Speech Recognition (AVSR) task.

Method	mAP (%)
SyncNet ( Chung and Zisserman, 2017 )	82.1
TalkNet ( Tao et al., 2021 )	79.9
SPELL+ ( Min et al., 2022 )	85.9
Audiovisual EgoASD ( Huh et al., 2025 )	87.6
Video-only EgoASD	82.3
MUTUD EgoASD	86.5

Table 9 : MUTUD effectiveness in Audiovisual Active Speaker Detection (AV-ASD) task.

Explore similar work

Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos
May 25, 2026Bonan Ding, Umair Nawaz, Ufaq Khan +5Fine-Grained Video Understanding Cross-Modal Attention
Mohamed bin Zayed University of Artificial Intelligence · Tianjin University · Chongqing University +1
Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition
Jul 10, 2026Xugang Lu, Peng Shen, Yu Tsao +1Cross-Modal Alignment Latent Representation Alignment
National Institute of Information and Communications Technology, Kyoto, Japan · Research Center for Information Technology Innovation, Academia Sinica, Taiwan
AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers
Apr 17, 2026Edson Araujo, Saurabhchand Bhati, M. Jehanzeb Mirza +5Audio-Visual Reasoning Multimodal Reasoning
University of Tübingen, Germany · MIT, Cambridge MA, USA · IBM Research, USA +2

May 25, 2026, cs.CVJ/K move · Enter open · S save

Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos

Bonan Ding, Umair Nawaz, Ufaq Khan, Abdelrahman M. Shaker+4

Mohamed bin Zayed University of Artificial Intelligence · Tianjin University · Chongqing University+1

Fine-Grained Video Understanding Cross-Modal Attention Modalities

Open paper AskPDF arXiv

Pre-trained video large language models excel at visual reasoning. However, they struggle when videos arrive with auxiliary streams, such as audio, depth map, or dense temporal evidence. In such a scenario, uniform fusion induces modality interference, allowing irrelevant channels to distract the model. To address this issue, we present a unified multimodal video understanding framework, named UniMVU, that performs instruction-aware fusion across video, audio, depth map, or any other modality inputs via two levels of dynamic gating: inner-modality gates emphasize salient regions within each modality, whereas modality-level gates re-weight whole streams; both are conditioned on the text instruction to adaptively balance modality importance. Our UniMVU combines cross-modal self-attention with instruction-driven inner-modality gating module and a modality-level gating module with control token; for time-aligned streams we further adopt a fast-to-slow fusion scheme that reduces redundancy. Across six benchmarks (AVQA, AVSD, Music-AVQA, ScanQA, SQA3D and MVBench), our UniMVU achieves consistent gains over static-fusion baselines achieving gains as high as 13.5 in terms of CIDEr metric. Further, our analysis shows that the gating mechanism aligns with the human-interpretable modality relevance, and ablations show the contributions of inner-modality and modality-level gating. Our UniMVU provides a simple, unified recipe for instruction-aware multimodal video understanding that scales to diverse modalities without hand-crafted fusion rules.

Abstract

Explore similar work

Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos

Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition

AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers