Blind Video Denoising for Vocal Tract rtMRI using a Self-Supervised Ensemble

Julian Andres Hernandez-Potes; Tomás Arias-Vergara; Daiqi Liu; Andreas Maier; Paula Andrea Perez-Toro

eess.IVOct 29, 2026

Blind Video Denoising for Vocal Tract rtMRI using a Self-Supervised Ensemble

Authors: Julian Andres Hernandez-Potes, Tomás Arias-Vergara, Daiqi Liu, Andreas Maier, Paula Andrea Perez-Toro

Organizations: German Cancer Research Center (DKFZ), 69120 Heidelberg, Germany · Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058 Erlangen, Germany · Technische Universität München, 80333 Munich, Germany

Abstract

Real-time magnetic resonance imaging of the vocal tract makes speech articulation directly observable, but the trade-off between temporal resolution and signal quality introduces substantial noise and obscures fine articulatory detail. Since clean references cannot be acquired for a moving vocal tract, supervised denoising approaches are not applicable in this setting. We propose a self-supervised denoising framework that combines spatiotemporal super-resolution learning with pseudo-supervision derived from a variance-stabilized non-local means filter. A dual-model fusion strategy balances noise suppression and preservation of high-frequency articulatory structure. Experiments on the USC-75 multi-speaker dataset under natural noise conditions show that the ensemble raises the signal-to-noise ratio by 11.9 dB on unseen subjects while attaining more high-frequency articulatory content in the frames. Among tuned implementations of anisotropic diffusion, block-matching filtering, non-local means and pre-smoothed non-local means, every configuration that achieves comparable noise suppression does so at the cost of high-frequency content or of structural fidelity. Under severe added Gaussian, Rayleigh and Rician noise, peak signal-to-noise ratio improves by 7.6 to 11.8 dB and structural similarity roughly triples. An ablation shows that the variance-stabilized pseudo-supervision is an essential element for denoising, and that the effect is attributable to the hard thresholding contained in the stabilization transform.

Figures & tables

Fig. 1: Training framework for our denoising network. The noisy rtMRI video is cut into overlapping clips; the spatial sub-sampling block retains one pixel from every $2\times 2$ cell, which forms the network input, and marks the remaining three quarters in the mask $M$ . The network, detailed in Sec. III-C , is optimized against a PSNLM-filtered version $y^{p}$ of the same clip with a composite loss that preserves edges and limits over-blurring. Only the fidelity term is restricted to $M$ . Eq. ( 5 ) defines all three terms of the loss function.

Fig. 2: One frame of the unseen sub075 clip. The top row is the full frame, with the signal region that forms the numerator of Eq. ( 7 ) (red) and the background region-of-interest that forms its denominator (blue). The bottom row displays a magnified view of the full frame (yellow); all printed metrics are 50-frame averages over the clip, not only the displayed frame. ADF quantizes the soft tissue into piecewise-constant regions, BM4D-3D removes the noise together with the textures, and $\mathrm{PSNLM}_{\mathrm{T}}$ preserves the anatomy but flattens the background, while $\mathrm{PSNLM}_{\mathrm{V}}$ changes the signal much less. The proposed ensemble is the only denoised output whose TV exceeds that of the input while improving SNR and preserving structural fidelity.

Method	rSNR $\uparrow$	TV	SSIM $\uparrow$
Original (noisy)	21.5 $\pm$ 2.4	9.32 $\pm$ 0.99	1.00 $\pm$ 0.00
ADF	28.2 $\pm$ 4.6	8.98 $\pm$ 0.94	0.89 $\pm$ 0.02
BM4D (3-D)	26.7 $\pm$ 4.1	7.05 $\pm$ 0.64	0.92 $\pm$ 0.02
NLM (3-D)	31.6 $\pm$ 5.9	8.31 $\pm$ 0.71	0.95 $\pm$ 0.04
PSNLM T [pseudo-target]	35.7 $\pm$ 9.7	9.96 $\pm$ 1.12	0.57 $\pm$ 0.04
PSNLM T [max rSNR]	40.0 $\pm$ 9.9	5.81 $\pm$ 0.70	0.64 $\pm$ 0.04

TABLE I: Reference-free quality averaged over the 17 test subjects. [pseudo-target] is the setting that generates

y^{p}

, [max

r

SNR] the unconstrained optimum of the search and the unmarked rows its

\mathrm{SSIM}\geq 0.90

optimum.

f_{\mathrm{HF}}

and

f_{\mathrm{LF}}

: high- and low-frequency components of the network respectively.

Fig. 3: Noise suppression against detail retention over the 17 test subjects. Each point is a method minus the noisy input of the same subject, and the bars are the standard deviation of that paired difference across subjects. All PSNLM points are the analytical filter alone, without the network.

Fig. 4: The ensemble at $w=0.1$ on the original videos after Gaussian, Rayleigh and Rician noise is added at $\sigma=0.2$ . Top: the corrupted input. Bottom: the output. The reference for the three metrics is the uncorrupted USC-75 frame on the left, which carries the scanner’s own Rician noise.

Fig. 5: Ablation of the pseudo-supervision with both ensemble members re-trained under a common setting except for the regression target. rSNR of

f_{\mathrm{HF}}

f_{\mathrm{LF}}

and their fusion, trained against the thresholded pseudo-target

y^{p}_{\mathrm{T}}

, the Rician-VST pseudo-target

y^{p}_{\mathrm{V}}

, and the raw noisy clip, with the input’s own rSNR as the dashed line.

Pseudo-target	rSNR $\uparrow$	$\Delta$	SSIM
Noisy input	22.4	n/a	1.00
None (raw noisy clip)	22.3	$-0.1$	0.97
Anscombe root only	22.7	$+0.3$	0.88
$F_{\mathrm{V}}$ , Rician VST	28.9	$+6.5$	0.92
Threshold only	33.0	$+10.6$	0.88
$F_{\mathrm{T}}$ (threshold $+$ root)	35.6	$+13.2$	0.86

TABLE II: Contribution of each component of the pseudo-target transform. Each row retrains

f_{\mathrm{HF}}

against the threshold component, the Anscombe root component, or their combination

F_{\mathrm{T}}

, compared with the Rician-stabilized transform

F_{\mathrm{V}}

and with no pseudo-target.

Image Restoration

Image restoration uses computational methods to recover useful visual content from noisy, blurred, low-resolution, or otherwise degraded images.

See all 375 papers

Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI
May 18, 2026Daiqi Liu, Lukas Mulzer, Md Hasan +11Cross-Modality Medical Image Segmentation Magnetic Resonance Imaging
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Smart Imaging Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Harvard Medical School / Massachusetts General Hospital, Boston, USA +3
SNOAR: Self-training with Noise filtering, Observation adding, and Remixing for Speech Denoising Without Clean References
Oct 14, 2026Samuele Cornell, Matthew MaciejewskiSpeech Enhancement
Carnegie Mellon University, USA · Johns Hopkins University, USA
Arti-JEPA: Adapting Video World Model to Real-Time MRI of the Vocal Tract for Speech-Production Analysis
Sep 9, 2026Hong Nguyen, Sean Foley, Christina Hagedorn +4Unsupervised Domain Adaptation Joint-Embedding Predictive Architecture
Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA, USA · Department of English, College of Staten Island, City University of New York, New York, NY, USA · Department of Linguistics, University of Potsdam, Germany +1

May 18, 2026J/K move · Enter open · S save

Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI

Daiqi Liu, Lukas Mulzer, Md Hasan, Nyvenn de Castro+10

Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Smart Imaging Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Harvard Medical School / Massachusetts General Hospital, Boston, USA+3

Cross-Modality Medical Image Segmentation Magnetic Resonance Imaging Medical Image Segmentation Multimodal Fusion

Open paper AskPDF arXiv

Segmenting vocal tract articulators in real-time MRI (rtMRI) is a challenging dynamic image segmentation problem characterized by low contrast, rapid motion, and limited spatial resolution. However, while rtMRI acquisitions may provide synchronized acoustic signals, existing methods discard this information, and the few multimodal approaches that incorporate audio cannot be deployed when audio is unavailable. We propose a three-stage framework that leverages acoustic and phonological supervision during training while requiring only the rtMRI image at inference: phonological representations are converted into spatial bounding-box priors for articulator localization, visual and acoustic encoders are aligned via dual-level cross-modal contrastive pretraining, and the learned representations are fused through a cross-attention decoder, effectively transferring multimodal knowledge into a single-modality inference pipeline. Evaluated on 75-Speaker~Annot-16 and USC-TIMIT datasets, our method outperforms existing unimodal and multimodal methods, demonstrating that multimodal supervision provides transferable benefits for precise and clinically deployable vocal tract segmentation.

Abstract

Image Restoration

Explore similar work

Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI

SNOAR: Self-training with Noise filtering, Observation adding, and Remixing for Speech Denoising Without Clean References

Arti-JEPA: Adapting Video World Model to Real-Time MRI of the Vocal Tract for Speech-Production Analysis