End-to-End Historical Music Restoration in Latent Space

Steven Cho; Junghyun Koo; Raphael Lafargue; Tushar Dhyani; Eloi Moliner; Yuki Mitsufuji

eess.ASSep 30, 2026

End-to-End Historical Music Restoration in Latent Space

Authors: Steven Cho, Junghyun Koo, Raphael Lafargue, Tushar Dhyani, Eloi Moliner, Yuki Mitsufuji

Organizations: ETH Zurich · Sony Corporate Technology Center America, Inc. · Sony Europe Ltd. · Acoustics Lab, DICE, Aalto University · Sony Group Corporation

Abstract

Historical music restoration (HMR) has almost exclusively focused on constrained problems such as Super-Resolution or the restoration of solo pieces, under-exploring the general task of restoring orchestral historical music, which has multiple instruments. This under-exploration is largely because the HMR domain, early-20th-century recordings, has no pre-degradation ground-truth pairs, making the restoration task unsupervised and more challenging. This paper presents a supervised end-to-end orchestral HMR benchmark by exploring both the synthetic degradation functions and the end-to-end generative deep-learning restoration methods. We simulate the historical recording degradation chain more faithfully than prior work, which makes orchestral restoration into a tractable supervised problem. A latent flow-matching model trained on the resulting synthetic pairs outperforms existing HMR baselines on intrusive, non-intrusive, and subjective evaluations. We also curate and release a 9.3-hour license-free, unpaired, historical classical-music test set, along with code and audio demos.

Figures & tables

Figure 1 : Overview of the proposed historical music restoration pipeline $\mathcal{R}_{\theta,\mathrm{CFM}}$ .

	MERT [ 10 ]		Fx-Enc. [ 33 ]		SAME-L [ 27 ]
Degradation	Full Orch.	Light Orch.	Full Orch.	Light Orch.	Full Orch.	Light Orch.
Clean	5.39	5.40	5.43	5.49	5.33	5.45
Gaussian Noise	5.18	5.13	4.71	5.08	5.40	5.44
Low-Pass + Gaussian Noise	5.31	5.46	5.42	5.49	5.24	5.37
Gramophone Noise	3.90	4.42	3.86	4.63	4.57	4.82
WH + Gramophone Noise (Ours)	4.23	4.55	4.42	4.99	4.54	4.95

Table 1 : MAD (

\downarrow

) between degradation ablations embeddings on FOS and historical unpaired test set. The final row directly compares the two subsets of the unpaired test set.

Codec	Paired Clean	Paired 5-Stage	Full Orchestra	Light Orchestra
CoDiCodec [ 28 ]	$1.08\pm 0.45$	$0.25\pm 0.27$	$1.37\pm 0.68$	$1.24\pm 0.69$
SAO [ 7 ]	$0.39\pm 0.19$	$0.12\pm 0.17$	$0.80\pm 0.50$	$0.88\pm 0.58$
DAC [ 14 ]	$\underline{0.22\pm 0.10}$	$\underline{0.08\pm 0.15}$	$\underline{0.72\pm 0.57}$	$\underline{0.87\pm 0.63}$
SAME-L	$\mathbf{0.11\pm 0.08}$	$\mathbf{0.05\pm 0.12}$	$\mathbf{0.62\pm 0.57}$	$\mathbf{0.81\pm 0.62}$

Table 2 : Autoencoder reconstruction Mel-MSE with equal-window aggregation. Waveforms were encoded and decoded by the corresponding autoencoder and compared against the input using Mel-MSE

Method	Params	Historical Unpaired Test Set				Synthetic Paired Test Set
		AA-PQ ( $\uparrow$ )		MOS-P ( $\uparrow$ )	MOS-Q ( $\uparrow$ )	AA-PQ ( $\uparrow$ )	Mel MSE ( $\downarrow$ )	CLAP Cos. ( $\uparrow$ )	MAD ( $\downarrow$ )
		Full-Orch.	Light-Orch.	MOS-P ( $\uparrow$ )	MOS-Q ( $\uparrow$ )	AA-PQ ( $\uparrow$ )	Mel MSE ( $\downarrow$ )	CLAP Cos. ( $\uparrow$ )	MAD ( $\downarrow$ )
Unprocessed input	–	4.94	5.00	–	1.17	5.74	3.94	0.44	5.03
Ground truth	–	–	–	–	4.44	7.44	0.00	1.00	0.00
BEHM-GAN	82M	4.87	4.93	–	–	6.05	5.33	0.45	4.94
BABE2	40M	5.88	5.87	3.70	3.52	6.35	4.99	0.56	5.46

Table 3 : Restoration performance on objective and subjective evaluations.

Panel of Figure 2 — Figure 2 : MOS-Q and MOS-P means, 95% confidence intervals, and non-outlier ranges

Explore similar work

Music Restoration via Latent Operator Optimization and Diffusion Model Priors
Aug 3, 2026Michal Švento, Eloi Moliner, Valtteri Kallinen +3Audio Editing Signal Reconstruction
Dept. of Telecommunications, Brno University of Technology, Czech Republic · Acoustic Lab, Dept. of Information and Communications Engineering, Aalto University, Finland
On the Geometry of Music Bandwidth Extension in Latent Spaces of Audio Codecs
Aug 4, 2026Hendrik Vincent Koops, Hao Hao Tan, Elio QuintonNeural Codecs Audio Editing
Music & Audio Machine Learning Lab · Universal Music Group, London, U.K.
OmniVR: Audio-Video Conditional Generation for Archival Footage Restoration
Aug 4, 2026Xin Lu, Zihao Fan, Jie Huang +4Video Restoration Audio-Video Generation
University of Science and Technology of China · JD Explore Academy

Aug 3, 2026, eess.ASJ/K move · Enter open · S save

Music Restoration via Latent Operator Optimization and Diffusion Model Priors

Michal Švento, Eloi Moliner, Valtteri Kallinen, Lauri Juvela+2

Dept. of Telecommunications, Brno University of Technology, Czech Republic · Acoustic Lab, Dept. of Information and Communications Engineering, Aalto University, Finland

Audio Editing Signal Reconstruction Autoencoder Architectures Probabilistic Operator Learning

Open paper AskPDF arXiv

Music restoration seeks to recover a clean signal from an observed recording degraded by an unknown effect, distortion, or corruption. Existing systems often rely on paired training data and distortion-specific supervision, which limits their use when the forward process is not known in advance. We propose LOUDAR (Latent-space Optimization of Unknown Distortion for Audio Restoration) a general-purpose restoration method that operates in the latent space of a pretrained audio autoencoder and models the unknown distortion as a learnable latent operator. At inference time, LOUDAR alternates between estimating the clean latent variable and updating the latent operator parameters. An unconditional latent diffusion model provides a prior over clean audio and regularizes this inference by steering the latent estimate toward the manifold of clean recordings. Because the degradation model is adapted per input, the approach is broadly applicable across diverse restoration problems. We evaluate LOUDAR on singing voice effect removal and restoration, as well as guitar distortion removal, and show that it consistently improves over degraded inputs and is competitive with supervised and unsupervised baselines in waveform and latent domains.

Abstract

Explore similar work

Music Restoration via Latent Operator Optimization and Diffusion Model Priors

On the Geometry of Music Bandwidth Extension in Latent Spaces of Audio Codecs

OmniVR: Audio-Video Conditional Generation for Archival Footage Restoration