Organizations: Tokyo University of Agriculture and Technology, Tokyo 184-8588, Japan · Acoustics Research Institute, Austrian Academy of Sciences, Wohllebengasse 12–14, 1040 Vienna, Austria
Phase vocoder-based time-stretching is a widely used technique for the time-scale modification of audio signals. However, conventional implementations suffer from ``percussion smearing,'' a well-known artifact that significantly degrades the quality of percussive components. We attribute this artifact to a fundamental time-scale mismatch between the temporally smeared magnitude spectrogram and the localized, newly generated phase. To address this, we propose SELEBI, a signal-adaptive phase vocoder algorithm that significantly reduces percussion smearing while preserving stability and the perfect reconstruction property. Unlike conventional methods that rely on heuristic processing or component separation, our approach leverages the nonstationary Gabor transform. By dynamically adapting analysis window lengths to assign short windows to intervals containing significant energy associated with percussive components, we directly compute a temporally localized magnitude spectrogram from the time-domain signal. This approach ensures greater consistency between the temporal structures of the magnitude and phase. Furthermore, the perfect reconstruction property of the nonstationary Gabor transform guarantees stable, high-fidelity signal synthesis, in contrast to previous heuristic approaches. Experimental results demonstrate that the proposed method effectively mitigates percussion smearing and yields natural sound quality.
Figures & tables
Fig. 1: Block diagrams of the basic method, PV with identity phase locking [ 15 ] (left), and the proposed method (right). By leveraging NSDGT , the proposed method synthesizes the target signal from T-F representations in which both the magnitude and phase spectrograms of percussive components are highly concentrated in the time direction.
Fig. 2: Comparison of windows and spectrograms for the DGT and NSDGT . The upper boxes illustrate the window shift in the time domain, where representative DGT windows are highlighted in black for clarity. The bottom boxes display the corresponding spectrograms for (a) DGT with a long window, (b) DGT with a short window, and (c) the NSDGT .
Fig. 3: Conceptual illustration of time-directional spectrogram “squeezing.” (a) The conventional PV-based method using DGT. (b) The proposed method utilizing transient concentration. The top row displays the input time-domain signal (amplitude vs. time) and the analysis window functions. The second and third rows schematically represent the magnitude spectrogram and the corresponding generated phase, respectively. The bottom row shows the synthesized time-stretched signal. In these schematic representations, the percussive component is colored red, and the windows capturing this component are emphasized (non-percussive components are omitted in this row for clarity). In the spectrograms, the red area highlights the percussive component, while the blue and green areas represent the magnitude and phase of the other components, respectively. Because the percussive interval is maintained at its original time-scale, it appears compressed relative to the new, stretched time axis (illustrated below the spectrograms). The bottom panel details the synthesis of the percussive component across the new time frames. The markers ⋆1 and ⋆2 highlight the key innovations of the proposed method: shortening the window length and reducing the number of time frames, respectively.
Fig. 4: The flow of the proposed method. The left column details the NSDGT parameter calculation, while the right column illustrates the subsequent processing steps.
Fig. 5: Example of the computation of the compression rate. From top left to bottom right, (a) the magnitude spectrogram ∣X∣ , (b) the enhancement mask, (c) the enhanced spectrogram ∣Xp∣ , (d) the frequency-directional sum of ∣X∣ , (e) the frequency-directional sum of ∣Xp∣ , and (f) the compression rate r (the detected peaks are plotted in yellow). The mask in (b) is colored in white where M(X,Φmix)[m,n]=1 .
Fig. 6: Concept of the window length adjustment in the proposed method. The top row shows the target window length, while the bottom row shows the corresponding analysis windows. The colors in the top row match those in the bottom row. Stages (a)–(e) illustrate the window behavior as it encounters an impulse.
Fig. 7: Effect of compression rate r on window length (left) and the resulting impulse magnitude spectrogram (right). The left panels plot the window lengths (vertical axis) versus time (horizontal axis), where the original window length is shown in gray, and the modified window length is shown in blue. The right panels show the corresponding magnitude spectrograms (frequency vs. time) of an impulse, obtained using the window functions from the left. From top to bottom, the rows show compression rates of r=1.0,0.8,0.4,and 0 , chosen for visibility.
Fig. 8: Example of adaptive window length adjustment and the resulting NSDGT spectrogram for a bongo signal. The panels display, from top to bottom: the original waveform, the adaptive window length at each time index, the resulting NSDGT spectrogram, and the standard DGT spectrogram. For clarity, the middle panel is color-coded to correspond with the adjustment stages shown in Fig. 6 : non-percussive regions (gray; (a), (e)), pre-pulse (red; (b)), pulse center (green; (c)), and post-pulse (blue; (d)).
Type of transitions
Hop size settings
Before
Transition
After
ast
aend
(A)
(i)
(iii)
(ii)
a
ak
(B)
(ii)
(iv)
(i)
ak
a
(C)
(ii)
(iv)
(iii)
ak
a˘k
(D)
(iv)
(iii)
(ii)
a˘k
ak
(E)
(ii)
(iii) , (iv)
(ii)
ak
ak+1
TABLE I: Hop size definitions for each transition type.
Fig. 9: Example of smooth hop size adjustment and impulse observation (corresponding to Sec. IV-D ). The left column shows a single impulse, while the right column shows two adjacent impulses of differing amplitudes. Note that the small impulse on the right is embedded in Gaussian noise to simulate a less-percussive sound. (a) plots the adaptive window length (stems) at each hop position, with gray line indicating the positions of a large, fixed hop size; (b) illustrates the corresponding shifting windows; and (c) shows the observed amplitude, with the DGT-based magnitude in gray for comparison. The window coloring scheme is the same as in Fig. 8 .
Fig. 10: Conceptual diagram of a bounded-delay, online implementation. The left panel illustrates the pre-analysis of a target percussive component (red) using the DGT. The left edge represents the current time position, and the analysis windows corresponding to the required look-ahead frames are highlighted in blue. The top-right panel depicts the onset detection and compression rate calculation (corresponding to Fig. 5 (f)), displaying the magnitude ratio in gray and the detected peak in yellow. The bottom-right panel outlines the subsequent procedures.
PV
PVDR
PVHP
DIALGA
DIALGA
SELEBI
SELEBI
(PV)
(PVDR)
(PV)
(PVDR)
α=2
Impulse
0.2357
0.2402
0.6993
0.1009
0.1110
0.2326
0.2326
Sinusoid + Impulse
0.0113
0.0091
0.0130
0.0090
0.0092
0.0106
0.0051
Harmonic Sinusoid + Impulse
0.0129
0.0107
0.0139
0.0136
0.0107
0.0128
0.0098
Transient
0.5806
0.5666
0.5924
0.5659
0.5647
0.5791
0.5552
Sinusoid + Transient
0.2506
0.2446
0.4244
0.2455
0.2445
0.2521
0.2414
TABLE II: Comparison of error measurements for synthetic signals (Sec. VI-A ). The top and bottom rows show the results of 2× and 4× stretching, respectively. “DIALGA (PVDR)” denotes DIALGA with PVDR phase generation, and similarly for other methods. The best and second-best scores are in bold with blue and light blue backgrounds, respectively.
Fig. 11: An impulse stretched 4× . The leftmost column shows the ground truth (blue), while the remaining columns display the stretched signal for each comparison method (red). The figure is divided into two rows: the top row shows a magnified view of the impulse, and the bottom row displays the relative error spectrogram (log scale). In the top row, for visual clarity, the ground truth is plotted as a gray dashed line. In the error spectrogram (bottom), brighter colors indicate larger errors.
Fig. 12: Example of a transient signal stretched 4× . The leftmost column shows the ground truth (blue), while the remaining columns display the stretched signal for each comparison method (red). The figure is divided into three rows: the top row shows the entire time waveform, the middle row shows a magnified view of the waveform onset, and the bottom row displays the relative error spectrogram (log scale). The horizontal axis for all panels is time (seconds). In the error spectrogram (bottom), brighter color indicates greater error.
Fig. 13: Examples of the stretched signals of “CastanetsViolin” and “Glockenspiel”. The leftmost column shows the original signal, while the remaining columns display the stretched signal for each comparison method. The horizontal axis for all panels is time (seconds). Transient smearing in classical PV is apparent in both examples, whereas PVDR and DIALGA demonstrate different, characteristic artifacts at percussive events in the synthesized signals: PVDR exhibits slight smearing and both methods result in somewhat reduced amplitude of attacks. Additionally, in the “Glockenspiel” excerpt PVDR and DIALGA show a separation of the transient attack portion from the more harmonic decay segment of each played note. In contrast, SELEBI yields a more natural waveform that preserves attacks and harmonic signal components and more closely resembles the original signal.
Fig. 14: Screenshot of the UI of the code used for subjective evaluation.
Overall Quality
Sharpness of Attacks
Clarity of Tonal Components
PV
PVDR
DIALGA
SELEBI
PV
PVDR
DIALGA
SELEBI
PV
PVDR
DIALGA
SELEBI
α=2
Bongo
14.98
82.48
73.28
94.18
19.62
85.92
72.88
92.31
15.71
77.22
69.11
85.72
CastanetsViolin
28.77
72.73
80.69
78.78
10.01
69.17
71.35
75.38
50.33
72.27
81.55
78.32
DrumSolo
17.43
56.48
58.07
64.49
10.56
54.93
54.49
59.88
22.74
57.35
51.17
57.69
Glockenspiel
33.57
67.24
63.93
74.21
29.48
63.26
63.73
73.26
40.28
67.59
66.47
74.66
Jazz
33.57
68.26
63.71
68.58
28.56
62.05
61.80
68.65
33.92
67.63
64.59
62.09
TABLE III: Comparison of the mean scores across three metrics: Overall Quality, Sharpness of Attacks, and Clarity of Tonal Components. The best and second-best scores are in bold with blue and light blue backgrounds, respectively.
Time-frequency representations, such as the short-time Fourier transform (STFT), are fundamental tools for analyzing non-stationary signals. However, their ability to achieve sharp localization in both time and frequency is inherently limited by the Gabor-Heisenberg uncertainty principle. In this paper, we address this limitation by introducing a method to generate super-resolution spectrograms through the fusion of two or more spectrograms with varying resolutions. Specifically, we compute the super-resolution spectrogram as the barycenter of input spectrograms using optimal transport (OT) divergences. Unlike existing fusion approaches, our method does not require the input spectrograms to share the same time-frequency grid. Instead, the input spectrograms can be computed using any STFT parameters, and the resulting super-resolution spectrogram can be defined on an arbitrary user-specified grid. We explore various OT divergences based on different transportation costs. Notably, we introduce a novel transportation cost that preserves time-frequency geometry while significantly reducing computational complexity compared to standard Wasserstein barycenters. We adopt the unbalanced OT framework and derive a new block majorization-minimization algorithm for efficient barycenter computation. We validate the proposed method on controlled synthetic signals and recorded speech using both quantitative and qualitative evaluations. The results show that our approach combines the best localization properties of the input spectrograms and outperforms an unsupervised state-of-the-art fusion method.
Differentiable Digital Signal Processing (DDSP) pipelines for voice conversion rely on subtractive synthesis, where a periodic excitation signal is shaped by a learned spectral envelope to reconstruct the target voice. In DDSP-QbE, the excitation is generated via phase accumulation, producing a sawtooth-like waveform whose abrupt discontinuities introduce aliasing artefacts that manifest perceptually as buzziness and spectral distortion, particularly at higher fundamental frequencies. We propose two targeted improvements to the excitation stage of the DDSP-QbE subtractive synthesizer. First, we incorporate explicit voicing detection to gate the harmonic excitation, suppressing the periodic component in unvoiced regions and replacing it with filtered noise, thereby avoiding aliased harmonic content where it is most perceptually disruptive. Second, we apply Polynomial Band-Limited Step (PolyBLEP) correction to the phase-accumulated oscillator, substituting the hard waveform discontinuity at each phase wrap with a smooth polynomial residual that cancels alias-generating components without oversampling or spectral truncation. Together, these modifications yield a cleaner harmonic roll-off, reduced high-frequency artefacts, and improved perceptual naturalness, as measured by MOS. The proposed approach is lightweight, differentiable, and integrates seamlessly into the existing DDSP-QbE training pipeline with no additional learnable parameters.
Suhita Ghosh, Yamini Sinha, Sebastian Stober
Artificial Intelligence Lab, Otto-von-Guericke University Magdeburg, Germany · MDS-IIKT Otto-von-Guericke University Magdeburg, Germany
A vocoder is a pivotal component of modern text-to-speech (TTS) systems. Despite the significant progress of neural network-based vocoders, accurate phase reconstruction remains the main challenge limiting both audio quality and modeling efficiency. We introduce PhaseGAN, a lightweight vocoder that addresses this limitation through a "mel → Amplitude → Phase" reconstruction pipeline. By reconstructing amplitude and phase spectra via distinct methodologies, the proposed PhaseGAN outperforms state-of-the-art baselines while utilizing fewer model parameters and reduced computational requirements. The compact version generates high-fidelity audio with approximately 500K parameters and 1 GMAC computational load, making it highly suitable for real-time applications on edge devices. In addition, our approach exhibits exceptional musical audio synthesis capabilities despite no training on musical data, illustrating unprecedented cross-domain generalization. See https://github.com/phasegan/phasegan-audio-demo for demos of our work.
Wenzheng Zhang, Xueliang Zhang, Shulin He +7
College of Computer Science, Inner Mongolia University, China