SELEBI: Percussion-aware Time Stretching via Selective Magnitude Spectrogram Compression by Nonstationary Gabor Transform

Natsuki Akaishi; Nicki Holighaus; Kohei Yatabe

eess.ASFeb 18, 2026

SELEBI: Percussion-aware Time Stretching via Selective Magnitude Spectrogram Compression by Nonstationary Gabor Transform

Authors: Natsuki Akaishi, Nicki Holighaus, Kohei Yatabe

Organizations: Tokyo University of Agriculture and Technology, Tokyo 184-8588, Japan · Acoustics Research Institute, Austrian Academy of Sciences, Wohllebengasse 12–14, 1040 Vienna, Austria

Abstract

Phase vocoder-based time-stretching is a widely used technique for the time-scale modification of audio signals. However, conventional implementations suffer from ``percussion smearing,'' a well-known artifact that significantly degrades the quality of percussive components. We attribute this artifact to a fundamental time-scale mismatch between the temporally smeared magnitude spectrogram and the localized, newly generated phase. To address this, we propose SELEBI, a signal-adaptive phase vocoder algorithm that significantly reduces percussion smearing while preserving stability and the perfect reconstruction property. Unlike conventional methods that rely on heuristic processing or component separation, our approach leverages the nonstationary Gabor transform. By dynamically adapting analysis window lengths to assign short windows to intervals containing significant energy associated with percussive components, we directly compute a temporally localized magnitude spectrogram from the time-domain signal. This approach ensures greater consistency between the temporal structures of the magnitude and phase. Furthermore, the perfect reconstruction property of the nonstationary Gabor transform guarantees stable, high-fidelity signal synthesis, in contrast to previous heuristic approaches. Experimental results demonstrate that the proposed method effectively mitigates percussion smearing and yields natural sound quality.

Figures & tables

Fig. 1: Block diagrams of the basic method, PV with identity phase locking [ 15 ] (left), and the proposed method (right). By leveraging NSDGT , the proposed method synthesizes the target signal from T-F representations in which both the magnitude and phase spectrograms of percussive components are highly concentrated in the time direction.

Fig. 2: Comparison of windows and spectrograms for the DGT and NSDGT . The upper boxes illustrate the window shift in the time domain, where representative DGT windows are highlighted in black for clarity. The bottom boxes display the corresponding spectrograms for (a) DGT with a long window, (b) DGT with a short window, and (c) the NSDGT .

Fig. 3: Conceptual illustration of time-directional spectrogram “squeezing.” (a) The conventional PV-based method using DGT. (b) The proposed method utilizing transient concentration. The top row displays the input time-domain signal (amplitude vs. time) and the analysis window functions. The second and third rows schematically represent the magnitude spectrogram and the corresponding generated phase, respectively. The bottom row shows the synthesized time-stretched signal. In these schematic representations, the percussive component is colored red, and the windows capturing this component are emphasized (non-percussive components are omitted in this row for clarity). In the spectrograms, the red area highlights the percussive component, while the blue and green areas represent the magnitude and phase of the other components, respectively. Because the percussive interval is maintained at its original time-scale, it appears compressed relative to the new, stretched time axis (illustrated below the spectrograms). The bottom panel details the synthesis of the percussive component across the new time frames. The markers $\star 1$ and $\star 2$ highlight the key innovations of the proposed method: shortening the window length and reducing the number of time frames, respectively.

Fig. 4: The flow of the proposed method. The left column details the NSDGT parameter calculation, while the right column illustrates the subsequent processing steps.

Fig. 5: Example of the computation of the compression rate. From top left to bottom right, (a) the magnitude spectrogram $|\mathbf{X}|$ , (b) the enhancement mask, (c) the enhanced spectrogram $|\mathbf{X}_{\text{p}}|$ , (d) the frequency-directional sum of $|\mathbf{X}|$ , (e) the frequency-directional sum of $|\mathbf{X}_{\text{p}}|$ , and (f) the compression rate $\mathbf{r}$ (the detected peaks are plotted in yellow). The mask in (b) is colored in white where $\mathcal{M}(\mathbf{X},\mathbf{\Phi}_{\textrm{mix}})[m,n]=1$ .

Fig. 6: Concept of the window length adjustment in the proposed method. The top row shows the target window length, while the bottom row shows the corresponding analysis windows. The colors in the top row match those in the bottom row. Stages (a)–(e) illustrate the window behavior as it encounters an impulse.

Fig. 7: Effect of compression rate $r$ on window length (left) and the resulting impulse magnitude spectrogram (right). The left panels plot the window lengths (vertical axis) versus time (horizontal axis), where the original window length is shown in gray, and the modified window length is shown in blue. The right panels show the corresponding magnitude spectrograms (frequency vs. time) of an impulse, obtained using the window functions from the left. From top to bottom, the rows show compression rates of $r=1.0,0.8,0.4,\text{and }0$ , chosen for visibility.

Fig. 8: Example of adaptive window length adjustment and the resulting NSDGT spectrogram for a bongo signal. The panels display, from top to bottom: the original waveform, the adaptive window length at each time index, the resulting NSDGT spectrogram, and the standard DGT spectrogram. For clarity, the middle panel is color-coded to correspond with the adjustment stages shown in Fig. 6 : non-percussive regions (gray; (a), (e)), pre-pulse (red; (b)), pulse center (green; (c)), and post-pulse (blue; (d)).

	Type of transitions			Hop size settings
	Before	Transition	After	$a_{\text{st}}$	$a_{\text{end}}$
(A)	(i)	(iii)	(ii)	$a$	$\widehat{a}_{k}$
(B)	(ii)	(iv)	(i)	$\widehat{a}_{k}$	$a$
(C)	(ii)	(iv)	(iii)	$\widehat{a}_{k}$	$\breve{a}_{k}$
(D)	(iv)	(iii)	(ii)	$\breve{a}_{k}$	$\widehat{a}_{k}$
(E)	(ii)	(iii) , (iv)	(ii)	$\widehat{a}_{k}$	$\widehat{a}_{k+1}$

TABLE I: Hop size definitions for each transition type.

Fig. 9: Example of smooth hop size adjustment and impulse observation (corresponding to Sec. IV-D ). The left column shows a single impulse, while the right column shows two adjacent impulses of differing amplitudes. Note that the small impulse on the right is embedded in Gaussian noise to simulate a less-percussive sound. (a) plots the adaptive window length (stems) at each hop position, with gray line indicating the positions of a large, fixed hop size; (b) illustrates the corresponding shifting windows; and (c) shows the observed amplitude, with the DGT-based magnitude in gray for comparison. The window coloring scheme is the same as in Fig. 8 .

Fig. 10: Conceptual diagram of a bounded-delay, online implementation. The left panel illustrates the pre-analysis of a target percussive component (red) using the DGT. The left edge represents the current time position, and the analysis windows corresponding to the required look-ahead frames are highlighted in blue. The top-right panel depicts the onset detection and compression rate calculation (corresponding to Fig. 5 (f)), displaying the magnitude ratio in gray and the detected peak in yellow. The bottom-right panel outlines the subsequent procedures.

		PV	PVDR	PVHP	DIALGA	DIALGA	SELEBI	SELEBI
					(PV)	(PVDR)	(PV)	(PVDR)
$\alpha=2$	Impulse	0.2357	0.2402	0.6993	0.1009	0.1110	0.2326	0.2326
	Sinusoid + Impulse	0.0113	0.0091	0.0130	0.0090	0.0092	0.0106	0.0051
	Harmonic Sinusoid + Impulse	0.0129	0.0107	0.0139	0.0136	0.0107	0.0128	0.0098
	Transient	0.5806	0.5666	0.5924	0.5659	0.5647	0.5791	0.5552
	Sinusoid + Transient	0.2506	0.2446	0.4244	0.2455	0.2445	0.2521	0.2414

TABLE II: Comparison of error measurements for synthetic signals (Sec. VI-A ). The top and bottom rows show the results of

2\times

and

4\times

stretching, respectively. “DIALGA (PVDR)” denotes DIALGA with PVDR phase generation, and similarly for other methods. The best and second-best scores are in bold with blue and light blue backgrounds, respectively.

Fig. 11: An impulse stretched $4\times$ . The leftmost column shows the ground truth (blue), while the remaining columns display the stretched signal for each comparison method (red). The figure is divided into two rows: the top row shows a magnified view of the impulse, and the bottom row displays the relative error spectrogram (log scale). In the top row, for visual clarity, the ground truth is plotted as a gray dashed line. In the error spectrogram (bottom), brighter colors indicate larger errors.

Fig. 12: Example of a transient signal stretched $4\times$ . The leftmost column shows the ground truth (blue), while the remaining columns display the stretched signal for each comparison method (red). The figure is divided into three rows: the top row shows the entire time waveform, the middle row shows a magnified view of the waveform onset, and the bottom row displays the relative error spectrogram (log scale). The horizontal axis for all panels is time (seconds). In the error spectrogram (bottom), brighter color indicates greater error.

Fig. 13: Examples of the stretched signals of “CastanetsViolin” and “Glockenspiel”. The leftmost column shows the original signal, while the remaining columns display the stretched signal for each comparison method. The horizontal axis for all panels is time (seconds). Transient smearing in classical PV is apparent in both examples, whereas PVDR and DIALGA demonstrate different, characteristic artifacts at percussive events in the synthesized signals: PVDR exhibits slight smearing and both methods result in somewhat reduced amplitude of attacks. Additionally, in the “Glockenspiel” excerpt PVDR and DIALGA show a separation of the transient attack portion from the more harmonic decay segment of each played note. In contrast, SELEBI yields a more natural waveform that preserves attacks and harmonic signal components and more closely resembles the original signal.

Fig. 14: Screenshot of the UI of the code used for subjective evaluation.

		Overall Quality				Sharpness of Attacks				Clarity of Tonal Components
		PV	PVDR	DIALGA	SELEBI	PV	PVDR	DIALGA	SELEBI	PV	PVDR	DIALGA	SELEBI
$\alpha=2$	Bongo	14.98	82.48	73.28	94.18	19.62	85.92	72.88	92.31	15.71	77.22	69.11	85.72
	CastanetsViolin	28.77	72.73	80.69	78.78	10.01	69.17	71.35	75.38	50.33	72.27	81.55	78.32
	DrumSolo	17.43	56.48	58.07	64.49	10.56	54.93	54.49	59.88	22.74	57.35	51.17	57.69
	Glockenspiel	33.57	67.24	63.93	74.21	29.48	63.26	63.73	73.26	40.28	67.59	66.47	74.66
	Jazz	33.57	68.26	63.71	68.58	28.56	62.05	61.80	68.65	33.92	67.63	64.59	62.09

TABLE III: Comparison of the mean scores across three metrics: Overall Quality, Sharpness of Attacks, and Clarity of Tonal Components. The best and second-best scores are in bold with blue and light blue backgrounds, respectively.

Audio Editing

Audio editing studies how to modify speech, music, and sound while controlling the change and preserving the rest.

See all 55 papers

Enhancing time-frequency resolution with optimal transport and barycentric fusion of multiple spectrogram
Apr 16, 2026David Valdivia, Elsa Cazelles, Cédric FévotteOptimal Transport
IRIT, University of Toulouse, CNRS, France
DDSP-QbE++: Improving Speech Quality for Speech Anonymisation for Atypical Speech
Apr 10, 2026Suhita Ghosh, Yamini Sinha, Sebastian StoberSpeech Processing Voice Conversion
Artificial Intelligence Lab, Otto-von-Guericke University Magdeburg, Germany · MDS-IIKT Otto-von-Guericke University Magdeburg, Germany
PhaseGAN: High-Fidelity Vocoder via Decoupled Amplitude and GAN-Driven Phase Reconstruction
Sep 11, 2026Wenzheng Zhang, Xueliang Zhang, Shulin He +7Neural Vocoders Speech Processing
College of Computer Science, Inner Mongolia University, China

Apr 16, 2026J/K move · Enter open · S save

Enhancing time-frequency resolution with optimal transport and barycentric fusion of multiple spectrogram

David Valdivia, Elsa Cazelles, Cédric Févotte

IRIT, University of Toulouse, CNRS, France

Optimal Transport

Open paper AskPDF arXiv

Time-frequency representations, such as the short-time Fourier transform (STFT), are fundamental tools for analyzing non-stationary signals. However, their ability to achieve sharp localization in both time and frequency is inherently limited by the Gabor-Heisenberg uncertainty principle. In this paper, we address this limitation by introducing a method to generate super-resolution spectrograms through the fusion of two or more spectrograms with varying resolutions. Specifically, we compute the super-resolution spectrogram as the barycenter of input spectrograms using optimal transport (OT) divergences. Unlike existing fusion approaches, our method does not require the input spectrograms to share the same time-frequency grid. Instead, the input spectrograms can be computed using any STFT parameters, and the resulting super-resolution spectrogram can be defined on an arbitrary user-specified grid. We explore various OT divergences based on different transportation costs. Notably, we introduce a novel transportation cost that preserves time-frequency geometry while significantly reducing computational complexity compared to standard Wasserstein barycenters. We adopt the unbalanced OT framework and derive a new block majorization-minimization algorithm for efficient barycenter computation. We validate the proposed method on controlled synthetic signals and recorded speech using both quantitative and qualitative evaluations. The results show that our approach combines the best localization properties of the input spectrograms and outperforms an unsupervised state-of-the-art fusion method.

Abstract

Audio Editing

Explore similar work

Enhancing time-frequency resolution with optimal transport and barycentric fusion of multiple spectrogram

DDSP-QbE++: Improving Speech Quality for Speech Anonymisation for Atypical Speech

PhaseGAN: High-Fidelity Vocoder via Decoupled Amplitude and GAN-Driven Phase Reconstruction