Revisiting Input Time-frequency Representations in Multi-pitch Estimation for Vocal Ensembles

Junyoung Koh; Hao-Wen Dong

cs.SDOct 2, 2026

Revisiting Input Time-frequency Representations in Multi-pitch Estimation for Vocal Ensembles

Authors: Junyoung Koh, Hao-Wen Dong

Organizations: Yonsei University · University of Michigan

Abstract

Multi-pitch estimation in vocal ensembles is challenging because singers occupy overlapping pitch ranges and often sing at closely spaced fundamental frequencies, causing their harmonics to overlap in time-frequency representations. Existing models commonly use harmonic constant-Q transform (HCQT)-based representations to provide frequency-adaptive resolution, at the cost of expensive feature extraction when training mixtures are generated on the fly. We revisit this design and compare HCQT with a linear short-time Fourier transform (STFT), whose frequency bins are directly provided as model inputs. Despite its fixed frequency resolution and the absence of a pitch-aligned input grid, the linear STFT outperforms HCQT while substantially reducing feature-extraction cost. Further analysis shows that a longer analysis window or broader spectral coverage provides no additional improvement, while restricting the input to the predicted pitch range reduces the advantage of the linear STFT. These results suggest that finer frequency resolution does not necessarily improve vocal-ensemble MPE, and that shorter analysis windows can be more effective for time-varying vocal pitches.

Figures & tables

Figure 1: The two representations compared, on a four-second a cappella recording in jaCappella [ 20 ] : (a) the linear STFT with 2048-sample window and (b) the harmonic CQT. Reference fundamental frequencies are shown in white, with $h=1,2,3$ shown in red, green, and blue, respectively, in (b). The dashed line in (a) marks C7, the upper bound of the predicted pitch range.

Figure 2: Model architecture. The first 360 bins of a 2048-sample STFT (0–7.7 kHz) are processed by a convolutional stem and an encoder, followed by a frame-wise prediction head over 360 pitch bins (C1–C7).

	jaCappella [ 20 ]		ACappellaSet [ 22 ]		Korean [ 18 ]		Cantoría [ 6 ]		ChoralSynth ∗ [ 21 ]
Model	20 cent	50 cent	20 cent	50 cent	20 cent	50 cent	20 cent	50 cent	20 cent	50 cent
HCQT [ 3 ]
Late/Deep CNN [ 9 ]	.387 $\pm$ .039	.804 $\pm$ .033	.334 $\pm$ .026	.716 $\pm$ .042	.309 $\pm$ .008	.791 $\pm$ .013	.545 $\pm$ .012	.863 $\pm$ .010	.626 $\pm$ .020	.932 $\pm$ .014
VoasCNN [ 8 ]	.291 $\pm$ .015	.747 $\pm$ .040	.208 $\pm$ .030	.513 $\pm$ .074	.255 $\pm$ .009	.715 $\pm$ .020	.358 $\pm$ .010	.791 $\pm$ .022	.469 $\pm$ .027	.904 $\pm$ .018
VoasCLSTM [ 8 ]	.299 $\pm$ .018	.731 $\pm$ .040	.248 $\pm$ .018	.573 $\pm$ .055	.269 $\pm$ .008	.745 $\pm$ .017	.416 $\pm$ .011	.819 $\pm$ .013	.527 $\pm$ .029	.894 $\pm$ .022
ConvNeXt [ 14 ]	.648 $\pm$ .036	.884 $\pm$ .019	.492 $\pm$ .027	.828 $\pm$ .023	.584 $\pm$ .016	.880 $\pm$ .009	.561 $\pm$ .017	.875 $\pm$ .012	.638 $\pm$ .020	.934 $\pm$ .016

Table 1: A cappella ensembles and choirs. Each evaluation song is scored independently with multipitch accuracy,

TP/(TP+FP+FN)

, on the 360-bin, 20-cent pitch grid: exact-bin matching (20 cent) and a 50-cent tolerance. The activation threshold is selected separately for each model to maximize the corresponding metric. Intervals denote the half-width of the 95% bootstrap confidence interval over songs. ∗ ChoralSynth is excluded from the training data of both our models and the published baselines.

Dataset	Language	Songs	Hours	Songs	Hours
		Training		Validation
Korean [ 18 ]	Korean	51	50.9	48	14.5
jaCappella [ 20 ]	Japanese	32	40.3	8	4.8
ACappellaSet [ 22 ]	Chinese	66	32.8	17	6.4
Cantoría [ 6 ]	Latin	11	8.2	3	0.9
Dagstuhl [ 25 ]	German	6	0.8	2	0.2

Table 2: Training datasets. Training and validation splits are defined separately for each dataset. Hours denote the duration of songs.

	Dagstuhl [ 25 ]		ESMUC [ 7 ]
	20 cent	50 cent	20 cent	50 cent
HCQT
Late/Deep CNN	.470 $\pm$ .012	.904 $\pm$ .004	.564 $\pm$ .028	.910 $\pm$ .005
VoasCNN	.374 $\pm$ .015	.871 $\pm$ .003	.381 $\pm$ .018	.862 $\pm$ .007
VoasCLSTM	.394 $\pm$ .013	.858 $\pm$ .003	.406 $\pm$ .024	.841 $\pm$ .010
ConvNeXt	.617 $\pm$ .009	.907 $\pm$ .010	.475 $\pm$ .038	.826 $\pm$ .011

Table 3: Choirs recorded in one room. Dagstuhl ChoirSet and ESMUC are the only evaluation datasets whose singers were recorded together, and both are included in the training data of the Late/Deep CNN and our models.

Figure 3: F1 and computational cost. Time per epoch re-timed on one A100. All models are trained and evaluated on 30 s segments. InceptionNeXt coincides with ConvNeXt.

Figure 4: Analysis window. Linear STFT with the ConvNeXt encoder across analysis-window sizes, scored against the reference $F_{0}$ before 20-cent quantization. Dashed lines: HCQT with the same encoder. The top panel reports F1 at three pitch tolerances, and the bottom panel reports the pitch error of matched pitches.

Explore similar work

May 31, 2026cs.SD

A Lightweight Slot-Attention Framework for Multi-Instrument Multi-Pitch Estimation

Multi-pitch estimation (MPE) typically predicts which pitches are active in a mixture, but not which instrument or source produced them. This paper investigates a lightweight slot-attention framework for multi-instrument MPE (MI-MPE), where a mixture CQT is mapped to an unordered set of source-like pitch maps. The model uses permutation-invariant Hungarian matching to avoid fixed output semantics and treats the number of slots as an upper bound on the number of active sources. We further study two modular extensions: a self-supervised timbre encoder that provides training-time targets for slot-level timbre embeddings, and a polyphony branch that regularizes the pitch density of mixture- and slot-level predictions. Experiments show that Hungarian matching substantially improves instrument family decomposition on URMP. Stem-level prediction remains more challenging: timbre and polyphony supervision improve selected configurations, but do not consistently resolve source assignment. The results suggest that slot-based architectures are a promising direction for source-aware MPE, while highlighting the need to couple auxiliary musical cues to slot identity more carefully.

Michael Taenzer

Electronic Media Technology Group Ilmenau University of Technology Ilmenau, Germany

Music Structure Analysis Symbolic Music Generation Lightweight

Ask PDF

Jun 22, 2026eess.AS

PHAST-Net: Attention-Guided, Physics-Informed Network for Unified Estimation of Ideal Time-Frequency Representations

We introduce PHAST-Net, an attention-guided, physics-informed network for unified estimation of Ideal Time-Frequency Representations (ITFRs), spanning spectral, tempo-based, metrical, and harmonic representations such as Spectrograms, Tempograms, and Metrograms. PHAST-Net learns an application-general mapping from a constellation of wavelet transforms, the proposed Continuous Log-frequency Adaptive Wavelet Transform (CLAWT), to high-resolution, cross-term-suppressed time-frequency (T-F) representations. The proposed constellation of CLAWTs is selected through Cohen's class kernel analysis to maximise curvature coverage in a logarithmic-frequency T-F plane tailored to harmonic signal structure. PHAST-Net further incorporates a proposed physics-informed auxiliary reprojection loss designed to reconstruct the idealised observed CLAWT constellation from the predicted ITFR and the corresponding Cohen's class kernels during training. This auxiliary objective promotes transform consistency and energy conservation, mitigates pathological target sparsity, and enhances optimisation stability. Attention layers further promote effective cross-term suppression across the input constellation. The log-frequency formulation also enables Harmonic PHAST-Net, which estimates a Harmonic ITFR that isolates fundamental structure, supporting robust fundamental-only representations for speech and music, such as derived fundamental Tempograms and Metrograms. We further introduce Spline-PHAST-Net, which parameterises detected and associated T-F ridges as continuous spline trajectories, enabling arbitrary-grid re-rendering and signal reconstruction. Trained on an effectively unbounded procedurally generated dataset, PHAST-Net demonstrates improved accuracy over established approaches, providing a unified framework for high-resolution, cross-term-robust analysis of speech, music, and broader nonstationary signals.

James M. Cozens, Simon J. Godsill

Engineering Department, University of Cambridge, CB2 1PZ Cambridge, U.K.

Time-Frequency Physics-Informed Learning Adaptive Boundary Suppression Attention Layers

Ask PDF

May 10, 2026cs.SD

Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems

Timbre transfer aims to modify the timbral identity of a musical recording while preserving the original melody and rhythm. While single-instrument timbre transfer has made substantial progress, existing approaches to multi-instrument settings rely on separate-then-transfer pipelines that propagate source separation artifacts and produce incoherent synthesized timbres across stems. This paper proposes MixtureTT, to the best of our knowledge the first system for flexible per-stem timbre transfer directly from a polyphonic mixture. Given a mixture and a separate timbre reference for each target voice, MixtureTT jointly transfers all stems to the specified instruments through a shared diffusion process. Modeling the dependencies across the per-stem content and cross-stem harmonic, the proposed joint stem diffusion transformer eliminates cascaded separation error, reduces inference cost by a factor equal to the number of stems, and yields more coherent multi-stem outputs. Despite operating under a strictly harder input condition, evaluations on the SATB choral dataset show that MixtureTT outperforms single-instrument baselines on both objective and subjective metrics demonstrating the necessity of dedicated multi-instrument timbre transfer over the naive separate-then-transfer pipelines. As a result, this work confirms that the cross-stem modeling is essential for mixture-level timbre transfer as the proposed joint setting consistently exceeds an equivalent single-stem ablation.

Leduo Chen, Junchuan Zhao, Shengchen Li

XJTLU

Speech Separation Mix Stemming Diffusion Transformers Mixtures Style

Ask PDF