Revisiting Input Time-frequency Representations in Multi-pitch Estimation for Vocal Ensembles

Junyoung Koh; Hao-Wen Dong

cs.SDOct 2, 2026

Revisiting Input Time-frequency Representations in Multi-pitch Estimation for Vocal Ensembles

Authors: Junyoung Koh, Hao-Wen Dong

Organizations: Yonsei University · University of Michigan

Abstract

Multi-pitch estimation in vocal ensembles is challenging because singers occupy overlapping pitch ranges and often sing at closely spaced fundamental frequencies, causing their harmonics to overlap in time-frequency representations. Existing models commonly use harmonic constant-Q transform (HCQT)-based representations to provide frequency-adaptive resolution, at the cost of expensive feature extraction when training mixtures are generated on the fly. We revisit this design and compare HCQT with a linear short-time Fourier transform (STFT), whose frequency bins are directly provided as model inputs. Despite its fixed frequency resolution and the absence of a pitch-aligned input grid, the linear STFT outperforms HCQT while substantially reducing feature-extraction cost. Further analysis shows that a longer analysis window or broader spectral coverage provides no additional improvement, while restricting the input to the predicted pitch range reduces the advantage of the linear STFT. These results suggest that finer frequency resolution does not necessarily improve vocal-ensemble MPE, and that shorter analysis windows can be more effective for time-varying vocal pitches.

Figures & tables

Figure 1: The two representations compared, on a four-second a cappella recording in jaCappella [ 20 ] : (a) the linear STFT with 2048-sample window and (b) the harmonic CQT. Reference fundamental frequencies are shown in white, with $h=1,2,3$ shown in red, green, and blue, respectively, in (b). The dashed line in (a) marks C7, the upper bound of the predicted pitch range.

Figure 2: Model architecture. The first 360 bins of a 2048-sample STFT (0–7.7 kHz) are processed by a convolutional stem and an encoder, followed by a frame-wise prediction head over 360 pitch bins (C1–C7).

	jaCappella [ 20 ]		ACappellaSet [ 22 ]		Korean [ 18 ]		Cantoría [ 6 ]		ChoralSynth ∗ [ 21 ]
Model	20 cent	50 cent	20 cent	50 cent	20 cent	50 cent	20 cent	50 cent	20 cent	50 cent
HCQT [ 3 ]
Late/Deep CNN [ 9 ]	.387 $\pm$ .039	.804 $\pm$ .033	.334 $\pm$ .026	.716 $\pm$ .042	.309 $\pm$ .008	.791 $\pm$ .013	.545 $\pm$ .012	.863 $\pm$ .010	.626 $\pm$ .020	.932 $\pm$ .014
VoasCNN [ 8 ]	.291 $\pm$ .015	.747 $\pm$ .040	.208 $\pm$ .030	.513 $\pm$ .074	.255 $\pm$ .009	.715 $\pm$ .020	.358 $\pm$ .010	.791 $\pm$ .022	.469 $\pm$ .027	.904 $\pm$ .018
VoasCLSTM [ 8 ]	.299 $\pm$ .018	.731 $\pm$ .040	.248 $\pm$ .018	.573 $\pm$ .055	.269 $\pm$ .008	.745 $\pm$ .017	.416 $\pm$ .011	.819 $\pm$ .013	.527 $\pm$ .029	.894 $\pm$ .022
ConvNeXt [ 14 ]	.648 $\pm$ .036	.884 $\pm$ .019	.492 $\pm$ .027	.828 $\pm$ .023	.584 $\pm$ .016	.880 $\pm$ .009	.561 $\pm$ .017	.875 $\pm$ .012	.638 $\pm$ .020	.934 $\pm$ .016

Table 1: A cappella ensembles and choirs. Each evaluation song is scored independently with multipitch accuracy,

TP/(TP+FP+FN)

, on the 360-bin, 20-cent pitch grid: exact-bin matching (20 cent) and a 50-cent tolerance. The activation threshold is selected separately for each model to maximize the corresponding metric. Intervals denote the half-width of the 95% bootstrap confidence interval over songs. ∗ ChoralSynth is excluded from the training data of both our models and the published baselines.

Dataset	Language	Songs	Hours	Songs	Hours
		Training		Validation
Korean [ 18 ]	Korean	51	50.9	48	14.5
jaCappella [ 20 ]	Japanese	32	40.3	8	4.8
ACappellaSet [ 22 ]	Chinese	66	32.8	17	6.4
Cantoría [ 6 ]	Latin	11	8.2	3	0.9
Dagstuhl [ 25 ]	German	6	0.8	2	0.2

Table 2: Training datasets. Training and validation splits are defined separately for each dataset. Hours denote the duration of songs.

	Dagstuhl [ 25 ]		ESMUC [ 7 ]
	20 cent	50 cent	20 cent	50 cent
HCQT
Late/Deep CNN	.470 $\pm$ .012	.904 $\pm$ .004	.564 $\pm$ .028	.910 $\pm$ .005
VoasCNN	.374 $\pm$ .015	.871 $\pm$ .003	.381 $\pm$ .018	.862 $\pm$ .007
VoasCLSTM	.394 $\pm$ .013	.858 $\pm$ .003	.406 $\pm$ .024	.841 $\pm$ .010
ConvNeXt	.617 $\pm$ .009	.907 $\pm$ .010	.475 $\pm$ .038	.826 $\pm$ .011

Table 3: Choirs recorded in one room. Dagstuhl ChoirSet and ESMUC are the only evaluation datasets whose singers were recorded together, and both are included in the training data of the Late/Deep CNN and our models.

Figure 3: F1 and computational cost. Time per epoch re-timed on one A100. All models are trained and evaluated on 30 s segments. InceptionNeXt coincides with ConvNeXt.

Figure 4: Analysis window. Linear STFT with the ConvNeXt encoder across analysis-window sizes, scored against the reference $F_{0}$ before 20-cent quantization. Dashed lines: HCQT with the same encoder. The top panel reports F1 at three pitch tolerances, and the bottom panel reports the pitch error of matched pitches.

Explore similar work

A Lightweight Slot-Attention Framework for Multi-Instrument Multi-Pitch Estimation
May 31, 2026Michael TaenzerMusic Structure Analysis Symbolic Music Generation
Electronic Media Technology Group Ilmenau University of Technology Ilmenau, Germany
PHAST-Net: Attention-Guided, Physics-Informed Network for Unified Estimation of Ideal Time-Frequency Representations
Jun 22, 2026James M. Cozens, Simon J. GodsillTime-Frequency Physics-Informed Learning
Engineering Department, University of Cambridge, CB2 1PZ Cambridge, U.K.
Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems
May 10, 2026Leduo Chen, Junchuan Zhao, Shengchen LiSpeech Separation Mix
XJTLU

May 31, 2026, cs.SDJ/K move · Enter open · S save

A Lightweight Slot-Attention Framework for Multi-Instrument Multi-Pitch Estimation

Michael Taenzer

Electronic Media Technology Group Ilmenau University of Technology Ilmenau, Germany

Music Structure Analysis Symbolic Music Generation Lightweight

Open paper AskPDF arXiv

Multi-pitch estimation (MPE) typically predicts which pitches are active in a mixture, but not which instrument or source produced them. This paper investigates a lightweight slot-attention framework for multi-instrument MPE (MI-MPE), where a mixture CQT is mapped to an unordered set of source-like pitch maps. The model uses permutation-invariant Hungarian matching to avoid fixed output semantics and treats the number of slots as an upper bound on the number of active sources. We further study two modular extensions: a self-supervised timbre encoder that provides training-time targets for slot-level timbre embeddings, and a polyphony branch that regularizes the pitch density of mixture- and slot-level predictions. Experiments show that Hungarian matching substantially improves instrument family decomposition on URMP. Stem-level prediction remains more challenging: timbre and polyphony supervision improve selected configurations, but do not consistently resolve source assignment. The results suggest that slot-based architectures are a promising direction for source-aware MPE, while highlighting the need to couple auxiliary musical cues to slot identity more carefully.

Abstract

Explore similar work

A Lightweight Slot-Attention Framework for Multi-Instrument Multi-Pitch Estimation

PHAST-Net: Attention-Guided, Physics-Informed Network for Unified Estimation of Ideal Time-Frequency Representations

Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems