JIT-MPE: Just-in-Time Multi-Pitch Estimation under Output Deadlines

Yoshiyuki Kobayashi

cs.SDOct 23, 2026

JIT-MPE: Just-in-Time Multi-Pitch Estimation under Output Deadlines

Authors: Yoshiyuki Kobayashi

Organizations: Frieve

Abstract

Live pitch displays, score followers and accompaniment systems must know which pitches are sounding within tens of milliseconds, but instrument-agnostic multi-pitch estimation (MPE) has mainly been designed and evaluated offline. We present JIT-MPE, a causal MPE model of 86k parameters for mixed ensembles. It gives an immediate estimate of each 20 ms frame and re-estimates the same frame as 20, 40, 80 and 160 ms of further audio arrive, so that a live display can show pitches at once and then refine them. We score each output only at a deadline by which it can be computed from the audio that has arrived, on five development sets and four held-out sets evaluated only after all design choices were frozen. At a 50 ms deadline, which reads the update made with 40 ms of lookahead, JIT-MPE is significantly more accurate than five of six published offline MPE systems in 38 of 45 comparisons (15 of 20 on the held-out sets) and less accurate in only two, although these systems read the whole recording and all but one have 6 to 170 times as many parameters; the sixth, supervised on URMP, is higher on four of the nine sets. The deadline costs little: on the held-out sets, JIT-MPE at 50 ms is within $-1.27$ to $+0.60$ points of recording-level macro F1 of its own final output with 160 ms of lookahead and within $-2.08$ to $+0.79$ points of a bidirectional model of similar size. Waiting pays off where the immediate estimate is weakest: the update at 50 ms adds 1.7-3.6 points over it on the development data and three of the four held-out sets, and raises the recall of reference frames in the first 20 ms after a note onset from 9.8% to 38.4% on the held-out AAM set. The slowest measured frame takes at most 4.15 ms on one CPU thread. Code and trained weights are available at https://github.com/Frieve-A/jit-mpe.

Figures & tables

Figure 1: Timing of one output row of JIT-MPE. The analysis window (the shorter of the two is shown) ends at the target time

\tau

, so the

d=0

head needs no audio after

\tau

. The bracket marks the 20 ms during which the row is shown; its training label refers to the middle,

\tau+10

ms (triangle). Delay head

d

re-estimates the same row from the audio up to

\tau+20d

ms. With a deadline of

D=50

ms, heads 0, 1 and 2 are ready in time (filled markers), and head 2, the one selected for this deadline (Section 4.4 ), is read (large marker); heads 4 and 8 (hollow markers) are not ready in time.

	Radius	Parameters		F1, All-29 (%)
Model	(st)	causal	bidir.	causal	bidir.
Dilated CRNN	47	95,767	93,936	75.93	79.84
Harmonic CRNN	128	85,813	83,729	76.47	80.46
JIT-MPE	128	86,425	—	80.55	—

Table 1: Our models, from the ablation’s starting point to JIT-MPE (Section 4.2 ); a bidirectional counterpart (BiCRNN) of each of the first two is trained on the same data. Radius: frequency reach of the convolutional front end in semitones. Parameters exclude the onset output layer used only for an auxiliary training loss. F1: development aggregate All-29, full-length outputs, primary truth, mean of three seeds.

	Development	Held-out
System	All-29	AAM	ChoraleBricks	BSED-real	BSED-syn.
JIT-MPE, $D=50$ ms	80.83	82.58	77.96	64.52	77.01
JIT-MPE, full length (160 ms lookahead)	80.55 $+0.28$ $\circ$ [ $-0.001$ , 0.58]	83.85 $-1.27$ $\blacktriangledown$ [ $-1.40$ , $-1.14$ ]	77.36 $+0.60$ $\blacktriangle$ [0.47, 0.71]	65.49 $-0.97$ $\blacktriangledown$ [ $-1.27$ , $-0.72$ ]	78.28 $-1.27$ $\blacktriangledown$ [ $-1.70$ , $-0.92$ ]
Harmonic BiCRNN, offline (whole recording)	80.46 $+0.38$ $\circ$ [ $-0.03$ , 0.76]	84.67 $-2.08$ $\blacktriangledown$ [ $-2.55$ , $-1.62$ ]	77.17 $+0.79$ $\blacktriangle$ [0.08, 1.49]	65.04 $-0.52$ $\circ$ [ $-1.24$ , 0.27]	77.50 $-0.50$ $\circ$ [ $-1.03$ , 0.06]

Table 2: Accuracy at the deadline, on the primary truth of each set; all rows are our models. Each cell gives the recording-level macro F1 (%) in normal type, the highest in each column in bold, and below it in small type the paired difference

\Delta

(points) with its 95% bootstrap CI.

\Delta

= JIT-MPE at

D=50

-

the system of the row, so a negative value is the cost of the deadline. JIT-MPE at 50 ms is read on the stale-hold clock; its full-length output and Harmonic BiCRNN take the nearest output frame (Section 3.4 ).

\blacktriangle

above,

\circ

not significantly different,

\blacktriangledown

below. Bounds that would round to zero are given to their first nonzero decimal.

Figure 2: Recording-level macro F1 on the primary truth against the output deadline

D

(0–200 ms), which bounds the audio each answer may use. Shown for the development aggregate All-29 and the four held-out sets. Lines: causal models on the stale-hold clock; the single-head models and JIT-MPE d=0 read

d=0

at every deadline and give flat lines. The dashed vertical line marks

D=50

ms. Shaded band: 95% CI of JIT-MPE. Markers at F: full-length outputs with 95% CIs. Points below the plotted range are drawn at the bottom of the panel with their values. Line styles and marker shapes distinguish the series also in greyscale.

Ours: causal, output deadline $D=50$ ms
		URMP F0s	Bach10-mf0-synth	ChoraleBricks f0
System	Training domain	(dev, 9/7)	(dev, 10/10)	(held-out, 10/10)
JIT-MPE, $D=50$ ms	public, recording-disjoint	84.67	91.06	77.96
Published systems: offline, each reads the whole recording
SS-MPE v0.1.0	NSynth	60.81 $+23.87$ $\blacktriangle$ [17.84, 31.52]	72.28 $+18.79$ $\blacktriangle$ [17.48, 20.24]	69.76 $+8.20$ $\blacktriangle$ [6.37, 9.92]
SS-MPE v0.2.0	URMP (sup.)	88.39 † $-3.72$ $\blacktriangledown$ [ $-5.51$ , $-1.87$ ]	92.87 $-1.80$ $\blacktriangledown$ [ $-2.20$ , $-1.38$ ]	80.88 $-2.92$ $\blacktriangledown$ [ $-3.85$ , $-1.96$ ]

Table 3: Comparison on the continuous-f0 truth (50-cent one-to-one matching). Each cell gives the recording-level macro F1 (%) in normal type, the highest in each column in bold, and below it in small type the paired difference

\Delta

(points) with its 95% bootstrap CI.

\Delta

= JIT-MPE

-

system, so positive values favour JIT-MPE. JIT-MPE is read at

D=50

ms on the stale-hold clock; the published systems are scored on their full-length outputs.

\blacktriangle

above,

\circ

not significantly different,

\blacktriangledown

below. Column headers give recordings/work clusters.

\dagger

: trained or selected on URMP. JIT-MPE emits semitones refined in 20-cent steps; Basic Pitch and the WP models emit semitone frames; SS-MPE and Timbre-Trap emit continuous pitch.

Ours: causal, output deadline $D=50$ ms
		Development			Held-out
System	Training domain	note-19 ‡	TRIOS	PHENICX	AAM	ChoraleBricks ∗	BSED-real ‡	BSED-syn. ‡
		19/17	5/5	4/4	100/100	10/10	80/9	20/9
JIT-MPE, $D=50$ ms	public, rec.-disjoint	77.0	77.6	77.2	82.6	78.9	64.5	77.0
Published systems: offline, each reads the whole recording
SS-MPE v0.1.0	NSynth	54.8 $+22.2$ $\blacktriangle$ [18.5, 26.2]	58.0 $+19.6$ $\blacktriangle$ [14.3, 25.7]	54.0 $+23.2$ $\blacktriangle$ [19.7, 27.2]	60.9 $+21.7$ $\blacktriangle$ [19.7, 23.8]	71.4 $+7.5$ $\blacktriangle$ [5.6, 9.3]	45.5 $+19.0$ $\blacktriangle$ [15.7, 22.7]	51.9 $+25.1$ $\blacktriangle$ [21.6, 28.7]

Table 4: Comparison on the semitone metric. Each cell gives the recording-level macro F1 (%) in normal type, the highest in each column in bold, and below it in small type the paired difference

\Delta

(points) with its 95% bootstrap CI.

\Delta

= JIT-MPE

-

system, so positive values favour JIT-MPE; JIT-MPE is read at

D=50

ms (stale hold), and the systems are scored at full length.

\blacktriangle

above,

\circ

not significantly different,

\blacktriangledown

below. For the transcription systems only

\circ

and

\blacktriangledown

are marked. Column headers give recordings/work clusters.

\ast

: uniform check only; the primary truth of ChoraleBricks is f0 (Table 3 ).

\ddagger

: partly in-domain for JIT-MPE through MusicNet recordings (note-19) or repertoire (BSED).

\dagger

: system trained on recordings from the same dataset as part of the column.

\lx@sectionsign

: system trained on MusicNet, whose repertoire overlaps BSED. Bounds that would round to zero are given to their first nonzero decimal. MAE/Slakh/GS/iK/MDB: the Basic Pitch training data of Table 3 .

Figure 3: Recall by time since onset on held-out AAM. Each reference frame is assigned to a bin by the time since the onset of its pitch. JIT-MPE is read at

D=50

ms, at full length (160 ms lookahead) and through its

d=0

head alone; the other systems are scored at full length. Error bars: 95% CIs.

	Development	Held-out
Model	All-29	AAM	ChoraleBricks	BSED-real	BSED-syn.
JIT-MPE	80.83	82.58	77.96	64.52	77.01
$-$ later read of the delay heads (= JIT-MPE d=0 )	77.24 $+3.59$ $\blacktriangle$ [2.81, 4.51]	80.41 $+2.17$ $\blacktriangle$ [1.96, 2.40]	77.95 $+0.02$ $\circ$ [ $-0.09$ , 0.12]	62.81 $+1.70$ $\blacktriangle$ [1.33, 2.05]	74.32 $+2.69$ $\blacktriangle$ [2.32, 2.97]
$-$ window and multi-head training (= Harmonic CRNN)	75.69 $+1.55$ $\blacktriangle$ [1.09, 2.08]	80.32 $+0.09$ $\circ$ [ $-0.28$ , 0.44]	77.66 $+0.28$ $\circ$ [ $-0.24$ , 0.80]	62.99 $-0.18$ $\circ$ [ $-0.50$ , 0.14]	73.31 $+1.01$ $\blacktriangle$ [0.56, 1.53]
$-$ harmonic front end (= Dilated CRNN)	75.13 $+0.55$ $\blacktriangle$ [0.18, 0.91]	80.34 $-0.02$ $\circ$ [ $-0.46$ , 0.44]	77.92 $-0.26$ $\circ$ [ $-0.76$ , 0.24]	60.78 $+2.20$ $\blacktriangle$ [1.86, 2.62]	72.17 $+1.14$ $\blacktriangle$ [0.02, 2.11]

Table 5: Ablation of JIT-MPE at

D=50

ms on the stale-hold clock, primary truth; all rows are our causal models. Each row removes one step from the model of the row above, so the single-head models give their

d=0

output. Each cell gives the recording-level macro F1 (%) in normal type, the highest in each column in bold, and below it in small type the paired difference

\Delta

(points) with its 95% bootstrap CI.

\Delta

= row above

-

this row, the gain of the removed step.

\blacktriangle

above,

\circ

not significantly different,

\blacktriangledown

below. Bounds that would round to zero are given to their first nonzero decimal.

System	Parameters	GMAC/s	Output
Ours
JIT-MPE	86k	0.68	causal, $D=50$ ms
Harmonic BiCRNN	84k	0.66	offline
Published MPE systems
Basic Pitch	17k	0.30	offline
SS-MPE v0.1.0	545k	0.65	offline

Table 6: Model size and computation. Parameters: trained weights, without the fixed time-frequency front end (spectrogram, CQT, HCQT or sliCQ) and, for our models, without the auxiliary onset layer. GMAC/s: multiply-accumulates per second of audio, as executed by each system’s released inference code (ours: our implementation) on the first 20 s of a held-out ChoraleBricks recording, counted with the PyTorch FLOP counter (Basic Pitch: from its ONNX graph), front end excluded. Output: causal at the 50 ms deadline, or offline, computed with the whole recording available. a The released code recomputes a 75-frame context for every output frame. b Autoregressive decoding, so the count grows with the number of notes; with guidance, MuScriptor runs its decoder twice.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Clips	Label source
GuitarSet	138	note annotations
Slakh2100-redux	137	MIDI notes with sustain pedal
MusicNet	67	aligned nominal note pitches
Winterreise	10	aligned nominal note pitches
Cantoria	64	union of per-voice pYIN tracks
Dagstuhl ChoirSet	9	aligned score notes

Appendix

Table 7: Training clips by dataset. About one song in eight of each dataset is held out from training; these clips, except those of Groove, form the calibration set (Appendix C ). Cantoria, Dagstuhl ChoirSet and vocadito form the vocal pool.

System	Source	Checkpoint	Training and selection
SS-MPE v0.1.0	github.com/cwitkowitz/ss-mpe , release v0.1.0	SS-MPE experiment	self-supervised on NSynth, chosen by NSynth validation loss
SS-MPE v0.2.0	github.com/cwitkowitz/ss-mpe , release v0.2.0	URMP supervised	supervised on URMP, chosen on the URMP F0 recordings
Basic Pitch	github.com/spotify/basic-pitch	ICASSP 2022 model	MAESTRO, Slakh, GuitarSet, iKala, MedleyDB
Timbre-Trap	authors’ release	original model	URMP, selected on a nine-recording URMP subset
WP-DRCNN	github.com/christofw/ multipitch_architectures	DRCNN size L, MuN-10full	MusicNet
WP-PUnet XL	github.com/christofw/ multipitch_architectures	PUnet XL, MuN-10full	MusicNet

Appendix

Table 8: Published systems. All are offline and are scored on their full-length outputs. Their size and computation are given in Table 6 .

Our models (causal models at $D=50$ ms unless noted; bidirectional models at full length)
	Development					Held-out
System	URMP f0	Bach10 f0	note-19	TRIOS	PHENICX	AAM	ChoraleBricks f0	BSED-real	BSED-syn.
JIT-MPE, 50 ms	84.7	91.1	77.0	77.6	77.2	82.6	78.0	64.5	77.0
JIT-MPE, full length	83.4	90.0	77.0	77.3	77.2	83.9	77.4	65.5	78.3
JIT-MPE d=0	82.3	88.3	73.0	75.6	75.7	80.4	77.9	62.8	74.3
Harmonic CRNN	81.2	86.9	71.5	75.1	75.4	80.3	77.7	63.0	73.3

Appendix

Table 9: Absolute recording-level macro F1 (%) on the primary truth of each set: continuous-f0 F1 for URMP, Bach10-mf0-synth and ChoraleBricks, and semitone F1 otherwise. Our models: mean of three seeds; outputs at 50 ms on the stale-hold clock, full-length outputs on the nearest output frame (Section 3.4 ).

Music Information Retrieval

Music Information Retrieval develops computational methods for finding, classifying, describing, and analyzing music from audio, scores, and metadata.

See all 153 papers

May 31, 2026

A Lightweight Slot-Attention Framework for Multi-Instrument Multi-Pitch Estimation

Multi-pitch estimation (MPE) typically predicts which pitches are active in a mixture, but not which instrument or source produced them. This paper investigates a lightweight slot-attention framework for multi-instrument MPE (MI-MPE), where a mixture CQT is mapped to an unordered set of source-like pitch maps. The model uses permutation-invariant Hungarian matching to avoid fixed output semantics and treats the number of slots as an upper bound on the number of active sources. We further study two modular extensions: a self-supervised timbre encoder that provides training-time targets for slot-level timbre embeddings, and a polyphony branch that regularizes the pitch density of mixture- and slot-level predictions. Experiments show that Hungarian matching substantially improves instrument family decomposition on URMP. Stem-level prediction remains more challenging: timbre and polyphony supervision improve selected configurations, but do not consistently resolve source assignment. The results suggest that slot-based architectures are a promising direction for source-aware MPE, while highlighting the need to couple auxiliary musical cues to slot identity more carefully.

Michael Taenzer

Electronic Media Technology Group Ilmenau University of Technology Ilmenau, Germany

Slot Attention Music Information Retrieval Permutation-Invariant Learning

Ask PDF

Oct 2, 2026

Revisiting Input Time-frequency Representations in Multi-pitch Estimation for Vocal Ensembles

Multi-pitch estimation in vocal ensembles is challenging because singers occupy overlapping pitch ranges and often sing at closely spaced fundamental frequencies, causing their harmonics to overlap in time-frequency representations. Existing models commonly use harmonic constant-Q transform (HCQT)-based representations to provide frequency-adaptive resolution, at the cost of expensive feature extraction when training mixtures are generated on the fly. We revisit this design and compare HCQT with a linear short-time Fourier transform (STFT), whose frequency bins are directly provided as model inputs. Despite its fixed frequency resolution and the absence of a pitch-aligned input grid, the linear STFT outperforms HCQT while substantially reducing feature-extraction cost. Further analysis shows that a longer analysis window or broader spectral coverage provides no additional improvement, while restricting the input to the predicted pitch range reduces the advantage of the linear STFT. These results suggest that finer frequency resolution does not necessarily improve vocal-ensemble MPE, and that shorter analysis windows can be more effective for time-varying vocal pitches.

Junyoung Koh, Hao-Wen Dong

Yonsei University · University of Michigan

Music Information Retrieval Audio Understanding Time-Frequency Analysis Frequency-Domain Feature Learning

Ask PDF

Sep 20, 2026

Low-Rank Frequency Convolution and Noise-Range Augmentation for Real-Time Pitch Estimation on Edge Devices

Pitch estimation on an edge device is constrained in three ways at once. The model must be small, it must stay accurate when the input is noisy, and one frame must be produced inside the frame period. In this report the Frequency Convolution Network (FrCN) of our earlier work is factored into a low-rank form. The number of parameters is reduced by 35.9%, from 17,787 to 11,397, and accuracy is not reduced, either in domain or on two corpora the model was never trained on. The range of the noise used during training is also shown to dominate the architecture in setting how the model behaves when the interference is severe. When the training noise floor is lowered from +6.02 dB to -20 dB, 0.35 points of clean RPA50 are lost and RPA50 at -20 dB is raised from 2.44 to 22.41. This effect is about two orders of magnitude larger than any architectural effect that was measured. The semi-orthogonal constraint used in TDNN-F is found to be redundant with the normalization inside the bottleneck, and accuracy is reduced when both are applied. Whether the factorization saves time depends on the runtime: in eager PyTorch the factored model is 45% slower, while in a compiled kernel it is 15% faster. For deployment, a small C kernel was written. It needs 83 kB on disk and no runtime library beyond libc and libm. It is faster than OpenBLAS on all four CPUs that were tested, faster than ONNX Runtime by 3.8 times, and faster than PyTorch by 13 times. Its output was checked against PyTorch on 271,893 held-out frames per model, and the same pitch bin was selected on every one of them.

Venkat Suprabath Bitra, Homayoon Beigi

Nonlinear Control Lab, Columbia University, New York, USA · Dept. of Electrical Engineering, Columbia University, New York, USA · Dept. of Mechanical Engineering, Columbia University, New York, USA +1

Edge Inference Noise Robustness Low-Rank Compression

Ask PDF

Abstract

Music Information Retrieval

Explore similar work

A Lightweight Slot-Attention Framework for Multi-Instrument Multi-Pitch Estimation

Revisiting Input Time-frequency Representations in Multi-pitch Estimation for Vocal Ensembles

Low-Rank Frequency Convolution and Noise-Range Augmentation for Real-Time Pitch Estimation on Edge Devices