Whistle and Wrong Lyrics: Synthetic Queries Are Not Enough for Real-World Query-by-Melody
Organizations: Independent Researcher
Abstract
People look up a song by humming it, but also by whistling it or singing it with the wrong words. Published query-by-humming systems and benchmarks focus on hums and sung queries, and no public dataset pairs whistles or wrong-lyrics singing with song recordings. We study both query types with a melody-only retrieval system: a key-normalized pitch track embedded by a 5.2M-parameter contour encoder, with a spectral-peak tracker for whistles. First, a physics-grounded generator renders synthetic whistles and nonsense-syllable singing from the vocal melodies of 11,083 songs. Training on them raises top-1 retrieval of held-out synthetic queries against an 11,050-song gallery from 0.180 to 0.251 for whistles and from 0.193 to 0.264 for wrong-lyrics singing, but the gain does not transfer: on the MLEnd test performers the best synthetic-only model gets 210 of 339 real whistles right at top-1, against 214 for the baseline. Second, a closed-set fine-tune on real MLEnd whistles with performer-disjoint splits raises whistle-to-hum top-1 from 0.631 (95% CI 0.581-0.681) to 0.673 (0.622-0.723), on known songs only. Third, two further generations of synthetic queries also fail to transfer: melody-conditioned whistles and la-la singing from a commercial neural music model leave real-whistle top-1 at 0.631, and an open-source pipeline (YingMusic-Singer-Plus singing with partly wrong, mumbled or nonsense lyrics, a DDSP whistle synthesizer, and a melody-error model fitted on real performers) gives at best 0.619 (paired difference -0.012, 95% CI -0.038 to +0.015). Across physics-based, commercial neural and open-source neural synthetic data, nothing transferred to real whistles, which points to collecting real queries as the missing step. Code is public; models and data will be released.
Figures & tables
| Model | Whistle | Wrong lyrics | Both (8,526) |
|---|---|---|---|
| E2b (baseline) | 0.180 / 0.291 / 0.220 | 0.193 / 0.314 / 0.235 | 0.186 / 0.303 / 0.227 |
| D-040 (step 3,000) | 0.226 / 0.342 / 0.267 | 0.238 / 0.365 / 0.283 | 0.232 / 0.354 / 0.275 |
| D-041 best (9,500) | 0.253 / 0.378 / 0.297 | 0.269 / 0.401 / 0.314 | 0.261 / 0.389 / 0.305 |
| D-042 best (11,250) | 0.251 / 0.374 / 0.295 | 0.264 / 0.399 / 0.312 | 0.258 / 0.387 / 0.303 |
| Set | E2b | D-040 | D-041 best | D-042 best |
|---|---|---|---|---|
| HumTrans test (769) | 0.996 / 1.000 | 0.991 / 1.000 | 0.993 / 1.000 | 0.992 / 1.000 |
| HumTrans test, st | 0.996 / 1.000 | 0.995 / 1.000 | 0.995 / 1.000 | 0.993 / 1.000 |
| MIR-QBSH 48 (4,431) | 0.955 / 0.993 | 0.944 / 0.991 | 0.949 / 0.991 | 0.950 / 0.992 |
| Model | Hakuna + Potter (90) | Other 6 songs (249) | All 8 (339) |
|---|---|---|---|
| E2b | 0.589 / 0.711 / 0.698 (53) | 0.647 / 0.791 / 0.742 (161) | 0.631 / 0.770 / 0.731 (214) |
| D-040 | 0.656 / 0.789 / 0.741 (59) | 0.594 / 0.779 / 0.720 (148) | 0.611 / 0.782 / 0.725 (207) |
| D-041 best † | 0.456 / 0.700 / 0.620 (41) | 0.707 / 0.863 / 0.803 (176) | 0.640 / 0.820 / 0.754 (217) |
| D-042 best | 0.622 / 0.722 / 0.709 (56) | 0.618 / 0.763 / 0.724 (154) | 0.619 / 0.752 / 0.720 (210) |
| † Trained on real whistles of the other 6 songs: only the Hakuna + Potter column is clean. | |||
| Model | Train performers | Top-1 [clip 95% CI] | Top-3 | MRR |
|---|---|---|---|---|
| E2b zero-shot | 0 | 0.631 [0.581, 0.681] | 0.770 | 0.731 |
| FT 10% | 14 | 0.634 [0.584, 0.684] | 0.776 | 0.735 |
| FT 25% | 34 | 0.625 [0.575, 0.678] | 0.794 | 0.738 |
| FT 50% | 68 | 0.655 [0.605, 0.705] | 0.811 | 0.757 |
| FT 100% | 136 | 0.673 [0.622, 0.723] | 0.820 | 0.768 |
| Phase A (32 classics) | Phase B (79 chart songs) | |
| Sections accepted / blocked | 37 / 55 | 54 / 183 |
| Clips (whistle + la-la) | 74 | 106 |
| La-la top-1 / top-10 | 6/37 / 11/37 | 17/53 / 22/53 |
| Whistle top-1 / top-10 | 3/37 / 11/37 | 7/53 / 11/53 |
| MLEnd test whistles hums [real] : E2b 0.631 [0.581, 0.681] (339), 0.589 [0.489, 0.689] (held-out 90); | ||
| D-047 fine-tune: identical (best checkpoint = step 0). | ||
| All 8 songs (339) | Hakuna + Potter, held out (90) | P90 (304) | ||||
| Model | Top-1 [CI] | [CI] | Top-1 [CI] | [CI] | Top-3 | Top-1 [CI] |
| E2b | 0.631 [0.581, 0.681] | – | 0.589 [0.489, 0.689] | – | 0.711 | 0.648 [0.592, 0.701] |
| D-048 best (= step 0) | identical to E2b | |||||
| D-048 last | 0.546 [0.496, 0.599] | n/a | 0.533 [0.433, 0.633] | n/a | 0.756 | 0.549 [0.493, 0.605] |
| D-049 A best (= step 0) | identical to E2b | |||||
| D-049 A last | 0.546 [0.493, 0.599] | [ , ] | 0.511 [0.411, 0.611] | [ , ] | 0.733 | 0.553 [0.497, 0.609] |
Music Information Retrieval
Music Information Retrieval develops computational methods for finding, classifying, describing, and analyzing music from audio, scores, and metadata.
See all 153 papers