A Semantic Timbre Dataset for the Electric Guitar

Joseph Cameron; Alan Blackwell

cs.SDMar 17, 2026

A Semantic Timbre Dataset for the Electric Guitar

Authors: Joseph Cameron, Alan Blackwell

Organizations: Department of Computer Science & Technology, University of Cambridge, Cambridge, United Kingdom

Abstract

Understanding and manipulating timbre is central to audio synthesis, yet this remains under-explored in machine learning due to a lack of annotated datasets linking perceptual timbre dimensions to semantic descriptors. We present the Semantic Timbre Dataset, a curated collection of monophonic electric guitar sounds, each labeled with one of 19 semantic timbre descriptors and corresponding magnitudes. These descriptors were derived from a qualitative analysis of physical and virtual guitar effect units and applied systematically to clean guitar tones. The dataset bridges perceptual timbre and machine learning representations, supporting learning for timbre control and semantic audio generation. We validate the dataset by training a variational autoencoder (VAE) on its latent space and evaluating it using human perceptual judgments and descriptor classifiers. Results show that the VAE captures timbral structure and enables smooth interpolation across descriptors. We release the dataset, code, and evaluation protocols to support timbre-aware generative AI research.

Figures & tables

Spectral Timbre		Temporal Timbre
DistortionFX	FilterFX	DynamicsFX	OscillationFX
	Bright	Punchy	Fluttery
Crunchy	Dark	Sharp	Jittery
Crushed	Fat	Soft	Shimmering
Dirty	Resonant	Smooth	Stuttering
Fuzzy	Thin	Tight	WahWah

Table 1: Final 19 timbre descriptors selected for the Semantic Timbre Dataset.

Figure 1: Semantic Timbre Generation VAE model architecture. The encoder maps input spectrograms to a 128D latent space. The decoder reconstructs log-magnitude spectrograms, which are converted back to audio.

Timbre Group	Timbre Descriptor	25	50	75	100	Average MOS
		Timbre Magnitude
DistortionFX	Clean	N/A	N/A	N/A	4.65	4.65
	Crunch	4.15	3.90	4.40	4.55	4.25
	Crush	4.70	4.75	4.15	4.15	4.44
	Dirt	4.55	3.80	4.05	4.15	4.14
	Fuzz	4.75	4.75	4.65	4.70	4.71

Table 2: Average Mean Opinion Scores from all 20 participants for the Semantic Timbre Generation System’s reconstructions of E4 notes.

Figure 2: Timbre classifier CNN model architecture.

Panel of Figure 3 — Figure 3: Classification probabilities for interpolations between the top 2 descriptors for each timbre group.

Panel of Figure 4 — Figure 4: Classification probabilities for interpolations between the top descriptors of each timbre group.

Figure 5: The triangular template used by participants to order interpolated sounds.

Figure 6: Participants’ placements for interpolated audio samples. Refer to Figure 5 for class locations on the experiment’s triangle.

Figure 7: Null distribution of Kendall’s Tau correlation values from 1,000,000 permutations of true rankings. The observed $\tau$ = 0.879 from participant rankings is highlighted, demonstrating strong statistical support ( $p<0.001$ ) for perceptual coherence of VAE-generated timbre interpolations.

Controllable Music Generation

Controllable music generation steers composition and audio synthesis through prompts, musical structure, references, and interactive controls.

See all 91 papers

MuseTimbre: Zero-Shot Timbre Transfer by Controlling a Frozen Music Generator
Sep 24, 2026Yuan-Chiao Cheng, Zhiyao DuanTimbre-Conditioned Music Generation Controllable Music Generation
Music Informatics Group Georgia Institute of Technology Atlanta, GA, USA · Audio Information Research (AIR) Lab University of Rochester Rochester, NY, USA
Evaluating Latent Space Structure in Timbre VAEs: A Comparative Study of Unsupervised, Descriptor-Conditioned, and Perceptual Feature-Conditioned Models
Mar 17, 2026Joseph Cameron, Alan BlackwellVariational Autoencoders Audio Representation Learning
Department of Computer Science & Technology, University of Cambridge, Cambridge, United Kingdom
Predicting Timbre Traits for Interpretable Assessment of Musical Sound Synthesizers
Jun 29, 2026Théo Chasle Cauchy, Modan Tailleur, Lindsey Reymore +2Music Generation Evaluation Music Information Retrieval
Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France · Arizona State University, School of Music, Dance and Theatre, Tempe, AZ, USA · Arturia, Montbonnot Saint-Martin, France

Sep 24, 2026J/K move · Enter open · S save

MuseTimbre: Zero-Shot Timbre Transfer by Controlling a Frozen Music Generator

Yuan-Chiao Cheng, Zhiyao Duan

Music Informatics Group Georgia Institute of Technology Atlanta, GA, USA · Audio Information Research (AIR) Lab University of Rochester Rochester, NY, USA

Timbre-Conditioned Music Generation Controllable Music Generation Audio Representation Learning Music Generation

Open paper AskPDF arXiv

Instrument timbre transfer re-voices a performance using the timbre of another instrument. Extracting the target timbre from an audio reference capture more nuances than inferring it from a text prompt. Systems that read timbre from such a clip train a dedicated model for the task, which captures the timbre cleanly but stays a narrow, single-purpose system. More versatile approaches add control to a pretrained music generator, yet a reference clip entangles timbre with genre and melody, so these systems fall back on text to name the timbre. We present MuseTimbre, the first system, to our best knowledge, that transfers timbre from an audio reference to a polyphonic source through conditioning a pretrained music generator. This system employs a multi-pitch estimator to extract pitch information from the source and finetune a CLAP encoder to extract timbre information from the reference audio. Experiments show that across four datasets of real polyphonic recordings, MuseTimbre achieves pitch alignment on par with the baselines while matching the reference timbre far more closely. Results also show that the finetuned CLAP-based timbre extractor is robust to pitch variations, making it useful in timbre similarity measures.

Abstract

Controllable Music Generation

Explore similar work

MuseTimbre: Zero-Shot Timbre Transfer by Controlling a Frozen Music Generator

Evaluating Latent Space Structure in Timbre VAEs: A Comparative Study of Unsupervised, Descriptor-Conditioned, and Perceptual Feature-Conditioned Models

Predicting Timbre Traits for Interpretable Assessment of Musical Sound Synthesizers