cs.SDNov 16, 2026

Evaluating the Evaluation of Music Generation Models with Generative Psychometrics

Authors: João Carlos Santana Silveira, Caio Von Rondow Morais, Frederico Pedrosa, Lucas N. Ferreira

Organizations: Departamento de Ciência da Computação, UFMG – Belo Horizonte, MG, Brazil · Escola de Música, UFMG – Belo Horizonte, MG, Brazil

Abstract

Subjective user studies are one of the major methods for evaluating music generation models. However, subjective evaluation protocols vary widely and are rarely specified in enough detail to be reproduced, hindering comparison across studies. In this paper, we map subjective evaluation methods in ISMIR music generation papers from 2020-2025 and apply generative psychometrics to assess the stability of their items. We extract 284 items from the papers' methodological descriptions, using an LLM and manual validation to standardize them. We represent these items as contextual embeddings and apply Unique Variable Analysis to remove redundant, locally dependent items and Bootstrap Exploratory Graph Analysis to estimate their latent dimensions. Results show the network converges onto a taxonomy of six stable quality dimensions: Rhythm, Melody, Audio, Composition, Structure, and Lyrics, pointing to a unified methodology for conducting user studies to evaluate music generation models.

Figures & tables

Explore similar work

CardsList
  1. Do Music Generative Models Understand Musical Qualities? Automatic Music Evaluation with Model-Intrinsic Signals

    Sep 29, 2026Xiaosha Li, Chun Liu, Ziyu WangMusic Generation EvaluationAutomated Evaluation

  2. How Well Do Generative Music Models Follow Emotion Conditioning?

    Aug 30, 2026Morteza HeydariText-to-Music GenerationMusic Generation Evaluation

  3. MusGU+: Toward a Musician-Centered Evaluation Framework and Discovery Tool for Generative Music AI

    Aug 31, 2026Laura Ibáñez-Martínez, Roser Batlle-Roca, Xavier Serra +1Music Generation EvaluationControllable Music Generation