PANEL: An Open-Source, Self-Hosted Web Platform for Human Evaluation of Generative Models
Organizations: University of Padova, Padova, Italy · Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain
Abstract
Human judgement is the reference measure for evaluating generative models, yet the software used to collect it lags behing the methodology. Researchers adapt listening-test frameworks designed for perceptual protocols such as MUSHRA, rely on closed commercial survey platforms, or implement single-use web applications. Live arenas such as Chatbot Arena and Music Arena rank publicly deployed systems at scale, but do not support controlled comparisons of a laboratory's own models with its own participants. We present PANEL, an open-source, self-hosted platform for such studies. A study is authored in the browser and distributed as a single link, with audio, video, image, and text stimuli, seven question types, and screening and skip logic. The platform reports per-question summaries, across-condition significance tests, pairwise win rates and Bradley--Terry scores, and supports power analysis from pilot data. Consent versioning, self-service withdrawal, retention enforcement, and audit logging support GDPR-compliant operation. Each study exports as a machine-readable specification. PANEL is available at https://github.com/matteospanio/panel.
Figures & tables
| Open | Self- | Beyond | Survey | Built-in | |
|---|---|---|---|---|---|
| source | hosted | audio | logic | stats | |
| webMUSHRA [ 7 ] | (✓) | ✓ | – | (✓) | – |
| BeaqleJS [ 8 ] | ✓ | ✓ | – | – | – |
| Go Listen [ 9 ] | ✓ | ✓ | – | (✓) | – |
| PsyToolkit [ 12 ] | – | – | ✓ | ✓ | (✓) |
| Qualtrics [ 11 ] | – | – | ✓ | ✓ | ✓ |
Human-in-the-Loop Evaluation
Human-in-the-Loop Evaluation studies how human judgment and feedback can assess, guide, and monitor AI systems in realistic settings.
See all 158 papers