CoTTAGE: An Explainable Judge for Text-to-Audio Generation
Organizations: National Taiwan University · Amazon AGI · NTU AI-CoRE
Abstract
Rapid progress in text-to-audio (TTA) generation demands reliable and explainable automatic evaluation. However, existing metrics fail to simultaneously deliver high correlation with human judgments and meaningful, audio-grounded explanations, forcing reliance on labor-intensive human evaluation that bottlenecks model development iteration. To address this, we propose CoTTAGE, a reasoning-based evaluator for text relevance (REL) and audio quality (OVL). Following a reason-then-predict paradigm, CoTTAGE first generates an audio-grounded rationale to diagnose errors before predicting scores. Supported by over 10K newly collected human OVL annotations, CoTTAGE is first trained via supervised fine-tuning and then refined with reinforcement learning using a Thurstone-based preference objective that converts sampled score distributions into pairwise preference probabilities and rewards agreement with rankings derived from human judgments. Across six benchmarks spanning REL and OVL, CoTTAGE achieves state-of-the-art correlation with human judgments while producing human-preferred explanations to accelerate TTA development.
Figures & tables
| XACLE | RELATE | PAM-Audio | Clotho | |||||
| LCC | SRCC | LCC | SRCC | LCC | SRCC | LCC | SRCC | |
| Embedding-based Methods | ||||||||
| CLAPScore | 0.472 | 0.398 | 0.437 | 0.411 | 0.493 | 0.488 | 0.396 | 0.378 |
| ELSA | 0.398 | 0.371 | 0.375 | 0.377 | 0.497 | 0.467 | 0.416 | 0.404 |
| Proprietary LALMs | ||||||||
| Gemini-3.1 Pro | 0.551 | 0.489 | 0.471 | 0.423 | 0.473 | 0.442 | 0.548 | 0.528 |
| PAM-Audio | Clotho | |||
| LCC | SRCC | LCC | SRCC | |
| Embedding-based Methods | ||||
| CLAPScore | 0.379 | 0.373 | 0.201 | 0.223 |
| PAM | 0.587 | 0.610 | 0.445 | 0.451 |
| ELSA | 0.387 | 0.363 | 0.305 | 0.283 |
| Regression-based Predictors | ||||
| Type | Instruction | Qwen2.5-Omni | C o ttage | |
| REL | Temporal | “A loud motor begins to pick up speed then cuts out and slowly stops.” Human-annotated MOS: 2.10 | The audio describes a motor that starts loudly, then slows down and eventually stops. The text also describes a motor that begins to pick up speed, then cuts out and slowly stops. The key elements of the motor’s behavior (starting loudly, slowing down, and stopping) are consistent between the audio and the text. Model predicted MOS: 4.50 | […] Listening to the audio, there is a continuous, loud humming or buzzing sound that resembles a motor or drone. However, the audio remains at a constant, steady pitch and intensity from the beginning to the end. […] there is no sense of the motor “picking up speed,” nor is there a sudden “cutting out” or stopping sound. […] Therefore, the alignment between the text and the audio is very poor. Model predicted MOS: 1.80 |
| Counting | “A bug chirps once and then starts chirping repetitively.” Human-annotated MOS: 2.00 | The audio contains a continuous sound of a bug chirping , which aligns with the description of a bug chirping repetitively. The description does not mention the bug chirping once before starting to chirp repetitively […] Model predicted MOS: 4.50 | […] The audio, however, consists of a continuous, repetitive bug chirping from the very beginning. It completely misses the initial “chirp once” action and plays the repetitive sound throughout its duration. […] Model predicted MOS: 1.80 | |
| OVL | Distortion | “Rapid gunfire with male yelling.” Human-annotated MOS: 1.90 | […] The gunshots are clear and distinct, indicating a high-quality sound effect. The man’s speech is also clear and understandable, […] The overall audio quality is consistent with a real-world recording. Model predicted MOS: 4.50 | The audio clip suffers from severe quality degradations throughout its duration. The prominent sounds, which resemble loud bursts or synthetic impacts, are heavily distorted and clipped, resulting in a harsh, abrasive, and unnatural tone. […] Model predicted MOS: 2.00 |
| Artifacts | “A jackhammer drilling and vibrating continuously.” Human-annotated MOS: 2.60 | The audio clip contains a sound that resembles a vehicle engine idling , which is a common and recognizable sound in real-world recordings. The sound is clear and consistent, without any noticeable background noise or unnatural patterns. Therefore, the audio quality is excellent. Model predicted MOS: 5.00 | […] The most prominent issue with this recording is the significant amount of background noise and static. A persistent hiss and low-frequency rumble are present throughout the entire clip, which masks the finer details of the mechanical sound. Additionally, the audio feels somewhat muffled and lacks high-frequency clarity , […] Model predicted MOS: 2.60 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Annotation | ICC(1,1) | ICC(1, ) | |
| RELATE | REL | 0.449 | 0.762 | 0.446 |
| OVL | 0.317 | 0.699 | 0.317 | |
| XACLE | REL | 0.435 | 0.755 | 0.435 |
| OVL | 0.322 | 0.704 | 0.322 |
| Type | Instruction | Qwen2.5-Omni | C o ttage | |
| REL | Temporal | “A loud motor begins to pick up speed then cuts out and slowly stops.” Human-annotated MOS: 2.10 | The audio describes a motor that starts loudly, then slows down and eventually stops. The text also describes a motor that begins to pick up speed, then cuts out and slowly stops. The key elements of the motor’s behavior (starting loudly, slowing down, and stopping) are consistent between the audio and the text. Model predicted MOS: 4.50 | The text instruction describes a sequence of events: a loud motor beginning to pick up speed, then cutting out, and finally slowly stopping. Listening to the audio, there is a continuous, loud humming or buzzing sound that resembles a motor or drone. However, the audio remains at a constant, steady pitch and intensity from the beginning to the end. It completely lacks the dynamic changes requested in the prompt: there is no sense of the motor “picking up speed,” nor is there a sudden “cutting out” or stopping sound. While the audio provides a basic motor-like noise, it fails to capture any of the specific actions or the temporal progression detailed in the instruction. Therefore, the alignment between the text and the audio is very poor. Model predicted MOS: 1.80 |
| Counting | “A bug chirps once and then starts chirping repetitively.” Human-annotated MOS: 2.00 | The audio contains a continuous sound of a bug chirping , which aligns with the description of a bug chirping repetitively. The description does not mention the bug chirping once before starting to chirp repetitively , but the continuous chirping is consistent with the description of repetitive chirping. Therefore, the audio aligns well with the text description. Model predicted MOS: 4.50 | The text instruction specifies a bug chirping once initially, followed by repetitive chirping. The audio, however, consists of a continuous, repetitive bug chirping from the very beginning. It completely misses the initial “chirp once” action and plays the repetitive sound throughout its duration. Because the temporal structure and the distinct events described in the prompt are absent, the audio fails to faithfully represent the instruction. Model predicted MOS: 1.80 | |
| Multi-event | “Ducks quack with blowing wind and faint splashes.” Human-annotated MOS: 2.80 | The audio contains the sound of ducks quacking , which aligns with the text description. The presence of wind and faint splashes also suggests an outdoor setting , which is consistent with the idea of ducks in a natural environment. The audio captures the essence of the described scene, making the semantic alignment quite strong. Model predicted MOS: 4.50 | The text instruction specifies three distinct elements: ducks quacking, blowing wind, and faint splashes. Listening to the audio, the sound of ducks quacking is very prominent and clearly recognizable throughout the clip. However, the other elements described in the prompt are noticeably absent. There is no audible sound of blowing wind, nor are there any faint splashes to be heard. Because the audio successfully captures the primary subject (the ducks quacking) but completely misses the background environmental sounds (the wind and splashes), it only partially fulfills the text instruction. This results in a moderate alignment score. Model predicted MOS: 2.80 |
| Type | Instruction | Qwen2.5-Omni | C o ttage | |
| OVL | Distortion | “Rapid gunfire with male yelling.” Human-annotated MOS: 1.90 | The audio clip contains a series of gunshots followed by a man speaking. The gunshots are clear and distinct, indicating a high-quality sound effect. The man’s speech is also clear and understandable, suggesting good audio quality. There is no background noise or unnatural sound patterns present. The overall audio quality is consistent with a real-world recording. Model predicted MOS: 4.50 | The audio clip suffers from severe quality degradations throughout its duration. The prominent sounds, which resemble loud bursts or synthetic impacts, are heavily distorted and clipped, resulting in a harsh, abrasive, and unnatural tone. This distortion significantly degrades the clarity and dynamic range of the recording. Additionally, there is a pervasive layer of background noise and static that further muddies the audio. The human voices heard towards the end are muffled, lack definition, and are difficult to discern clearly against the noisy background. The combination of intense clipping, high noise levels, and overall lack of fidelity makes the listening experience quite poor. These substantial recording artifacts and the absence of any clean audio elements strongly indicate a very low quality rating. Model predicted MOS: 2.00 |
| Artifacts | “A jackhammer drilling and vibrating continuously.” Human-annotated MOS: 2.60 | The audio clip contains a sound that resembles a vehicle engine idling , which is a common and recognizable sound in real-world recordings. The sound is clear and consistent, without any noticeable background noise or unnatural patterns. The quality of the audio is high, as it accurately represents the sound of an engine idling. Therefore, the audio quality is excellent. Model predicted MOS: 5.00 | The audio presents a continuous mechanical sound, resembling a small engine or motor running. The most prominent issue with this recording is the significant amount of background noise and static. A persistent hiss and low-frequency rumble are present throughout the entire clip, which masks the finer details of the mechanical sound. Additionally, the audio feels somewhat muffled and lacks high-frequency clarity , giving it a slightly distant or enclosed characteristic. While the primary sound source remains identifiable, the heavy presence of noise and the overall lack of crispness severely degrade the listening experience, resulting in a below-average quality rating. Model predicted MOS: 2.60 | |
| High quality | “Baby crying then man speaking and humming.” Human-annotated MOS: 4.20 | The audio clip contains a baby crying, which is a natural sound. However, the quality of the audio seems to be low, with some background noise and possibly some unnatural sound patterns. The audio does not sound like a high-quality real-world recording. Model predicted MOS: 2.50 | The audio presents a baby crying, which is the primary sound source. The crying itself is captured with good clarity and sounds very natural, without any noticeable distortion, clipping, or digital artifacts. The vocalizations are distinct and easy to hear. However, there is a minor amount of background noise or room ambiance present throughout the recording. While this background noise is not overly intrusive or distracting, it prevents the audio from sounding completely pristine or studio-quality. Overall, the recording is clean and the main subject is well-represented, but the slight background hiss slightly diminishes the overall fidelity. Model predicted MOS: 4.00 |