While Large Audio-Language Models (LALMs) have advanced audio captioning, robust evaluation remains difficult. Reference-based metrics are expensive and often fail to assess acoustic fidelity, while Contrastive Language-Audio Pretraining (CLAP)-based approaches frequently overlook syntactic errors and fine-grained details. We propose CAF-Score, a reference-free metric that calibrates CLAP's coarse-grained semantic alignment with the fine-grained comprehension and syntactic awareness of LALMs. By combining contrastive audio-text embeddings with LALM reasoning, CAF-Score effectively detects syntactic inconsistencies and subtle hallucinations. Experiments on the BRACE benchmark demonstrate that our approach achieves the highest correlation with human judgments, even outperforming reference-based baselines in challenging scenarios. These results highlight the efficacy of CAF-Score for reference-free audio captioning evaluation. Code and results are available at https://github.com/inseong00/CAF-Score.
Figures & tables
Figure 1: Overview of audio captioning evaluation metrics. Traditional metrics (top left) depend on ground-truth reference captions. Although CLAPScore enables reference-free evaluation, it lacks fine-grained semantic understanding and frequently overlooks syntactic errors. Inspired by FLEUR, a vision-domain approach, the proposed CAF-Score addresses these limitations by integrating the coarse-grained semantic alignment of CLAP with the fine-grained semantic reasoning and syntactic awareness of LALMs, resulting in stronger alignment with human preference judgments.
Figure 2: Overall architecture of CAF-Score. The framework comprises two parallel branches. The CLAP-based coarse-grained semantic alignment branch applies a sliding-window strategy to the input audio and computes cosine similarity with the candidate caption using CLAP encoders; Max pooling is then used to select the most salient segment score (S-CLAPScore). The LALM-based evaluation branch assesses caption fidelity using an LALM. Rather than relying on discrete text generation, it computes a FLEUR score from token probability distributions to capture fine-grained semantic and syntactic information. The final CAF-Score is obtained as a weighted combination of the two metrics. Notably, our framework operates entirely at inference time, utilizing frozen pre-trained backbones without requiring additional training or fine-tuning.
Model (Method)
BRACE-Main
BRACE-Hallucination
AudioCaps-Main
Clotho-Main
Overall
AudioCaps-Hallu
Clotho-Hallu
Overall
HH
HM
MM
Total
HH
HM
MM
Total
Reference-free
CLAPScore (CLAP-ONLY)
MS-CLAP
64.03
48.76
55.08
53.71
67.07
70.92
65.93
67.95
61.42
72.99
85.23
80.61
LAION-CLAP
63.31
77.30
65.42
69.78
59.28
75.64
65.63
68.62
69.15
87.03
84.31
85.34
Table 1: Performance comparison on the BRACE benchmark, including BRACE-Main and BRACE-Hallucination subsets. The upper section reports reference-free metrics (standalone CLAP, LALM, and the proposed CAF-Score), while the lower section reports reference-based baselines for comparison. The Overall column represents the weighted average across subsets based on sample counts. AF3 denotes AudioFlamingo3. Within the CAF-Score results, AF3 and Qwen3 correspond to AudioFlamingo3-Base and Qwen3-Omni-Instruct, respectively. The configuration combining Qwen3-Omni-Instruct with M2D-CLAP achieves the highest overall performance.
Backbone
Pool
Clotho-Main (CAF-Score)
HH
HM
MM
Total
MS-CLAP
X
68.86
78.00
67.56
71.65
Avg
66.47
78.59
69.48
72.54
Max
67.66
79.17
70.07
73.21
LAION-CLAP
X
61.08
78.59
67.85
71.06
Avg
61.68
80.35
67.70
71.72
Table 2: Comparison of pooling strategies for CAF-Score (using Qwen3-Omni-Instruct) evaluated on Clotho-Main across different CLAP backbones. The pooling methods are: no pooling (X), average pooling (Avg), and maximum pooling (Max).
Figure 3: Performance variation of CAF-Score across different weighting parameters α on BRACE-Main.
Method
MS-CLAP
LAION-CLAP
MGA-CLAP
M2D-CLAP
Best Fixed α
0.2
0.8
0.5
0.8
Fixed α
70.11
72.24
71.32
73.11
Adaptive α
70.03
71.99
71.15
69.99
Table 3: Comparison of optimal fixed α versus entropy-based adaptiveα on BRACE-Main.
Figure 4: Tie rates in raw scores for AudioFlamingo3 (AF3) and Qwen3-Omni models.
Figure 5: LALM-driven Correction on TIKTOK_1.wav .
Figure 6: CLAP-driven Stability on Cafeteria Ambience.wav .
Figure 7: Double Failure on lama2.wav .
Figure 8: Insufficient Calibration on 105166.wav .
Figure 9: Over-calibration on 108925.wav .
Model
RELATE
PAM
LCC
SRCC
KTAU
LCC
SRCC
KTAU
RELATE [ 28 ]
0.385
0.383
0.265
-
-
-
RELATE w/ CB [ 28 ]
0.377
0.374
0.259
-
-
-
CLAPScore (CLAP-ONLY)
MS-CLAP
0.256
0.224
0.153
0.345
0.312
0.215
LAION-CLAP
0.404
0.386
0.268
0.461
0.452
0.314
Table 4: Full performance comparison on TTA benchmarks including RELATE and PAM datasets. We report Pearson (LCC), Spearman (SRCC), and Kendall’s Tau (KTAU) correlations with human judgments.
Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.
X-LANCE Lab, Shanghai Jiao Tong University, China · SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China +1