Organizations: College of Computer Science and Technology, National University of Defense Technology, Changsha, China. · State Key Laboratory of Complex & Critical Software Environment
Few-shot audio classification requires audio--language models to distinguish variable acoustic events from limited labeled data. PALM's main formulation adapts encoded class representations without changing pretrained encoders, but its class representations remain input-independent and classification uses audio features without class-conditioned refinement. We introduce DF-PALM, a dynamic feature-space prompting framework that combines audio-conditioned class representations with asymmetric cross-attention for audio refinement. Class representations are conditioned after text encoding, held fixed across fusion layers for each input, and reused for final scoring. Both pretrained encoders remain frozen, avoiding audio-conditioned text-encoder passes. With PENGI as the frozen backbone, the main 16-shot evaluation covered 11 benchmarks and six baselines. DF-PALM achieved 80.69% mean accuracy and 80.32% macro-averaged F1, exceeding PALM by 4.11 and 4.30 percentage points, respectively. Paired benchmark-level tests supported both gains after Holm correction. Across four shot settings, DF-PALM achieved the highest mean accuracy and macro-averaged F1 among the four evaluated prompt-learning methods. The gains extended to LAION-CLAP and OpenFLAM, with significant improvements over PALM in both metrics. At 50 candidate classes, operator-counted inference floating-point operations increased by 0.39% relative to PALM. These results support instance-conditioned feature-space adaptation as an effective approach to few-shot audio classification. Our code is publicly available at GitHub https://github.com/sunyulin0421/DF-PALM.
Figures & tables
Fig. 1: Cross-benchmark mean accuracy and Macro-F1 (%) of the five methods shown across 11 audio classification benchmarks in the main 16-shot setting. Stars mark the highest mean for each metric among the methods shown.
Method
Learned
Opt. Space
Context
Time
Zero-Shot
✗
N/A
Manual
N/A
CoOp
✓
Input
Static
1 h 28 min
CoCoOp
✓
Input
Dynamic
5 h 51 min
PALM
✓
Feature
Static
1 h 24 min
DF-PALM
✓
Feature
Dynamic
1 h 27 min
TABLE I: Comparison of Prompt-Learning Paradigms. Learned indicates whether prompt context is optimized; Opt. Space denotes the optimization space. Static/Dynamic indicate whether context depends on the current audio input. Times were measured in the matched ESC-50 8-shot setting.
Fig. 2: Overview of DF-PALM. (A) Frozen encoders produce global audio and base class representations; Mean Pool denotes the backbone’s aggregation into a global audio vector. (B) Meta_Net denotes the dynamic context generator gϕ , and Dynamic Text Guidance denotes the conditioned class matrix T(x) . Class-specific mixing follows Equation ( 4 ). (C) Asymmetric fusion refines the audio representation against this matrix. (D) Normalized similarity scoring uses the logit scale in Equation ( 9 ).
Panel A: Zero-Shot and Input-Space Prompt Learning
Benchmark
Zero-Shot
CoOp
CoCoOp
–
S-0
S-1
S-2
AVG
S-0
S-1
S-2
AVG
Beijing-Opera
28.81
93.23
96.60
96.19
95.34
95.77
98.30
99.16
97.74
CREMA-D
23.10
31.30
41.97
27.60
33.62
25.39
33.58
31.56
30.18
ESC50-Actions
65.25
96.25
94.00
95.50
95.25
96.31
96.20
96.48
96.33
ESC-50
49.65
94.10
93.90
93.45
93.82
94.60
93.70
94.50
94.27
TABLE II: Complete Accuracy Results (%) Across 11 Audio Classification Benchmarks in the Main 16-Shot Setting. Zero-Shot uses no support samples; FS-AVG and ProKeR require no gradient-based training. S-0–S-2 denote matched seeds, including support subsets, and AVG denotes their mean. Folded benchmarks are averaged within each seed; AVERAGE is the unweighted cross-benchmark mean. Bold marks the highest mean across all seven methods in each row.
Metric
PALM
DF-PALM
Δ [95% CI]
padj
Accuracy
76.58
80.69
+4.11[2.90,5.37]
1.95×10−3
Macro-F1
76.02
80.32
+4.30[2.99,5.67]
1.95×10−3
TABLE III: Paired 16-Shot Comparison Between DF-PALM and PALM Across 11 Benchmarks. Means weight benchmarks equally after fold and seed aggregation. Δ (percentage points) and 95% bootstrap percentile intervals use benchmark-level DF-PALM minus PALM differences. padj denotes Holm-adjusted exact two-sided Wilcoxon signed-rank p -values.
Fig. 3: Cross-benchmark performance across shot settings. Panels (a) and (b) show mean accuracy and Macro-F1 for four prompt-learning methods; panels (c) and (d) show paired DF-PALM-minus-PALM differences. Markers denote mean benchmark-level differences, error bars denote 95% bootstrap percentile intervals from 200,000 resamples, and dotted lines mark zero. Scores use fold and seed aggregation with equal benchmark weighting; 16-shot points reproduce the main experiment.
Method
Adaptation space
Dynamic class
Cross-modal
Trainable parameters
Acc.
Macro-F1
representation
fusion
PALM
Encoder output
No
No
Prompt only
76.58
76.02
Linear Probe
Frozen audio output
No
No
Linear head
77.29
76.38
Static-PALM+MLP
Encoder output
No
No
12.60M audio MLP
79.47
79.11
PALM+Audio-LoRA
Audio encoder internal
No
No
141,312 LoRA + prompt
79.27
78.93
PALM+ASPL ∗
Audio encoder internal
No
No
704 ASPL + prompt
75.57
75.08
TABLE IV: Capacity and Adaptation Controls in the 16-Shot Setting (%). All methods use the same training settings. Values are unweighted means across 11 benchmarks.
Fig. 4: Cross-benchmark internal-validation performance across cross-attention depth L and head count H in the 16-shot setting. Scores summarize four-fold support-set validation, aggregated across dataset folds and seeds with equal benchmark weighting. Configuration selection is performed independently within each support set. The left and right panels show mean accuracy and Macro-F1, respectively. Red outlines mark L=3 , H=4 , selected in every independent run.
Metric
PALM
DF-PALM
Δ
95% CI
padj
LAION-CLAP
Accuracy
75.50
76.87
+1.37
[0.71, 1.98]
0.0098
Macro-F1
75.13
76.48
+1.35
[0.82, 1.87]
0.0088
OpenFLAM
Accuracy
76.94
77.83
+0.90
[0.55, 1.31]
0.0078
Macro-F1
76.88
77.62
+0.74
[0.43, 1.08]
0.0098
TABLE V: Cross-Backbone Comparison at 16 Shots. Scores are unweighted means across 11 benchmarks (%). Differences and 95% bootstrap intervals are percentage points. Holm correction covers all four tests.
Method Variant
Mean Accuracy
Macro-F1
PALM (Baseline)
76.58
76.02
Dynamic Context only
79.27
78.72
Asymmetric Fusion only
78.89
78.46
Joint Attention
78.31
76.83
DF-PALM (Full)
80.69
80.32
TABLE VI: Cross-Benchmark Component Ablations and Fusion Control (%).
Method
Params (M)
FLOPs
Latency (ms)
Memory
Total
(G)
Mean ± SD
(MB)
PALM
98.216
165.348
25.57±2.60
448.26
DF-PALM
110.950
165.990
28.00±2.66
497.40
TABLE VII: End-to-End Inference Resource Profile at C=50 and a Batch Size of One on an NVIDIA A40 GPU.
Fig. 5: Module-level scaling on an NVIDIA RTX 4090 GPU: (a) median latency and (b) peak allocated memory. The frozen PENGI backbone was excluded.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Year
Domain/task
Classes
Split protocol
Instrument and Music Classification
Beijing-Opera
2014
Instrument recognition
4
5-fold CV
NS-Instruments
2017
Instrument recognition
10
Archived train/test
GT-Music-Genre
2012
Music genre classification
10
Archived train/test
Environmental, Urban, and Event Audio
ESC-50
2015
Environmental sound classification
50
5-fold CV
Appendix
TABLE VIII: Characteristics and evaluation split structure of the 11 benchmarks. Year denotes the publication year of the benchmark reference cited in this paper; CV denotes cross-validation.
Benchmark
Zero-Shot
CoOp
CoCoOp
FS-AVG
ProKeR
PALM
DF-PALM
Beijing-Opera
13.21
95.21
97.63
99.85
95.98
95.23
99.56
CREMA-D
17.20
25.81
26.00
29.81
31.16
27.83
35.61
ESC50-Actions
61.38
95.13
96.57
98.91
98.46
96.54
99.00
ESC-50
44.78
93.68
94.10
96.92
96.76
95.83
97.10
GT-Music-Genre
28.07
70.93
74.74
79.55
78.15
80.20
81.25
NS-Instruments
26.00
59.25
61.52
66.79
73.16
64.38
72.88
Appendix
TABLE IX: Complete Mean Macro-F1 Results (%) in the Main 16-Shot Setting. Means average folds within each seed and then three matched seeds. AVERAGE weights benchmarks equally; bold marks the highest mean in each row.
Omitted Benchmark
Accuracy
Macro-F1
DF-PALM
ΔPALM
ΔProKeR
ΔFS-AVG
DF-PALM
ΔPALM
ΔProKeR
ΔFS-AVG
Beijing-Opera
78.80
+4.09
+1.41
+5.13
78.39
+4.29
+1.86
+5.43
CREMA-D
84.55
+3.77
+1.41
+4.56
84.79
+3.95
+1.77
+4.82
ESC50-Actions
78.86
+4.28
+1.73
+5.09
78.45
+4.48
+2.17
+5.39
ESC-50
79.04
+4.39
+1.75
+5.08
78.64
+4.60
+2.19
+5.38
GT-Music-Genre
80.64
+4.40
+1.53
+4.90
80.22
+4.62
+1.91
+5.23
Appendix
TABLE X: Leave-One-Benchmark-Out Robustness at 16 Shots. DF-PALM scores (%) are unweighted means over the ten retained benchmarks. Δ denotes DF-PALM minus the indicated reference in percentage points.
Benchmark
Dynamic Only
Joint Attention
Acc.
F1
Acc.
F1
Beijing-Opera
98.73
98.71
88.40
87.67
CREMA-D
40.63
34.02
43.68
28.98
ESC50-Actions
98.42
98.40
98.75
98.74
ESC-50
96.68
96.60
97.12
97.02
GT-Music-Genre
81.17
81.10
81.83
82.07
Appendix
TABLE XI: Benchmark-Level Component and Fusion Controls at 16 Shots. Values are mean Accuracy and Macro-F1 (%). DF-PALM results appear in Table II and Table IX .
Reference
Metric
Δ [95% CI]
padj
PALM
Accuracy
+4.28[2.97,5.61]
0.0039
Macro-F1
+4.48[3.08,5.93]
0.0039
ProKeR
Accuracy
+1.73[0.43,3.05]
0.0840
Macro-F1
+2.17[0.77,3.58]
0.0547
FS-AVG
Accuracy
+5.09[1.72,9.12]
0.0391
Macro-F1
+5.39[1.87,9.53]
0.0391
Appendix
TABLE XII: Paired Sensitivity Analysis Excluding ESC50-Actions at 16 Shots. Δ denotes DF-PALM minus reference differences in percentage points over the ten retained benchmarks. Intervals are bootstrap 95% CIs; Holm correction covers both metrics within each method pair.
Benchmark
Accuracy
Macro-F1
4-Shot
8-Shot
32-Shot
4-Shot
8-Shot
32-Shot
Beijing-Opera
-1.97
+0.42
0.00
-1.96
+0.39
0.00
CREMA-D
-1.21
+2.69
-0.18
-0.73
+1.44
+0.56
ESC50-Actions
+2.08
+0.67
+0.42
+2.36
+0.68
+0.43
ESC-50
+1.03
+0.93
+0.35
+1.07
+0.91
+0.36
GT-Music-Genre
+1.83
-0.67
+1.17
+1.58
-1.44
+0.98
Appendix
TABLE XIII: Benchmark-Level DF-PALM Minus PALM Differences at Additional Shot Settings. Values are percentage points after fold and matched-seed aggregation; the mean weights benchmarks equally.
Benchmark
LAION-CLAP
OpenFLAM
Δ Acc.
Δ F1
Δ Acc.
Δ F1
Beijing-Opera
+0.71
+0.75
-0.003
-0.001
CREMA-D
-0.94
+0.06
+2.53
+1.34
ESC50-Actions
+0.75
+0.76
+0.50
+0.42
ESC-50
+1.37
+1.39
+0.55
+0.57
GT-Music-Genre
+0.33
-0.10
+0.83
+0.59
Appendix
TABLE XIV: Benchmark-Level DF-PALM Minus PALM Differences on the Additional Backbones at 16 Shots. Values are percentage points; the mean weights all 11 benchmarks equally.
Audio-Language Models (ALMs) have shown remarkable success in zero-shot audio classification by aligning audio waveforms with text. Recent efforts to improve downstream performance focus on learning optimal text prompts. However, previous approaches focus on the text encoder, leaving the potential of learnable prompts within the audio encoder unexplored. In this paper, we propose a novel framework that introduces trainable prompts into the audio encoder to capture task-specific acoustic features. We demonstrate that integrating audio-side prompt learning with existing text-side approaches enhances few-shot adaptation. Through extensive experiments across 11 datasets show that integrating our method as a plug-and-play module alongside existing text prompt tuning generally leads to performance improvements. These findings suggest that explicitly modulating the audio representation space effectively complements text-only prompting approaches. The code is available at https://github.com/hyebin-c/aspl.
Hyebin Cho, Jaehyuk Jang, Changick Kim +1
Korea Advanced Institute of Science and Technology, South Korea
Contrastive audio-language models such as CLAP enable zero-shot audio classification: a sound is labelled by matching its embedding to text prompt embeddings, with no labelled audio. This matching breaks down under acoustic noise, where accuracy and mAP fall by 12-30 percentage points at 0 dB SNR on standard benchmarks. We propose Drift Augmented Scoring (DAS), a small per-class bonus added to the cosine score. The bonus rewards a class when the noisy audio embedding drifts in the direction that the class's noise-conditioned text prompts predict. It is derived from text alone, computed once and cached, and adds a single inner product per class at inference, with no gradients and no test-time batch. On a LAION CLAP backbone, we compare DAS against the four variants of Acevedo et al.'s concurrent method on UrbanSound8K and the full FSD50K eval set, mixing each clip with urban acoustic scene noise across a range of SNRs. DAS improves the metric on every test condition: by +2.60 to +5.75 accuracy points on UrbanSound8K and +1.50 to +1.74 mAP points on FSD50K.
Fully few-shot class-incremental audio classification (FFCAC) requires recognizing new sound classes from only a handful of labeled examples per session, without forgetting previously learned classes and without any large base dataset. Existing methods typically freeze a pre-trained audio--language encoder and classify with point prototypes, but they suffer from significant performance degradation throughout the sessions due to generic feature representations. We propose SPECTRA, a framework built on a frozen encoder which adds three components. (i) a lightweight trainable adapter that calibrates the generic embeddings to the task; (ii) subspace feature replay, an exemplar-free anti-forgetting scheme that replays old classes by sampling from the low-rank subspace of their stored features; and (iii) a transductive optimal-transport refinement of prototypes at test time. Our central finding is that the subspace structure of the replay diminishes forgetting and outperforms naive Gaussian replay of equal variance. On three FFCAC benchmarks (NSynth-100, FSC-89, LS-100), SPECTRA improves average accuracy and reduces forgetting over current state-of-the-art methods, and our ablations statistically validate each component.
Giries Abu Ayoub, Loay Mualem, Simon Korman
Department of Computer Science, University of Haifa · Institute for AI, University of Stuttgart · IMPRS-IS