Audio-Text Foundation Models (ATMs) fail catastrophically under severe acoustic noise, yet existing adaptation strategies either rely on gradient-based Test-Time Adaptation (TTA), which reinforces noise rather than signal, or on prompt tuning that requires privileged noise annotations unavailable at inference. We address these failures with PRISM (Prototype-Rectified Iterative Self-supervised Manifold Denoising), a training-free, source-free TTA framework grounded in the Affine Noise Hypothesis: severe acoustic noise induces a low-rank affine shift in the multimodal latent space, with more than 90% of distortion energy confined to the leading 60 principal components. PRISM estimates and reverses this distortion from an unlabeled target batch using frozen text prototypes as geometric anchors via three closed-form geometric corrections compiled into a single static projection matrix by Affine Bias Regression. At inference, adaptation reduces to one matrix-vector multiplication in 0.0009 ms, making it substantially faster than gradient-based TTA while requiring no additional training. On UrbanSound8K, PRISM improves over the zero-shot baseline by 12.94 percentage points and surpasses an oracle-assisted TTA baseline by 9.41 percentage points, despite never observing its privileged augmented noise prompts. We further identify the Polyphonic Trap, a principled failure mode of subspace deflation for broadband classes, and resolve it via Confidence-Aware Regression (CAR), recovering up to 8.16 percentage points for the worst-affected class.
Figures & tables
Figure 1. Overview of PRISM. (a) ContextDA (top-left) requires privileged oracle prompts, whereas PRISM (bottom-left) compiles geometric corrections into a static projection matrix W for fully blind adaptation. (b) Top-right: PRISM satisfies all key desiderata for edge inference. (c) Bottom-right: PRISM achieves higher accuracy with substantially lower inference latency than the strongest-performing baseline. Overview figure. Left: pipeline comparison showing ContextDA needing oracle noise prompts vs. PRISM doing blind adaptation. Top right: a table comparing key properties. Bottom right: scatter plot of accuracy vs. inference latency, showing PRISM is both faster and more accurate.
Figure 2. Empirical validation of the Affine Noise Hypothesis: Cumulative Singular Value energy of noise-induced distortion across four environments. Dashed lines denote retained rank K , while horizontal guides indicate 90% energy. Four cumulative energy curves, one per noise background. Each rises steeply and crosses the 90% threshold near rank 60, supporting the idea that noise distortion lives in a small low-rank subspace.
Figure 3. Proposed Framework. Phase 1: Noisy audio embeddings En are corrected via three closed-form geometric operations (OPCA, CCVD, and Per-Class Residual Translation) using frozen text prototypes T as semantic anchors. Corrections are compiled into a static projection matrix W via Affine Bias Regression (ABR). Phase 2: The frozen W adapts each future sample via a single matrix multiplication without requiring gradients, batch statistics, or parameter updates. Two-phase diagram. Left shows the calibration loop: noisy audio embeddings are corrected via OPCA, CCVD, and per-class translation, then distilled into a single projection matrix W. Right shows inference: a new sample is adapted by multiplying by W and classified against text prototypes.
US8K
ESC-50
DCASE ‡
Method
Acc
F1
Acc
Acc
Zero-Shot Baseline
LAION-CLAP ( Wu et al., 2023 )
58.77
60.48
88.82
17.36
Fully Blind Transductive (Test-Time Adaptation)
TDA ( Karmanov et al., 2024 )
62.75
64.35
91.10
17.38
PCA++ ( Wu et al., 2025a )
67.88
68.57
77.57
15.10
Table 1. Comparison of PRISM against SoTA methods across multiple benchmarks. Best results are bold ; second-best are underlined . † ContextDA uses oracle noise-type prompts unavailable for ESC-50/DCASE. ‡ DCASE violates the Affine Noise Hypothesis via device mismatch rather than additive noise, evaluating robustness under assumption violation. Table comparing accuracy of PRISM and baseline methods across US8K, ESC-50, and DCASE datasets.
Background
Zero-Shot
SubTTA
PCA++
TDA
PRISM + CAR
PRISM
shopping_mall
49.43
49.19
64.67
55.19
68.47
68.53
street_ped.
53.05
52.95
66.81
58.37
71.17
71.50
public_square
54.11
54.25
66.56
58.54
70.66
70.85
airport
56.02
56.16
67.02
60.89
70.76
70.67
metro_station
58.38
58.49
69.38
62.57
71.15
71.34
street_traffic
60.54
60.97
67.25
64.19
71.76
71.87
Table 2. Per-environment accuracy (%) on US8K (10-fold CV). Table showing accuracy breakdown for 10 individual noise environments on US8K.
Background
Zero-Shot
SubTTA
PCA++
TDA
PRISM + CAR
PRISM
shopping_mall
86.45
86.20
74.40
88.95
91.40
91.60
street_ped.
86.70
86.90
74.20
88.90
92.65
92.60
public_square
87.70
88.05
78.70
90.40
92.60
92.70
airport
86.35
86.05
75.25
88.90
91.55
91.60
metro_station
89.30
89.10
80.50
91.30
92.95
93.05
street_traffic
89.00
89.20
81.50
91.70
93.70
94.10
Table 3. Per-environment accuracy (%) on ESC-50 (5-fold CV). Table showing accuracy breakdown for 10 individual noise environments on ESC-50.
SNR
Zero-Shot
SubTTA
PCA++
TDA
PRISM + CAR
PRISM
− 6 dB
32.69
33.02
37.55
37.47
54.03
57.45
− 3 dB
49.70
49.99
56.56
54.32
64.76
66.49
0 dB
64.85
64.77
70.62
68.28
74.53
74.68
3 dB
72.38
72.33
76.76
75.45
80.29
81.24
6 dB †
75.54
75.43
77.80
78.34
82.71
83.79
8 dB †
76.59
76.53
77.58
79.17
83.65
84.78
Table 4. Accuracy (%) on US8K across varying SNR values. Red-shaded rows denote low-SNR regimes where noise dominates foreground events; † marks moderate noise levels, and ⋆ indicates anomalous performance drops. Table showing accuracy across different SNR levels from -6 dB to clean on US8K.
Figure 4. Operational Robustness of PRISM. (a) Mean gain over zero-shot in the severe regime and in moderate conditions. (b) Accuracy vs. input SNR on US8K. (c) Accuracy vs. test batch size N . PRISM calibrates its projection matrix once offline. Three side-by-side plots. Left: average accuracy gains for severe vs. moderate noise conditions. Middle: accuracy vs. SNR from -6 dB to clean. Right: accuracy vs. test batch size, showing PRISM holds up even for small batches.
Figure 5. Diagnostic Analysis of PRISM. Left: Per-class accuracy vs. zero-shot CLAP. Spectrally sparse classes gain substantially, confirming CCVD separates noise from signal for impulsive classes. Right: Cross-environment heatmap ( Δ Acc = PRISM − ZS). Consistent improvement across 8 classes validates the Affine Noise Hypothesis. Two panels. Left: a bar chart of per-class accuracy showing most classes improve under PRISM, but street music drops. Right: a heatmap of per-class accuracy change across all 10 noise environments, confirming that street music degrades uniformly regardless of background.
Figure 6. Sensitivity Analysis on US8K. Left (a,b): Overall accuracy per method and incremental component ablation. Right (c–e): Hyperparameter sensitivity across Noise Subspace rank K∈[30,100] , Shift Strength α∈[0,1] , and Ridge Penalty λ∈[10−3,10−1] .
Config
Component Added
Acc (%)
F1 (%)
Zero-Shot
—
58.77
60.48
+OPCA
Cross-modal alignment
68.35
68.87
+CCVD
Noise subspace removal
71.44
72.02
PRISM
Affine bias correction
71.71
72.36
PRISM + CAR
Polyphonic mitigation
71.23
71.82
Table 5. Incremental component ablation on US8K. Each row adds one component to the previous. Table showing accuracy and F1 score for incremental additions of OPCA, CCVD, and ABR.
Method
Precomp.
Inference
Grad.
Params
CoNMix
N/A
∼ 50 ms
✓
Millions
TDA
∼ 5 ms/samp
∼ 2 ms
×
Cache
PCA++
∼ 80 ms
∼ 0.008 ms
×
None
PRISM
311 ms
0.0009 ms
×
None
Table 6. Computational cost comparison across SoTA. Table comparing precomputation time, inference time, and parameter counts across methods.
Class
Zero-Shot
PRISM + CAR
PRISM
air_conditioner
22.06
58.35
64.77
gun_shot
60.80
86.36
92.14
children_playing
60.50
80.45
82.80
car_horn
65.62
82.63
85.36
drilling
30.97
49.42
50.60
engine_idling
45.55
62.01
59.95
Table 7. Per-class accuracy (%) on US8K averaged across all 10 backgrounds. PRISM + CAR partially mitigates the Polyphonic Trap for street music via confidence-aware regression. Table showing per-class accuracy across 10 classes on US8K for Zero-Shot, PRISM+CAR, and PRISM.
Few-shot parameter-efficient adaptation of pretrained Audio--Language Models (ALMs) often improves seen-class performance at the cost of unseen-class generalization, leading to the base-to-new trade-off. We study this failure through zero-shot drift in the text embedding space: few-shot tuning can distort inter-class structure and move adapted embeddings away from their pretrained anchors. We therefore propose Subspace Tuning (SubT), an embedding-space adaptation method that combines a geometry-aware shared transformation with anchoring to the zero-shot prototypes. The learned transformation is transferred to unseen classes, with subspace-aware gating to mitigate negative transfer. Across 11 audio benchmarks, SubT achieves strong few-shot generalization while operating directly on precomputed text embeddings without text-encoder backpropagation. Code is available at https://github.com/jhyukjang/SubT.
Large audio-language models (LALMs) are sensitive to input perturbations, such as noise, waveform corruption, and adversarial injections. We propose AnchorPrompt, an efficient adaptation method that keeps the model frozen and learns a single block of prompt vectors inserted at the decoder input, between the audio and question embeddings. We train these vectors through self-distillation over diverse audio and text perturbations. To improve answer consistency and mitigate hallucination, we use the model's prediction on the clean recording as the target for answerable inputs, and assign a refusal target when the audio lacks sufficient evidence to answer. Furthermore, AnchorPrompt is perturbation-agnostic at inference, requiring no prior detection of perturbations and enabling zero-shot transfer to unseen distortions. We evaluate three LALMs across three benchmarks and show that AnchorPrompt improves answer consistency in most tested conditions. Clean accuracy improves in six of nine model-benchmark pairs, with minimal impact on the remainder of 1.2% at most. Crucially, AnchorPrompt reduces hallucinations under severe audio corruption while keeping false refusals on clean audio rare. Finally, these consistency gains transfer to unseen perturbations, such as choice permutations and reverberation.
Pooneh Mousavi, Amir Ivry, Mirco Ravanelli +1
Concordia University · Mila – Quebec AI Institute · Technion – IIT +1
Speech LLM post-training increasingly relies on efficient cross-modal alignment and robust low-resource adaptation, yet collecting large-scale audio-text pairs remains costly. Text-only alignment methods such as TASU reduce this burden by simulating CTC posteriors from transcripts, but they provide limited control over uncertainty and error rate, making curriculum design largely heuristic. We propose TASU2, a controllable CTC simulation framework that simulates CTC posterior distributions under a specified WER range, producing text-derived supervision that better matches the acoustic decoding interface. This enables principled post-training curricula that smoothly vary supervision difficulty without TTS. Across multiple source-to-target adaptation settings, TASU2 improves in-domain and out-of-domain recognition over TASU, and consistently outperforms strong baselines including text-only fine-tuning and TTS-based augmentation, while mitigating source-domain performance degradation.
Jing Peng, Chenghao Wang, Yi Yang +5
Jiangsu Key Lab of Language Computing · AISpeech Ltd, Suzhou, China · Nanjing University, China