Prototype-Rectified Iterative Self-supervised Manifold Denoising under Severe Acoustic Shift

Ashish Anand Shukla; Rini Smita Thakur; Aryan Das; Vinod K. Kurmi

cs.SDAug 15, 2026

Prototype-Rectified Iterative Self-supervised Manifold Denoising under Severe Acoustic Shift

Authors: Ashish Anand Shukla, Rini Smita Thakur, Aryan Das, Vinod K. Kurmi

Organizations: Indian Institute of Science Education and Research Bhopal, India · Vellore Institute of Technology Bhopal, India

Abstract

Audio-Text Foundation Models (ATMs) fail catastrophically under severe acoustic noise, yet existing adaptation strategies either rely on gradient-based Test-Time Adaptation (TTA), which reinforces noise rather than signal, or on prompt tuning that requires privileged noise annotations unavailable at inference. We address these failures with PRISM (Prototype-Rectified Iterative Self-supervised Manifold Denoising), a training-free, source-free TTA framework grounded in the Affine Noise Hypothesis: severe acoustic noise induces a low-rank affine shift in the multimodal latent space, with more than 90% of distortion energy confined to the leading 60 principal components. PRISM estimates and reverses this distortion from an unlabeled target batch using frozen text prototypes as geometric anchors via three closed-form geometric corrections compiled into a single static projection matrix by Affine Bias Regression. At inference, adaptation reduces to one matrix-vector multiplication in 0.0009 ms, making it substantially faster than gradient-based TTA while requiring no additional training. On UrbanSound8K, PRISM improves over the zero-shot baseline by 12.94 percentage points and surpasses an oracle-assisted TTA baseline by 9.41 percentage points, despite never observing its privileged augmented noise prompts. We further identify the Polyphonic Trap, a principled failure mode of subspace deflation for broadband classes, and resolve it via Confidence-Aware Regression (CAR), recovering up to 8.16 percentage points for the worst-affected class.

Figures & tables

Panel of Figure 1 — Figure 1. Overview of PRISM. (a) ContextDA (top-left) requires privileged oracle prompts, whereas PRISM (bottom-left) compiles geometric corrections into a static projection matrix $\mathbf{W}$ for fully blind adaptation. (b) Top-right: PRISM satisfies all key desiderata for edge inference. (c) Bottom-right: PRISM achieves higher accuracy with substantially lower inference latency than the strongest-performing baseline. Overview figure. Left: pipeline comparison showing ContextDA needing oracle noise prompts vs. PRISM doing blind adaptation. Top right: a table comparing key properties. Bottom right: scatter plot of accuracy vs. inference latency, showing PRISM is both faster and more accurate.

Figure 2. Empirical validation of the Affine Noise Hypothesis: Cumulative Singular Value energy of noise-induced distortion across four environments. Dashed lines denote retained rank

K

, while horizontal guides indicate 90% energy. Four cumulative energy curves, one per noise background. Each rises steeply and crosses the 90% threshold near rank 60, supporting the idea that noise distortion lives in a small low-rank subspace.

Figure 3. Proposed Framework. Phase 1: Noisy audio embeddings

E_{n}

are corrected via three closed-form geometric operations (OPCA, CCVD, and Per-Class Residual Translation) using frozen text prototypes

T

as semantic anchors. Corrections are compiled into a static projection matrix

W

via Affine Bias Regression (ABR). Phase 2: The frozen

W

adapts each future sample via a single matrix multiplication without requiring gradients, batch statistics, or parameter updates. Two-phase diagram. Left shows the calibration loop: noisy audio embeddings are corrected via OPCA, CCVD, and per-class translation, then distilled into a single projection matrix W. Right shows inference: a new sample is adapted by multiplying by W and classified against text prototypes.

Zero-Shot Baseline
	US8K		ESC-50	DCASE ‡
Method	Acc	F1	Acc	Acc
LAION-CLAP ( Wu et al., 2023 )	58.77	60.48	88.82	17.36
Fully Blind Transductive (Test-Time Adaptation)
TDA ( Karmanov et al., 2024 )	62.75	64.35	91.10	17.38
PCA++ ( Wu et al., 2025a )	67.88	68.57	77.57	15.10

Table 1. Comparison of PRISM against SoTA methods across multiple benchmarks. Best results are bold ; second-best are underlined . † ContextDA uses oracle noise-type prompts unavailable for ESC-50/DCASE. ‡ DCASE violates the Affine Noise Hypothesis via device mismatch rather than additive noise, evaluating robustness under assumption violation. Table comparing accuracy of PRISM and baseline methods across US8K, ESC-50, and DCASE datasets.

Background	Zero-Shot	SubTTA	PCA++	TDA	PRISM + CAR	PRISM
shopping_mall	49.43	49.19	64.67	55.19	68.47	68.53
street_ped.	53.05	52.95	66.81	58.37	71.17	71.50
public_square	54.11	54.25	66.56	58.54	70.66	70.85
airport	56.02	56.16	67.02	60.89	70.76	70.67
metro_station	58.38	58.49	69.38	62.57	71.15	71.34
street_traffic	60.54	60.97	67.25	64.19	71.76	71.87

Table 2. Per-environment accuracy (%) on US8K (10-fold CV). Table showing accuracy breakdown for 10 individual noise environments on US8K.

Background	Zero-Shot	SubTTA	PCA++	TDA	PRISM + CAR	PRISM
shopping_mall	86.45	86.20	74.40	88.95	91.40	91.60
street_ped.	86.70	86.90	74.20	88.90	92.65	92.60
public_square	87.70	88.05	78.70	90.40	92.60	92.70
airport	86.35	86.05	75.25	88.90	91.55	91.60
metro_station	89.30	89.10	80.50	91.30	92.95	93.05
street_traffic	89.00	89.20	81.50	91.70	93.70	94.10

Table 3. Per-environment accuracy (%) on ESC-50 (5-fold CV). Table showing accuracy breakdown for 10 individual noise environments on ESC-50.

SNR	Zero-Shot	SubTTA	PCA++	TDA	PRISM + CAR	PRISM
$-$ 6 dB	32.69	33.02	37.55	37.47	54.03	57.45
$-$ 3 dB	49.70	49.99	56.56	54.32	64.76	66.49
0 dB	64.85	64.77	70.62	68.28	74.53	74.68
3 dB	72.38	72.33	76.76	75.45	80.29	81.24
6 dB †	75.54	75.43	77.80	78.34	82.71	83.79
8 dB †	76.59	76.53	77.58	79.17	83.65	84.78

Table 4. Accuracy (%) on US8K across varying SNR values. Red-shaded rows denote low-SNR regimes where noise dominates foreground events;

{\dagger}

marks moderate noise levels, and

\star

indicates anomalous performance drops. Table showing accuracy across different SNR levels from -6 dB to clean on US8K.

Figure 4. Operational Robustness of PRISM. (a) Mean gain over zero-shot in the severe regime and in moderate conditions. (b) Accuracy vs. input SNR on US8K. (c) Accuracy vs. test batch size

N

. PRISM calibrates its projection matrix once offline. Three side-by-side plots. Left: average accuracy gains for severe vs. moderate noise conditions. Middle: accuracy vs. SNR from -6 dB to clean. Right: accuracy vs. test batch size, showing PRISM holds up even for small batches.

Figure 5. Diagnostic Analysis of PRISM. Left: Per-class accuracy vs. zero-shot CLAP. Spectrally sparse classes gain substantially, confirming CCVD separates noise from signal for impulsive classes. Right: Cross-environment heatmap ( $\Delta$ Acc = PRISM $-$ ZS). Consistent improvement across 8 classes validates the Affine Noise Hypothesis. Two panels. Left: a bar chart of per-class accuracy showing most classes improve under PRISM, but street music drops. Right: a heatmap of per-class accuracy change across all 10 noise environments, confirming that street music degrades uniformly regardless of background.

Figure 6. Sensitivity Analysis on US8K. Left (a,b): Overall accuracy per method and incremental component ablation. Right (c–e): Hyperparameter sensitivity across Noise Subspace rank

K\in[30,100]

, Shift Strength

\alpha\in[0,1]

, and Ridge Penalty

\lambda\in[10^{-3},10^{-1}]

Config	Component Added	Acc (%)	F1 (%)
Zero-Shot	—	58.77	60.48
+OPCA	Cross-modal alignment	68.35	68.87
+CCVD	Noise subspace removal	71.44	72.02
PRISM	Affine bias correction	71.71	72.36
PRISM + CAR	Polyphonic mitigation	71.23	71.82

Table 5. Incremental component ablation on US8K. Each row adds one component to the previous. Table showing accuracy and F1 score for incremental additions of OPCA, CCVD, and ABR.

Method	Precomp.	Inference	Grad.	Params
CoNMix	N/A	$\sim$ 50 ms	✓	Millions
TDA	$\sim$ 5 ms/samp	$\sim$ 2 ms	$\times$	Cache
PCA++	$\sim$ 80 ms	$\sim$ 0.008 ms	$\times$	None
PRISM	311 ms	0.0009 ms	$\times$	None

Table 6. Computational cost comparison across SoTA. Table comparing precomputation time, inference time, and parameter counts across methods.

Class	Zero-Shot	PRISM + CAR	PRISM
air_conditioner	22.06	58.35	64.77
gun_shot	60.80	86.36	92.14
children_playing	60.50	80.45	82.80
car_horn	65.62	82.63	85.36
drilling	30.97	49.42	50.60
engine_idling	45.55	62.01	59.95

Table 7. Per-class accuracy (%) on US8K averaged across all 10 backgrounds. PRISM + CAR partially mitigates the Polyphonic Trap for street music via confidence-aware regression. Table showing per-class accuracy across 10 classes on US8K for Zero-Shot, PRISM+CAR, and PRISM.

Test-Time Adaptation

Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.

See all 322 papers

SubT: Subspace Tuning for Few-shot Generalization of Audio-Language Models
Jun 17, 2026Jaehyuk Jang, Kangwook Ko, Wonjun Lee +1Few-Shot Audio Classification Zero-Shot Learning
KAIST
AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models
Sep 30, 2026Pooneh Mousavi, Amir Ivry, Mirco Ravanelli +1Audio-Language Models Prompt Tuning
Concordia University · Mila – Quebec AI Institute · Technion – IIT +1
TASU2: Controllable CTC Simulation for Alignment and Low-Resource Adaptation of Speech LLMs
Apr 9, 2026Jing Peng, Chenghao Wang, Yi Yang +5Connectionist Temporal Classification Speech Language Models
Jiangsu Key Lab of Language Computing · AISpeech Ltd, Suzhou, China · Nanjing University, China

Jun 17, 2026J/K move · Enter open · S save

SubT: Subspace Tuning for Few-shot Generalization of Audio-Language Models

Jaehyuk Jang, Kangwook Ko, Wonjun Lee, Changick Kim

KAIST

Few-Shot Audio Classification Zero-Shot Learning Few-Shot Learning Audio-Language Models

Open paper AskPDF arXiv

Few-shot parameter-efficient adaptation of pretrained Audio--Language Models (ALMs) often improves seen-class performance at the cost of unseen-class generalization, leading to the base-to-new trade-off. We study this failure through zero-shot drift in the text embedding space: few-shot tuning can distort inter-class structure and move adapted embeddings away from their pretrained anchors. We therefore propose Subspace Tuning (SubT), an embedding-space adaptation method that combines a geometry-aware shared transformation with anchoring to the zero-shot prototypes. The learned transformation is transferred to unseen classes, with subspace-aware gating to mitigate negative transfer. Across 11 audio benchmarks, SubT achieves strong few-shot generalization while operating directly on precomputed text embeddings without text-encoder backpropagation. Code is available at https://github.com/jhyukjang/SubT.

Abstract

Test-Time Adaptation

Explore similar work

SubT: Subspace Tuning for Few-shot Generalization of Audio-Language Models

AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models

TASU2: Controllable CTC Simulation for Alignment and Low-Resource Adaptation of Speech LLMs