AudioTrace: File-Level Membership Detection for Audio Large Language Models
Authors: Guangzhi Sun, Xiao Zhan, Shuai Wang, Jose Such, Phil Woodland
Organizations: University of Cambridge, Cambridge, United Kingdom · School of Intelligence Science and Technology, Nanjing University, Suzhou, P.R. China · State Key Lab. of Novel Software Technology, Nanjing University, Nanjing, P.R. China · INGENIO (CSIC-Universitat Polit`ecnica de Val`encia), Valencia, Spain
Significant privacy and copyright concerns are raised by the use of publicly available audio recordings to train audio large language models (LLMs), yet effective auditing methods remain limited. Existing membership inference attacks typically assume knowledge of complete training examples, including their associated textual supervision. In practice, however, audio recordings may be paired with privately generated transcripts, captions, or question-answer pairs that are unavailable to owners of the recordings. This motivates audio-file-level membership inference, which determines whether a recording was used for training without assuming knowledge of its training tasks and annotations. This paper provides the first systematic investigation of this challenge and introduces AudioTraceBench, a controlled benchmark comprising 1,285 audio recordings with explicit fine-tuning membership labels. Evaluations across three audio LLMs show that existing text- and video-based membership inference methods adapted to audio perform close to random guessing. To address this challenge, AudioTrace is proposed, where audio-conditioned proxy questions are generated using the target model, and membership evidence is extracted from its responses and aggregated at the recording level. Consistent improvements over existing baselines are observed, with AUROC reaching 0.60 and increasing further under increased fine-tuning exposure or adaptation capacity. Further experiments were conducted to examine the factors influencing membership signals for audio-file-level detection. These findings establish audio-file-level membership inference as a distinct and practically important auditing challenge, with AudioTrace providing an effective initial approach to this challenge.
Figures & tables
Fig. 1: AudioTrace membership inference method. AudioTrace comprises of 3 stages from left to right. It first generates proxy questions from the target audio LLM. Then, for each proxy question, the target LLM generates an answer and the membership signals, such as Min-K% and predictive entropy, are recorded. Finally, these signals are aggregated at audio-file-level using top-k average, producing one single membership score for each audio file.
Fig. 2: Data generation and curation pipeline for AudioTraceBench. We first collect YouTube videos from time-sensitive, information-rich categories and extract their audio tracks. We then filter the recordings based on content recency, information density, and duration. Finally, we divide the filtered recordings into a member set for supervised fine-tuning and a non-member set held out for membership auditing.
Fig. 3: Pie chart showing the distribution of the number of audio files across six different audio categories in AudioTraceBench.
Attack Method
Setting
Qwen2.5-Omni
SALMONN-2
Step-Audio-2-mini
AUROC
TPR@5%FPR
AUROC
TPR@5%FPR
AUROC
TPR@5%FPR
PPL
0.4957
0.0419
0.4970
0.0403
0.5207
0.0636
Entropy
0.4953
0.0419
0.4958
0.0372
0.5200
0.0698
Min-K%
Min-10%
0.4987
0.0279
0.4916
0.0388
0.5141
0.0589
Min-30%
0.4946
0.0481
0.4933
0.0310
0.5173
0.0667
Min-60%
0.4959
0.0459
0.4959
0.0372
0.5204
0.0636
TABLE I: Audio-file-level membership inference performance on AudioTraceBench , where each model is fine-tuned on the member set of AudioTraceBench. We report AUROC and TPR at 5% FPR for all metrics. Audio-SME and VideoMIA-Audio are adaptations of existing video-level MIA methods to audio introduced in § III-C .
Membership score
AUROC
TPR@5%FPR
Combined
0.5515
0.0574
ROUGE-L
0.5253
0.0731
BERT score
0.5441
0.0592
TABLE II: AudioTrace operating under black-box setting.
Method
Accuracy
Pitch and Duration
51.25%
Text Embeddings
52.77%
Acoustic Embeddings
53.32%
TABLE III: Blind baselines based on audio and content features. Binary classifiers are trained on each feature.
Setting
Metric
Qwen2.5-Omni
SALMONN-2
AUROC
TPR@5%FPR
AUROC
TPR@5%FPR
AudioTrace - Known Training Set QA for Member Audio
Original Audio
Min-60%
0.8104
0.4106
0.8216
0.4044
Max-60% Rényi-2
0.8092
0.4121
0.8206
0.3810
Combined
0.8152
0.4401
0.8234
0.4059
Original Audio + 0dB SNR Noise
Min-60%
0.5788
0.0529
0.5850
0.0871
TABLE IV: Comparison of known training set QA annotations against unknown annotation condition for Qwen2.5-Omni and SALMONN-2 models under corruptive noise and audio tokeniser reconstruction perturbations. AudioTrace is used across the table with the standard setting as before.
Model
Source of Query
Similarity
AUROC
Qwen2.5-Omni
self-gen.
0.4109
0.5515
Qwen2.5-Omni
Qwen3-Omni gen.
0.3870
0.5383
SALMONN-2
self-gen.
0.4166
0.5902
SALMONN-2
Qwen3-Omni gen.
0.3870
0.5562
TABLE V: Similarity to the training set questions and influence of the source of queries on auditing performance. The combined score is used to measure AUROC. We adopt the Qwen3-Omni-30BA3B-Instruct model to generate proxy questions as a comparison reference.
Model
M
Max-60% Rényi-2
Combined
Qwen2.5-Omni
3
0.5995
0.6209
5
0.5791
0.5973
10
0.5527
0.5669
20
0.5568
0.5527
30
0.5563
0.5515
SALMONN-2
3
0.6778
0.6822
TABLE VI: AUROC of different models using the M most similar proxy queries to the training set questions.
Fig. 4: Variation of AUROC against the number of proxy questions generated for Qwen2.5-Omni and SALMONN-2 using AudioTrace Combined score. The shaded area represents the single standard deviation range.
Fig. 5: Variation of AUROC and TPR@5%FPR with different training data composition, including QA, CapASR (caption and transcription) or the combination of the two, which is the default setting.
Fig. 6: Ablation studies on training configurations, including the number of repetitions of the samples (represented by the number of epochs), and the number of trainable parameters controlled by the LoRA rank, using (a) SALMONN-2 and (b) Qwen2.5-Omni.
Composition
Qwen2.5-Omni
SALMONN-2
Step-Audio-2-mini
Caption + ASR
0.5276
0.6037
0.5549
QA
0.5450
0.5723
0.5845
All
0.5515
0.5902
0.5952
TABLE VII: Impact of composition of the proxy questions in AudioTrace measured by AUROC across all three models. The combined score is used as the membership score. For caption + ASR, the average of both tasks is used.
Fig. 7: Variations of AUROC against different top- K aggregations on AudioTraceBench on Qwen2.5-Omni and SALMONN-2 using Max-60% Rényi-2 Entropy and the combined score.
Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.