Membership Inference Attacks against Large Audio Language Models

Jia-Kai Dong; Yu-Xiang Lin; Hung-Yi Lee

cs.SDMar 30, 2026

Membership Inference Attacks against Large Audio Language Models

Authors: Jia-Kai Dong, Yu-Xiang Lin, Hung-Yi Lee

Organizations: National Taiwan University · NTU Artificial Intelligence Center of Research Excellence

Abstract

We present the first systematic Membership Inference Attack (MIA) evaluation of LALMs. Using Multi-modal Blind Baselines based on textual, spectral and prosodic features, we demonstrate that common audio datasets exhibit near-perfect train/test separability (AUC ~ 1.0) even without model inference, thus MIA may primarily detect distribution shift. We therefore introduce a blind-baseline protocol to control for this confound. Under this protocol, we identify that the distribution-matched datasets enable reliable MIA evaluation without distribution-shift artifacts. We benchmark multiple MIA methods and conduct modality disentanglement experiments on these datasets. The results reveal that LALM memorization is cross-modal, arising only from binding a speaker's vocal identity with its text. These findings establish a principled standard for auditing LALMs beyond spurious correlations. Our codebase is available at https://github.com/snooow1029/ALM_MIA.

Figures & tables

Figure 1: Overview of the 3-phase LALM privacy auditing framework: (1) Multi-Modal Blind Baseline: Filters out samples with confounding distributional shifts to yield a clean dataset. (2) MIA Auditing: Audits the target LALM on the clean dataset using membership indicators from a two-stage generation process. (3) Modality Disentanglement: Probes confirmed memorization to identify and characterize cross-modal binding.

Dataset	MIA AUC		Blind Baseline: AUC ( $r_{AF3}/r_{MF}$ )
	AF3	MF	Metadata	Text	Acoustic
Gigaspeech	87.6	90.5	71.9 (.59/.55)	75.6 (.41/.50)	78.9 (.52/.55)
SPGISpeech	51.9	50.7	51.6 (.35/.33)	50.0 (-.01/.06)	52.0 (0.01/0.02)
Librispeech	93.8	70.9	100.0 † (.75/.35)	61.1 (.16/.11)	99.8 (.78/.34)
Tedlium	70.0	68.9	64.9 (.48/.53)	85.7 (.24/.41)	98.7 (.34/.30)
VoxPopuli	62.0	68.1	55.1 (.24/.09)	60.3 (.10/.13)	66.7 (.07/.08)

Table 1: MIA performance (AUC) vs. blind baselines across datasets. Values in parentheses report Pearson correlation (

r_{\text{AF3}}/r_{\text{MF}}

) between LALM MIA and blind indicators. High correlation suggests spurious success driven by dataset artifacts.

Figure 2: Cross-dataset MIA AUC heatmap. Diagonal: standard intra-dataset MIA (Train vs. Test). Off-diagonal: membership-neutral pairings (e.g., Dataset A Train vs. Dataset B Train), where near-perfect AUCs indicate domain shifts rather than genuine memorization.

Dataset	Original		Silence		Noise		Synthesized
	$AF3$	$MF$	$AF3$	$MF$	$AF3$	$MF$	$AF3$	$MF$
VoxPopuli	62.0 ∗	68.1 ∗	52.1	51.6	53.9	51.6	$48.4_{\pm 2.1}$	$49.8_{\pm 1.0}$
SPGISpeech	51.9 ∗	50.7 ∗	49.4	50.3	50.3	51.2	$49.8_{\pm 0.2}$	$49.4_{\pm 0.6}$
Clotho	52.4 ∗	48.9	51.2	49.2	49.2	51.7	$47.9_{\pm 0.6}$	$50.0_{\pm 1.0}$
Nsynth	62.3 ∗	62.1 ∗	50.0	48.1	48.8	49.0	-	-
∗ Statistical significance ( $p<0.05$ ) comparing Original to all decoupled conditions under a paired $t$ -test.

Table 2: Modality Disentanglement (Sample-level AUC) on the four datasets identified as relatively “clean” via our Phase 1 Blind Baseline protocol. Results show a consistent collapse in MIA performance when acoustic or textual context is disrupted across both models. Subscripts in the Synthesized column denote standard deviation (

\pm

SD) across multiple realizations.

Membership Inference Attacks

Membership inference attacks test whether records or entities were used to train a machine-learning model, revealing potential privacy leakage.

See all 116 papers

OLMo-Detect: A Multi-Stage, Confounder-Controlled Benchmark for Membership Inference on Large Language Models
Oct 2, 2026Tao Shi, Chaoyi Xiang, Qiongkai Xu +1Membership Inference Attacks Privacy Leakage in Language Models
The University of Melbourne · Macquarie University
Detecting Data Contamination in Large Language Models
Apr 21, 2026Juliusz Janicki, Savvas Chamezopoulos, Evangelos Kanoulas +1Membership Inference Attacks LLM Auditing
University of Amsterdam · Elsevier
CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models
Jun 16, 2026Jeffrey G. Wang, Jason Wang, Marvin Li +1LLM Evaluation Membership Inference Attacks
Harvard University · Harvard Business School

Oct 2, 2026J/K move · Enter open · S save

OLMo-Detect: A Multi-Stage, Confounder-Controlled Benchmark for Membership Inference on Large Language Models

Tao Shi, Chaoyi Xiang, Qiongkai Xu, Jey Han Lau

The University of Melbourne · Macquarie University

Membership Inference Attacks Privacy Leakage in Language Models LLM Auditing

Open paper AskPDF arXiv

Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without access to its training corpus. Despite recent progress, existing benchmarks suffer from three limitations: limited coverage of training stages, insufficient distributional alignment between members and non-members, and lack of rigorous filtering of non-members against the training corpus. To address these limitations, we propose OLMo-Detect, a multi-stage, confounder-controlled benchmark built upon the fully open OLMo 2 pipeline. OLMo-Detect spans pre-training, mid-training, and post-training, explicitly aligns members and non-members on three key axes, and rigorously filters non-members via infini-gram. To assess robustness to distribution shifts, we further introduce OLMo-Detect (Shifted), a variant where members are misaligned with non-members. We evaluate 15 unsupervised and 3 supervised membership inference attacks (MIAs) across the OLMo 2 family, finding that: (i) overall performance is limited: the best unsupervised and supervised MIAs both reach an AUC of only 0.68, and supervised MIAs degrade under cross-domain evaluation; (ii) MIA performance peaks at mid-training and is lower at pre-training and post-training, a pattern driven by data type rather than a stage effect: curated math data is far more detectable than other types; (iii) overall scores improve from 1B to 13B but plateau at 32B; and (iv) no unsupervised MIA is robust to distribution shifts, with AUCs shifting by up to 0.42. Finally, we find that our findings on OLMo 2 generalize to OLMo 3 and non-OLMo models.

Abstract

Membership Inference Attacks

Explore similar work

OLMo-Detect: A Multi-Stage, Confounder-Controlled Benchmark for Membership Inference on Large Language Models

Detecting Data Contamination in Large Language Models

CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models