AutoMIA: Improved Baselines for Membership Inference Attack via Agentic Self-Exploration

Ruhao Liu; Weiqi Huang; Qi Li; Xinchao Wang

cs.CRApr 1, 2026

AutoMIA: Improved Baselines for Membership Inference Attack via Agentic Self-Exploration

Authors: Ruhao Liu, Weiqi Huang, Qi Li, Xinchao Wang

Organizations: National University of Singapore

Abstract

Membership Inference Attacks (MIAs) serve as a fundamental auditing tool for evaluating training data leakage in machine learning models. However, existing methodologies predominantly rely on static, handcrafted heuristics that lack adaptability, often leading to suboptimal performance when transferred across different large models. In this work, we propose AutoMIA, an agentic framework that reformulates membership inference as an automated process of self-exploration and strategy evolution. Given high-level scenario specifications, AutoMIA self-explores the attack space by generating executable logits-level strategies and progressively refining them through closed-loop evaluation feedback. By decoupling abstract strategy reasoning from low-level execution, our framework enables a systematic, model-agnostic traversal of the attack search space. Extensive experiments demonstrate that AutoMIA consistently matches or outperforms state-of-the-art baselines while eliminating the need for manual feature engineering.

Figures & tables

Panel of Figure 1 — Figure 1: Performance comparison between AutoMIA and baselines. Left: Comparison of the top five AutoMIA-discovered metrics and the top ten handcrafted baselines on the DALL·E dataset with LLaVA as the victim model. Middle: Comparing text-only membership inference performance across three target models (LLaVA, MiniGPT-4, and LLaMA-Adapter) under multiple dataset settings. Right: An example of an AutoMIA-generated attack strategy, showing its high-level definition alongside the corresponding executable code.

Figure 2: Overview of the AutoMIA framework. The system operates as a closed loop where the AutoMIA agent generates strategies based on historical context, the Code Execution module runs attacks against target VLMs, and the Guidance agent provides evaluation feedback to refine the Strategy Library .

Metric		LLaVA		MiniGPT-4		LLaMA Adapter
Metric		Text ${}_{\text{len}=32}$	Text ${}_{\text{len}=64}$	Text ${}_{\text{len}=32}$	Text ${}_{\text{len}=64}$	Text ${}_{\text{len}=32}$	Text ${}_{\text{len}=64}$
Perplexity		0.779	0.988	0.702	0.823	0.791	0.431
Max Prob Gap		0.462	0.545	0.637	0.418	0.583	0.616
Min- $k$ Prob	Min-0%	0.522	0.522	0.581	0.538	0.623	0.366
	Min-10%	0.461	0.883	0.585	0.668	0.658	0.375
	Min-20%	0.603	0.980	0.619	0.738	0.717	0.390

Table 1: AUC comparison of membership inference attacks under different text lengths (

L\in\{32,64\}

) on three vision–language models (LLaVA, MiniGPT-4, and LLaMAAdapter). Results are reported for representative baselines and our agent-generated strategy (Agent/Ours). We highlight the best, second-best, and third-best results in progressively lighter shades of blue, and mark the worst, second-worst, and third-worst results in progressively lighter shades of red.

Metric		DALL $\cdot$ E (LLaVA)				Fliker (LLaVA)
Metric		img	inst	desp	inst+desp	img	inst	desp	inst+desp
Perplexity		N/A	0.337	0.567	0.448	N/A	0.378	0.662	0.554
Max Prob Gap		0.529	0.578	0.598	0.603	0.579	0.603	0.645	0.646
Aug-KL		0.432	0.462	0.523	0.504	0.605	0.538	0.476	0.496
Min- $k$ Prob	Min-0%	N/A	0.481	0.556	0.481	N/A	0.358	0.647	0.358
	Min-10%	N/A	0.481	0.561	0.424	N/A	0.358	0.667	0.390

Table 2: VL-MIA AUC Comparison on DALL

\cdot

E and Fliker with LLaVA as the victim model. ‘img’ indicates the logits slice corresponding to image embedding, ‘inst’ indicates the instruction slice, ‘desp’ the generated description slice, and ‘inst+desp’ is the concatenation of the instruction slice and description slice. For the image slice, target-based MIAs are not applicable due to the absence of ground-truth token IDs, and the corresponding results are therefore reported as N/A. We highlight the best, second-best, and third-best results in progressively lighter shades of blue, and mark the worst, second-worst, and third-worst results in progressively lighter shades of red.

Figure 3: Ablation on Agent Backbone. Performance comparison of AutoMIA driven by different VLM backbones (Gemini 3 Flash, Grok 4.1 Fast, Qwen3-Max, and DeepSeek-V3.2-Reasoner) on LLaMA-Adapter.

Figure 4: Token Consumption Figure: Input vs Output for Different VLM Models. Total tokens per round are indicated for each model. Red represents the output tokens, and blue represents the input tokens.

Generated Strategy	Validation Set (50%)			Hold-out Set (50%)
Generated Strategy	AUC	Acc	TPR@5%	AUC	Acc	TPR@5%
True-Token Probability Momentum	0.784	0.723	0.152	0.741	0.699	0.096
True-Token Probability Consistency	0.784	0.724	0.164	0.738	0.706	0.104
Probability Curvature Sign Consistency	0.792	0.751	0.133	0.735	0.694	0.082
True-Token Relative-Confidence Momentum	0.792	0.733	0.182	0.735	0.703	0.170
True-Token Neighborhood Cohesion	0.773	0.727	0.176	0.663	0.628	0.096

Table 3: Generalizability of top AutoMIA strategies under a 50% validation / 50% hold-out test split.

Method	AUC	Acc	TPR@5%
Best AutoMIA	0.723	0.688	0.240
Top-5 AutoMIA	0.716	0.678	0.207
Max_100% renyi_05	0.716	0.674	0.216
Max_100% renyi_1	0.676	0.648	0.138
Max_100% renyi_2	0.642	0.618	0.130
Max_100% renyi_inf	0.633	0.612	0.138

Table 4: Performance comparison on the OLMo near-IID evaluation setting. We report the best AutoMIA strategy and the average performance of the top-5 AutoMIA strategies, together with representative baseline methods.

Text Length	Method	AUC	Acc	TPR@5%
32	w/o Guidance	0.709	0.660	0.147
32	AutoMIA	0.828	0.782	0.177
64	w/o Guidance	0.654	0.623	0.073
64	AutoMIA	0.787	0.722	0.143

Table 5: Ablation study on the effect of the guidance agent in AutoMIA under different text lengths.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Metric		LLaVA		MiniGPT-4		LLaMAAdapter
Metric		Text ${}_{\text{len}=32}$	Text ${}_{\text{len}=64}$	Text ${}_{\text{len}=32}$	Text ${}_{\text{len}=64}$	Text ${}_{\text{len}=32}$	Text ${}_{\text{len}=64}$
Perplexity		0.717	0.943	0.670	0.758	0.727	0.512
Max Prob Gap		0.513	0.555	0.627	0.512	0.588	0.600
Min- $k$ Prob	Min-0%	0.522	0.522	0.572	0.540	0.613	0.502
	Min-10%	0.507	0.808	0.575	0.642	0.627	0.502
	Min-20%	0.580	0.928	0.598	0.677	0.672	0.503

Appendix

Table 6: Accuracy comparison of membership inference attacks under different text lengths (

L\in\{32,64\}

) on three vision–language models (LLaVA, MiniGPT-4, and LLaMAAdapter). We highlight the best, second-best, and third-best results in progressively lighter shades of blue, and mark the worst, second-worst, and third-worst results in progressively lighter shades of red.

Metric		LLaVA		MiniGPT-4		LLaMAAdapter
Metric		Text ${}_{\text{len}=32}$	Text ${}_{\text{len}=64}$	Text ${}_{\text{len}=32}$	Text ${}_{\text{len}=64}$	Text ${}_{\text{len}=32}$	Text ${}_{\text{len}=64}$
Perplexity		0.253	0.913	0.193	0.317	0.303	0.007
Max Prob Gap		0.053	0.067	0.127	0.013	0.100	0.083
Min- $k$ Prob	Min-0%	0.000	0.000	0.107	0.070	0.070	0.013
	Min-10%	0.007	0.467	0.110	0.167	0.147	0.010
	Min-20%	0.110	0.890	0.117	0.227	0.200	0.007

Appendix

Table 7: TPR@5%FPR comparison of membership inference attacks under different text lengths (

L\in\{32,64\}

) on three vision–language models. We highlight the best, second-best, and third-best results in progressively lighter shades of blue, and mark the worst, second-worst, and third-worst results in progressively lighter shades of red.

Metric		LLaVA				MiniGPT-4				LLaMA Adapter
Metric		img	inst	desp	inst+desp	img	inst	desp	inst+desp	inst	desp	inst+desp
Perplexity		0.637	0.502	0.623	0.548	0.545	0.503	0.500	0.500	0.500	0.590	0.502
Max Prob Gap		0.575	0.582	0.620	0.623	0.533	0.571	0.505	0.510	0.513	0.622	0.607
Aug-KL		0.610	0.562	0.512	0.525	0.505	0.500	0.502	0.502	0.515	0.513	0.518
Min- $k$ Prob	Min-0%	0.573	0.502	0.615	0.502	0.550	0.507	0.502	0.507	0.502	0.530	0.502
	Min-10%	0.580	0.502	0.648	0.503	0.553	0.507	0.502	0.500	0.500	0.525	0.500

Appendix

Table 8: VL-MIA Accuracy comparison on Flickr with LLaVA, MiniGPT-4, and LLaMA Adapter. ‘img’ indicates the logits slice corresponding to image embedding, ‘inst’ indicates the instruction slice, ‘desp’ the generated description slice, and ‘inst+desp’ is the concatenation of the instruction slice and description slice. We highlight the best, second-best, and third-best results in progressively lighter shades of blue, and mark the worst, second-worst, and third-worst results in progressively lighter shades of red.

Metric		LLaVA				MiniGPT-4				LLaMA Adapter
Metric		img	inst	desp	inst+desp	img	inst	desp	inst+desp	inst	desp	inst+desp
Perplexity		0.070	0.003	0.130	0.083	0.020	0.010	0.024	0.013	0.003	0.097	0.010
Max Prob Gap		0.057	0.077	0.160	0.160	0.030	0.050	0.027	0.023	0.060	0.230	0.183
Aug-KL		0.040	0.057	0.057	0.067	0.033	0.027	0.017	0.010	0.043	0.067	0.063
Min- $k$ Prob	Min-0%	0.097	0.023	0.083	0.023	0.040	0.054	0.027	0.053	0.030	0.067	0.030
	Min-10%	0.113	0.023	0.083	0.013	0.027	0.054	0.020	0.020	0.010	0.060	0.007

Appendix

Table 9: VL-MIA TPR@5%FPR comparison on Flickr with LLaVA, MiniGPT-4, and LLaMA Adapter. The column notations (‘img’, ‘inst’, ‘desp’, ‘inst+desp’) follow the same definitions as in Table 8 . We highlight the best, second-best, and third-best results in progressively lighter shades of blue, and mark the worst, second-worst, and third-worst results in progressively lighter shades of red.

Metric		LLaVA				MiniGPT-4				LLaMA Adapter
Metric		img	inst	desp	inst+desp	img	inst	desp	inst+desp	inst	desp	inst+desp
Perplexity		0.549	0.505	0.569	0.507	0.566	0.568	0.564	0.568	0.511	0.508	0.536
Max Prob Gap		0.537	0.571	0.591	0.593	0.541	0.515	0.568	0.563	0.534	0.529	0.534
Aug-KL		0.500	0.510	0.529	0.522	0.549	0.568	0.541	0.557	0.573	0.556	0.578
Min- $k$ Prob	Min-0%	0.613	0.520	0.557	0.520	0.546	0.541	0.542	0.541	0.555	0.505	0.556
	Min-10%	0.659	0.520	0.561	0.510	0.544	0.541	0.551	0.544	0.530	0.505	0.513

Appendix

Table 10: VL-MIA Accuracy comparison on DALL·E with LLaVA, MiniGPT-4, and LLaMA Adapter. ‘img’ indicates the logits slice corresponding to image embedding, ‘inst’ indicates the instruction slice, ‘desp’ the generated description slice, and ‘inst+desp’ is the concatenation of the instruction slice and description slice. We highlight the best, second-best, and third-best results in progressively lighter shades of blue, and mark the worst, second-worst, and third-worst results in progressively lighter shades of red.

Metric		LLaVA				MiniGPT-4				LLaMA Adapter
Metric		img	inst	desp	inst+desp	img	inst	desp	inst+desp	inst	desp	inst+desp
Perplexity		0.020	0.027	0.078	0.054	0.128	0.044	0.057	0.051	0.051	0.051	0.044
Max Prob Gap		0.037	0.085	0.081	0.061	0.088	0.027	0.115	0.108	0.108	0.034	0.041
Aug-KL		0.027	0.054	0.085	0.081	0.047	0.105	0.041	0.081	0.108	0.098	0.098
Min- $k$ Prob	Min-0%	0.132	0.081	0.047	0.081	0.047	0.051	0.078	0.051	0.058	0.047	0.054
	Min-10%	0.135	0.081	0.064	0.041	0.088	0.051	0.057	0.051	0.054	0.047	0.044

Appendix

Table 11: VL-MIA TPR@5%FPR comparison on DALL·E with LLaVA, MiniGPT-4, and LLaMA Adapter. The column notations (‘img’, ‘inst’, ‘desp’, ‘inst+desp’) follow the same definitions as in Table 10 . We highlight the best, second-best, and third-best results in progressively lighter shades of blue, and mark the worst, second-worst, and third-worst results in progressively lighter shades of red.

Figure 7: Validation of representative AutoMIA-discovered metrics under a controlled synthetic memorization simulation. The results show that metrics such as avg_true_max_log_gap produce clear separation between simulated member and non-member distributions, supporting the claim that the discovered formulas capture meaningful memorization-related structure rather than incidental correlations.

Membership Inference Attacks

Membership inference attacks test whether records or entities were used to train a machine-learning model, revealing potential privacy leakage.

See all 116 papers

Automated Membership Inference Attacks (AutoMIA): Discovering MIA Signal Computations using LLM Agents
Mar 19, 2026Toan Tran, Olivera Kotevska, Li XiongMembership Inference Attacks
Emory University · Oak Ridge National Laboratory
ReproMIA: A Comprehensive Analysis of Model Reprogramming for Proactive Membership Inference Attacks
Mar 30, 2026Chihan Huang, Huaijin Wang, Shuai WangMembership Inference Attacks Privacy Leakage in Language Models
HKUST, Hong Kong, China
A Full-Pipeline Framework for Evaluating Membership Inference Attacks in Machine Learning
May 28, 2026Ding Chen, Xinwen Cheng, Xuyang Zhong +3Privacy Auditing Membership Inference Attacks
City University of Hong Kong · Shanghai Jiao Tong University · denotes the correspondence author.

Mar 19, 2026J/K move · Enter open · S save

Automated Membership Inference Attacks (AutoMIA): Discovering MIA Signal Computations using LLM Agents

Toan Tran, Olivera Kotevska, Li Xiong

Emory University · Oak Ridge National Laboratory

Membership Inference Attacks

Open paper AskPDF arXiv

Membership inference attacks (MIAs), which enable adversaries to determine whether specific data points were part of a model's training dataset, have emerged as an important framework to understand, assess, and quantify the potential information leakage associated with machine learning systems. Designing effective MIAs is a challenging task that usually requires extensive manual exploration of model behaviors to identify potential vulnerabilities. In this paper, we introduce AutoMIA -- a novel framework that leverages large language model (LLM) agents to automate the design and implementation of new MIA signal computations. By utilizing LLM agents, we can systematically explore a vast space of potential attack strategies, enabling the discovery of novel strategies. Our experiments demonstrate AutoMIA can successfully discover new MIAs that are specifically tailored to user-configured target model and dataset, resulting in improvements of up to 0.18 in absolute AUC over existing MIAs. This work provides the first demonstration that LLM agents can serve as an effective and scalable paradigm for designing and implementing MIAs with SOTA performance, opening up new avenues for future exploration.

Abstract

Membership Inference Attacks

Explore similar work

Automated Membership Inference Attacks (AutoMIA): Discovering MIA Signal Computations using LLM Agents

ReproMIA: A Comprehensive Analysis of Model Reprogramming for Proactive Membership Inference Attacks

A Full-Pipeline Framework for Evaluating Membership Inference Attacks in Machine Learning