From Acoustic Sensing to Physical Actions: A Comprehensive Survey of Security Risks in Edge-Deployed Audio LLM-Based Agents

Binhong Wang; Haidong Kang; Yanfeng Zhang; Hong-Ning Dai; Guo Yu; Shangce Gao

cs.CROct 11, 2026

From Acoustic Sensing to Physical Actions: A Comprehensive Survey of Security Risks in Edge-Deployed Audio LLM-Based Agents

Authors: Binhong Wang, Haidong Kang, Yanfeng Zhang, Hong-Ning Dai, Guo Yu, Shangce Gao

Organizations: Northeastern University, China · Department of Computer Science, Hong Kong Baptist University, Hong Kong · Institute of Intelligent Manufacturing, Nanjing Tech University, Nanjing, China · Faculty of Engineering, University of Toyama, Toyama, Japan

Abstract

Audio Large Language Models (Audio LLMs) are embedded in agents for smart homes, vehicles, wearables, and embodied systems, where acoustic inputs can propagate from perception into persistent state and consequential execution. This setting extends security analysis beyond model behavior to the system conditions that determine how acoustic inputs can influence agent behavior and external actions. Existing surveys examine acoustic security, LLM-based agents, and edge deployment in isolation, leaving their security interactions in deployed Audio LLM-based agents insufficiently characterized. This survey introduces an Acoustic Sensing-to-Action (S2A) system model that traces these security implications along the path from acoustic sensing to external execution, while accounting for edge-cloud runtime and lifecycle transformation. We use the model to synthesize attacks, defenses, and evaluation methods across the acoustic, agent, and edge-system literature. The synthesis shows that security gaps can emerge within components and at the transitions between stages, where protections developed for one stage do not necessarily provide end-to-end assurance in the deployed system. These findings motivate security evaluation that follows acoustic influence from sensing to action across persistent state, system boundaries, and changing deployment conditions in real-world agents.

Figures & tables

Fig. 1: Evolution of the capabilities and security of Audio and Speech LLMs toward real-time, agentic, and deployment-aware systems.

Fig. 2: Representative application settings and the shared perception-to-action workflow of edge-deployed Audio LLM-based agents. Individual systems may omit, combine, or repartition the illustrated functions.

Survey	L	A	P	T	E	D
Li et al. [ 10 ]	–	$\checkmark$	–	$\triangle$	$\triangle$	–
Yang et al. [ 11 ]	$\checkmark$	$\triangle$	–	–	–	–
Luo et al. [ 12 ]	$\checkmark$	$\checkmark$	–	$\triangle$	–	–
Rahimov et al. [ 13 ]	$\checkmark$	$\checkmark$	$\triangle$	$\checkmark$	$\triangle$	–
Ling et al. [ 14 ]	–	–	$\checkmark$	$\checkmark$	$\triangle$	–
Lin et al. [ 16 ]	–	–	$\checkmark$	$\triangle$	–	–

TABLE I: Scope comparison with representative surveys.

Application profile	Dominant input and context	Typical execution pattern	Representative evidence	Security emphasis
Conversational / Informational	Human speech, multi-turn dialogue, paralinguistic cues	Speech or text response with limited external actuation	URO-Bench [ 21 ]	Source and intended-address attribution, interactional privacy, and content safety
Personal Task / Service	Speech combined with personal context, memory, and task history	Personalized response, API access, or tool invocation	CarMem [ 5 ] ; Audio2Tool [ 6 ]	State ownership and provenance, delegated permissions, and tool authorization
Device / Environment Control	Speech plus device state, environmental variables, implicit intent, and temporal dependencies	Device actuation, automation, and scheduled multi-step workflows	DESAMO [ 1 ] ; Audio2Tool [ 6 ] ; SimuHome [ 22 ] ; HomeGuardian [ 23 ]	Identity and intended address, state verification, context-conditioned authorization, and physical consequences
Embodied / Operational	Speech together with environmental or machine audio and other physical context	Diagnosis, robotic action, operational response, or physical control	RoboGuard [ 3 ] ; industrial Audio LLM [ 24 ]	Sensing integrity, non-speech provenance, runtime availability, and action containment or recovery

TABLE II: Representative application profiles and scenario-specific security implications.

Fig. 4: Acoustic Sensing-to-Action (S2A) system model and security analysis areas for edge-deployed Audio LLM-based agents.

Characteristic	Model-centric setting	Security implication
Shared acoustic exposure	Controlled input	Perception does not establish source provenance, intended address, or instruction authority.
Persistent agent state	Transient context	Misattributed or malicious information may persist and affect later reasoning or other interactions.
External execution	Response generation	Security failures can extend from model behavior to unauthorized digital operations or physical effects.
Distributed edge deployment	Fixed execution environment	Security depends on cross-component communication and on the behavior of the final deployed configuration.

TABLE III: Security-relevant characteristics of edge-deployed Audio LLM-based agents.

Work	Edge-Side Role	Mode	Status
System implementations and prototypes
DESAMO [ 1 ]	Audio LLM inference, event detection, and device action	On-device	Device eval.
ASTA [ 25 ]	ASR, lightweight LLM, adaptive routing, and action execution	Hybrid	Device eval.
SoundHound Chat AI [ 2 ]	In-vehicle LLM and on-chip voice interaction	On-device	Industrial demo
CoFi-Agent [ 26 ]	Local Audio LLM and tools with remote reasoning	Hybrid	Framework eval.
Edge models and integration enablers

TABLE IV: Representative systems for edge-deployed Audio LLM-based agents and deployment enablers.

Attack Type	Key Mechanism	Security Impact	Main Limitation
Acoustic Sensing
Adversarial waveform manipulation [ 39 , 40 , 41 , 42 ]	Perturbations optimized in waveform, frequency, token, or codec-latent spaces.	Targeted behavior or jailbreaks; some variants survive lossy compression.	Often model- or access-dependent; physical transfer remains inconsistent.
Covert and sensor-mediated injection [ 36 , 37 , 43 ]	Commands encoded in inaudible, structural, optical, or other transduced carriers.	Covert prompts or commands outside ordinary audible addressed speech.	Depends on transducer and device characteristics, calibration, and delivery conditions.
Propagation-aware physical manipulation [ 38 , 44 , 45 ]	Attack construction models playback, room propagation, reverberation, and recapture.	Maintains malicious influence over the air and can hijack downstream behavior or actions.	Sensitive to room, device, distance, and mobility conditions.
Acoustic attribute inference and privacy exposure [ 46 , 4 ]	Infers identity or environmental attributes from voiceprints and non-linguistic cues.	Reveals sensitive information beyond transcript content.	Mostly evaluated as profiling or leakage, not propagation into later state or execution.
Provenance and Authority

TABLE V: Taxonomy and comparison of attacks on edge-deployed Audio LLM-based agents.

Fig. 5: Overview of Acoustic Sensing Attacks.

Fig. 6: Overview of Tool and Action Execution Attacks.

Defense Type	Key Mechanism	Main Strength	Main Limitation
Acoustic Sensing
Adversarial screening and input purification [ 114 , 115 , 116 ]	Transformation-based screening or signal reconstruction before inference.	Model-external protection across multiple perturbation families without retraining Audio LLMs.	Mostly ASR or classification; purification may remove useful cues, and adaptive codec-robust attacks remain challenging.
Physical-channel and sensor protection [ 117 , 118 , 119 ]	Uses array attenuation, time–frequency statistics, or carrier artifacts to detect injection.	Blocks or detects malicious input before reasoning using physical-channel properties.	Device, sensor, and attack-carrier dependent; transfer to unseen mechanisms is uncertain.
Acoustic privacy transformation [ 120 , 121 ]	Anonymizes speaker identity and/or transforms sensitive linguistic content.	Reduces leakage near the sensing boundary and supports streaming protection.	Privacy–utility–latency trade-offs; may remove speaker or affective cues needed downstream.
Provenance and Authority
Selective hearing and intended-address detection [ 7 , 122 ]	Separates intended speech from bystanders, background conversation, or non-directed speech.	Reduces unintended admission in shared acoustic environments.	Does not authenticate identity or establish action authority.

TABLE VI: Taxonomy and comparison of defenses for edge-deployed Audio LLM-based agents.

Fig. 7: Overview of Tool and Action Execution Defenses.

Fig. 8: Overview of Lifecycle Transformation Defenses.

Benchmark	Year	Reported Measures	Evaluation Coverage
	Year		Sec	Ac	Util	Priv	Act	Res	Edge
Audio Jailbreak and Adversarial Safety
JALMBench [ 161 ]	2026	Attack success rate and attack efficiency	E	P	P	–	–	–	–
AJailBench [ 162 ]	2026	Attack success, transferability, relevance, and perceptual similarity	E	P	E	–	–	–	–
Jailbreak-AudioBench [ 163 ]	2025	Attack success under speed, pitch, emotion, and accent variations	E	P	P	–	–	–	–
AudioJailbreak [ 164 ]	2026	Attack success, stealthiness, transferability, and over-the-air success	E	E	P	–	–	P	P

TABLE VII: Representative evaluation resources relevant to security of edge-deployed Audio LLM-based agents, organized by evaluation problem.

LLM Agent Security

Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.

See all 332 papers

A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook
May 18, 2026Kaiwen Luo, Zhenhong Zhou, Leo Wang +31Privacy Leakage in Language Models Audio-Language Models
Nanyang Technological University · Independent Researcher · The University of Melbourne +15
Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
Jun 9, 2026Yuchen Ling, Shengcheng Yu, Zhenyu Chen +1Prompt Injection LLM Agent Security
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Technical University of Munich, Heilbronn, Germany
AudioGuard: Toward Comprehensive Audio Safety Protection Across Diverse Threat Models
Apr 10, 2026Mintong Kang, Chen Fang, Bo LiLLM Safety Benchmarks
University of Illinois Urbana-Champaign

May 18, 2026J/K move · Enter open · S save

A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook

Kaiwen Luo, Zhenhong Zhou, Leo Wang, Liang Lin+30

Nanyang Technological University · Independent Researcher · The University of Melbourne+15

Privacy Leakage in Language Models Audio-Language Models

Open paper AskPDF arXiv

The foundational capabilities established by Large Language Models (LLMs) have paved the way for Multimodal Large Language Models (MLLMs), within which Large Audio Language Models (LALMs) are essential for realizing universal auditory intelligence. Despite their remarkable performance, the escalation of LALMs' capabilities has significantly outpaced the development of systemic frameworks to ensure their trustworthiness. This survey provides a comprehensive investigation into the endogenous mechanisms of LALMs, detailing the architectural innovations and alignment algorithms that facilitate emergent reasoning. Specifically, we analyze how the transition to unified end-to-end frameworks and the integration of continuous acoustic signals inherently expand the attack surface. To rigorously evaluate the risks within these paradigms, we establish a comprehensive taxonomy of trustworthiness, categorizing critical vulnerabilities such as cross-modal jailbreaking, latent acoustic backdoors, and biometric privacy leakage. We review the state-of-the-art through six analytical pillars: hallucination, robustness, safety, privacy, fairness, and authentication. The profound imbalance between a mature offensive landscape and underdeveloped defenses further validates the critical trustworthiness gaps and multidimensional risks facing audio-centric intelligence. Finally, we propose a strategic roadmap advocating for "Defense-in-Depth" architectures, causal auditory world modeling, and intrinsic representation engineering to bridge the gap between empirical performance and intrinsically trustworthy audio intelligence. Our project has been uploaded to GitHub https://github.com/Kwwwww74/Awesome-Trustworthy-AudioLLMs.

Abstract

LLM Agent Security

Explore similar work

A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook

Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation

AudioGuard: Toward Comprehensive Audio Safety Protection Across Diverse Threat Models