cs.CROct 11, 2026

From Acoustic Sensing to Physical Actions: A Comprehensive Survey of Security Risks in Edge-Deployed Audio LLM-Based Agents

Authors: Binhong Wang, Haidong Kang, Yanfeng Zhang, Hong-Ning Dai, Guo Yu, Shangce Gao

Organizations: Northeastern University, China · Department of Computer Science, Hong Kong Baptist University, Hong Kong · Institute of Intelligent Manufacturing, Nanjing Tech University, Nanjing, China · Faculty of Engineering, University of Toyama, Toyama, Japan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Audio Large Language Models (Audio LLMs) are embedded in agents for smart homes, vehicles, wearables, and embodied systems, where acoustic inputs can propagate from perception into persistent state and consequential execution. This setting extends security analysis beyond model behavior to the system conditions that determine how acoustic inputs can influence agent behavior and external actions. Existing surveys examine acoustic security, LLM-based agents, and edge deployment in isolation, leaving their security interactions in deployed Audio LLM-based agents insufficiently characterized. This survey introduces an Acoustic Sensing-to-Action (S2A) system model that traces these security implications along the path from acoustic sensing to external execution, while accounting for edge-cloud runtime and lifecycle transformation. We use the model to synthesize attacks, defenses, and evaluation methods across the acoustic, agent, and edge-system literature. The synthesis shows that security gaps can emerge within components and at the transitions between stages, where protections developed for one stage do not necessarily provide end-to-end assurance in the deployed system. These findings motivate security evaluation that follows acoustic influence from sensing to action across persistent state, system boundaries, and changing deployment conditions in real-world agents.

Figures & tables

Fig. 1: Evolution of the capabilities and security of Audio and Speech LLMs toward real-time, agentic, and deployment-aware systems.
Fig. 2: Representative application settings and the shared perception-to-action workflow of edge-deployed Audio LLM-based agents. Individual systems may omit, combine, or repartition the illustrated functions.
SurveyLAPTED
Li et al. [ 10 ]–✓\checkmark–△\triangle△\triangle–
Yang et al. [ 11 ]✓\checkmark△\triangle––––
Luo et al. [ 12 ]✓\checkmark✓\checkmark–△\triangle––
Rahimov et al. [ 13 ]✓\checkmark✓\checkmark△\triangle✓\checkmark△\triangle–
Ling et al. [ 14 ]––✓\checkmark✓\checkmark△\triangle–
Lin et al. [ 16 ]––✓\checkmark△\triangle––
TABLE I: Scope comparison with representative surveys.
Application profileDominant input and contextTypical execution patternRepresentative evidenceSecurity emphasis
Conversational / InformationalHuman speech, multi-turn dialogue, paralinguistic cuesSpeech or text response with limited external actuationURO-Bench [ 21 ]Source and intended-address attribution, interactional privacy, and content safety
Personal Task / ServiceSpeech combined with personal context, memory, and task historyPersonalized response, API access, or tool invocationCarMem [ 5 ] ; Audio2Tool [ 6 ]State ownership and provenance, delegated permissions, and tool authorization
Device / Environment ControlSpeech plus device state, environmental variables, implicit intent, and temporal dependenciesDevice actuation, automation, and scheduled multi-step workflowsDESAMO [ 1 ] ; Audio2Tool [ 6 ] ; SimuHome [ 22 ] ; HomeGuardian [ 23 ]Identity and intended address, state verification, context-conditioned authorization, and physical consequences
Embodied / OperationalSpeech together with environmental or machine audio and other physical contextDiagnosis, robotic action, operational response, or physical controlRoboGuard [ 3 ] ; industrial Audio LLM [ 24 ]Sensing integrity, non-speech provenance, runtime availability, and action containment or recovery
TABLE II: Representative application profiles and scenario-specific security implications.
Fig. 4: Acoustic Sensing-to-Action (S2A) system model and security analysis areas for edge-deployed Audio LLM-based agents.
CharacteristicModel-centric settingSecurity implication
Shared acoustic exposureControlled inputPerception does not establish source provenance, intended address, or instruction authority.
Persistent agent stateTransient contextMisattributed or malicious information may persist and affect later reasoning or other interactions.
External executionResponse generationSecurity failures can extend from model behavior to unauthorized digital operations or physical effects.
Distributed edge deploymentFixed execution environmentSecurity depends on cross-component communication and on the behavior of the final deployed configuration.
TABLE III: Security-relevant characteristics of edge-deployed Audio LLM-based agents.
WorkEdge-Side RoleModeStatus
System implementations and prototypes
DESAMO [ 1 ]Audio LLM inference, event detection, and device actionOn-deviceDevice eval.
ASTA [ 25 ]ASR, lightweight LLM, adaptive routing, and action executionHybridDevice eval.
SoundHound Chat AI [ 2 ]In-vehicle LLM and on-chip voice interactionOn-deviceIndustrial demo
CoFi-Agent [ 26 ]Local Audio LLM and tools with remote reasoningHybridFramework eval.
Edge models and integration enablers
TABLE IV: Representative systems for edge-deployed Audio LLM-based agents and deployment enablers.
Attack TypeKey MechanismSecurity ImpactMain Limitation
Acoustic Sensing
Adversarial waveform manipulation [ 39 , 40 , 41 , 42 ]Perturbations optimized in waveform, frequency, token, or codec-latent spaces.Targeted behavior or jailbreaks; some variants survive lossy compression.Often model- or access-dependent; physical transfer remains inconsistent.
Covert and sensor-mediated injection [ 36 , 37 , 43 ]Commands encoded in inaudible, structural, optical, or other transduced carriers.Covert prompts or commands outside ordinary audible addressed speech.Depends on transducer and device characteristics, calibration, and delivery conditions.
Propagation-aware physical manipulation [ 38 , 44 , 45 ]Attack construction models playback, room propagation, reverberation, and recapture.Maintains malicious influence over the air and can hijack downstream behavior or actions.Sensitive to room, device, distance, and mobility conditions.
Acoustic attribute inference and privacy exposure [ 46 , 4 ]Infers identity or environmental attributes from voiceprints and non-linguistic cues.Reveals sensitive information beyond transcript content.Mostly evaluated as profiling or leakage, not propagation into later state or execution.
Provenance and Authority
TABLE V: Taxonomy and comparison of attacks on edge-deployed Audio LLM-based agents.
Fig. 5: Overview of Acoustic Sensing Attacks.
Fig. 6: Overview of Tool and Action Execution Attacks.
Defense TypeKey MechanismMain StrengthMain Limitation
Acoustic Sensing
Adversarial screening and input purification [ 114 , 115 , 116 ]Transformation-based screening or signal reconstruction before inference.Model-external protection across multiple perturbation families without retraining Audio LLMs.Mostly ASR or classification; purification may remove useful cues, and adaptive codec-robust attacks remain challenging.
Physical-channel and sensor protection [ 117 , 118 , 119 ]Uses array attenuation, time–frequency statistics, or carrier artifacts to detect injection.Blocks or detects malicious input before reasoning using physical-channel properties.Device, sensor, and attack-carrier dependent; transfer to unseen mechanisms is uncertain.
Acoustic privacy transformation [ 120 , 121 ]Anonymizes speaker identity and/or transforms sensitive linguistic content.Reduces leakage near the sensing boundary and supports streaming protection.Privacy–utility–latency trade-offs; may remove speaker or affective cues needed downstream.
Provenance and Authority
Selective hearing and intended-address detection [ 7 , 122 ]Separates intended speech from bystanders, background conversation, or non-directed speech.Reduces unintended admission in shared acoustic environments.Does not authenticate identity or establish action authority.
TABLE VI: Taxonomy and comparison of defenses for edge-deployed Audio LLM-based agents.
Fig. 7: Overview of Tool and Action Execution Defenses.
Fig. 8: Overview of Lifecycle Transformation Defenses.
BenchmarkYearReported MeasuresEvaluation Coverage
  SecAcUtilPrivActResEdge
Audio Jailbreak and Adversarial Safety
JALMBench [ 161 ]2026Attack success rate and attack efficiencyEPP––––
AJailBench [ 162 ]2026Attack success, transferability, relevance, and perceptual similarityEPE––––
Jailbreak-AudioBench [ 163 ]2025Attack success under speed, pitch, emotion, and accent variationsEPP––––
AudioJailbreak [ 164 ]2026Attack success, stealthiness, transferability, and over-the-air successEEP––PP
TABLE VII: Representative evaluation resources relevant to security of edge-deployed Audio LLM-based agents, organized by evaluation problem.
More in this topic

LLM Agent Security

Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.

See all 332 papers

Explore similar work

CardsList
May 18, 2026

A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook

The foundational capabilities established by Large Language Models (LLMs) have paved the way for Multimodal Large Language Models (MLLMs), within which Large Audio Language Models (LALMs) are essential for realizing universal auditory intelligence. Despite their remarkable performance, the escalation of LALMs' capabilities has significantly outpaced the development of systemic frameworks to ensure their trustworthiness. This survey provides a comprehensive investigation into the endogenous mechanisms of LALMs, detailing the architectural innovations and alignment algorithms that facilitate emergent reasoning. Specifically, we analyze how the transition to unified end-to-end frameworks and the integration of continuous acoustic signals inherently expand the attack surface. To rigorously evaluate the risks within these paradigms, we establish a comprehensive taxonomy of trustworthiness, categorizing critical vulnerabilities such as cross-modal jailbreaking, latent acoustic backdoors, and biometric privacy leakage. We review the state-of-the-art through six analytical pillars: hallucination, robustness, safety, privacy, fairness, and authentication. The profound imbalance between a mature offensive landscape and underdeveloped defenses further validates the critical trustworthiness gaps and multidimensional risks facing audio-centric intelligence. Finally, we propose a strategic roadmap advocating for "Defense-in-Depth" architectures, causal auditory world modeling, and intrinsic representation engineering to bridge the gap between empirical performance and intrinsically trustworthy audio intelligence. Our project has been uploaded to GitHub https://github.com/Kwwwww74/Awesome-Trustworthy-AudioLLMs.

Kaiwen Luo, Zhenhong Zhou, Leo Wang +31

Nanyang Technological University · Independent Researcher · The University of Melbourne +15

Privacy Leakage in Language ModelsAudio-Language Models
AskPDF
Jun 9, 2026

Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation

Large language model (LLM) agents are rapidly moving from conversational interfaces to software components that plan, invoke tools, maintain memory, and act on external environments. This transition changes the nature of security risk. In agentic settings, failures are no longer limited to unsafe text generation. Untrusted content may redirect control flow, misuse tool privileges, corrupt persistent state, leak sensitive information, or trigger harmful external actions. At the same time, research on LLM agent security is expanding quickly but remains fragmented across attack families, defense layers, application domains, and evaluation settings. This paper synthesizes 247 papers through a lifecycle-based, systems-oriented framework that models agent security around the interaction of information flow, delegated authority, and persistent state. We organize the literature around four questions: how LLM agent security should be modeled, which threat surfaces and attack families dominate, what defenses have been proposed and with what tradeoffs, and how security claims are evaluated. We find that prompt injection and tool-mediated control-flow hijacking still dominate the field, while persistent state corruption and multi-agent propagation are becoming central emerging concerns. We further find that current defenses provide useful building blocks but remain weakly compositional, and that existing benchmarks still underrepresent long-horizon, stateful, and deployment-sensitive risks. We argue that secure LLM agents require explicit trust boundaries, principled privilege control, provenance-aware state management, and evaluation practices aligned with realistic operational settings.

Yuchen Ling, Shengcheng Yu, Zhenyu Chen +1

State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Technical University of Munich, Heilbronn, Germany

Prompt InjectionLLM Agent SecurityAI Agent Security Benchmarks
AskPDF
Apr 10, 2026

AudioGuard: Toward Comprehensive Audio Safety Protection Across Diverse Threat Models

Audio has rapidly become a primary interface for foundation models, powering real-time voice assistants. Ensuring safety in audio systems is inherently more complex than just "unsafe text spoken aloud": real-world risks can hinge on audio-native harmful sound events, speaker attributes (e.g., child voice), impersonation/voice-cloning misuse, and voice-content compositional harms, such as child voice plus sexual content. The nature of audio makes it challenging to develop comprehensive benchmarks or guardrails against this unique risk landscape. To close this gap, we conduct large-scale red teaming on audio systems, systematically uncover vulnerabilities in audio, and develop a comprehensive, policy-grounded audio risk taxonomy and AudioSafetyBench, the first policy-based audio safety benchmark across diverse threat models. AudioSafetyBench supports diverse languages, suspicious voices (e.g., celebrity/impersonation and child voice), risky voice-content combinations, and non-speech sound events. To defend against these threats, we propose AudioGuard, a unified guardrail consisting of 1) SoundGuard for waveform-level audio-native detection and 2) ContentGuard for policy-grounded semantic protection. Extensive experiments on AudioSafetyBench and four complementary benchmarks show that AudioGuard consistently improves guardrail accuracy over strong audio-LLM-based baselines with substantially lower latency.

Mintong Kang, Chen Fang, Bo Li

University of Illinois Urbana-Champaign

LLM Safety Benchmarks
AskPDF