cs.CRMar 12, 2026

Cascade: Composing Software-Hardware Attack Gadgets for Adversarial Threat Amplification in Compound AI Systems

Authors: Sarbartha Banerjee, Prateek Sahu, Anjo Vahldiek-Oberwagner, Jose Sanchez Vicarte, Mohit Tiwari

Organizations: The University of Texas at Austin · Georgia Tech · Intel Labs · Microsoft · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Rapid progress in generative AI has given rise to Compound AI systems - pipelines comprised of multiple large language models (LLM), software tools and database systems. Compound AI systems are constructed on a layered traditional software stack running on a distributed hardware infrastructure. Many of the diverse software components are vulnerable to traditional security flaws documented in the Common Vulnerabilities and Exposures (CVE) database, while the underlying distributed hardware infrastructure remains exposed to timing attacks, bit-flip faults, and power-based side channels. Today, research targets LLM-specific risks like model extraction, training data leakage, and unsafe generation -- overlooking the impact of traditional system vulnerabilities. This work investigates how traditional software and hardware vulnerabilities can complement LLM-specific algorithmic attacks to compromise the integrity of a compound AI pipeline. We demonstrate two novel attacks that combine system-level vulnerabilities with algorithmic weaknesses: (1) Exploiting a software code injection flaw along with a guardrail Rowhammer attack to inject an unaltered jailbreak prompt into an LLM, resulting in an AI safety violation, and (2) Manipulating a knowledge database to redirect an LLM agent to transmit sensitive user data to a malicious application, thus breaching confidentiality. These attacks highlight the need to address traditional vulnerabilities; we systematize the attack primitives and analyze their composition by grouping vulnerabilities by their objective and mapping them to distinct stages of an attack lifecycle. This approach enables a rigorous red-teaming exercise and lays the groundwork for future defense strategies.

Figures & tables

Fig. 1 : The building blocks of a Compound AI pipeline with cross-stack attack gadgets comprising of adversarial attacks, software vulnerabilities and hardware side-channels.
Fig. 2 : The building blocks of a Compound AI pipeline with cross-stack attack gadgets comprising of adversarial attacks, software vulnerabilities and hardware side-channels.
Fig. 3 : The building blocks of a Compound AI pipeline with cross-stack attack gadgets comprising of adversarial attacks, software vulnerabilities and hardware side-channels.
Fig. 4 : Cascade framework: Given an attacker’s objective, capability, and query, the framework uses LLM-based reasoning to retrieve candidate gadgets, evaluate them against the target AI pipeline, and iteratively refine attack chains – instantiating open-source testbeds when needed (e.g., for GCG jailbreaks) – until success or timeout.
Fig. 5 : Efficacy of guardrail and language model against harmful prompts. Guardrails are trained to filter unsafe queries but do not modify the query themselves. Guardrails perform better for certain categories with an overall efficiency of being able to block 63% queries that generative models fail to stop.
Fig. 6 : A multi-stage attack on AI pipeline components leveraging system gadgets to bypass safety mechanisms and effectively utilize a jailbreak prompt on generator model.
AttackBitflipGuardrailJailbreakASR
VariantProbabilityEvasion  
Type 197.3% (Phoenix)82%82%0.654
99.3%(HW Trojan)0.667
Type 297.3% (Phoenix)72%82%0.574
99.3%(HW Trojan)0.586
Type 397.3% (Phoenix)94%82%0.75
TABLE I: Attack success of various techniques used to exploit Guardrails and Generative LLM models. Type 3 gadgets show high reliability (94%) in evading guardrail defenses. Targeted token or attention mask bitflips (Type 1 & 2) show lower reliability in prompts which have multiple unsafe tokens or have contextual malicious intent. We use typical bitflip probabilities reported in prior works: Phoenix [ 50 ] and Rowhammer Trojan [ 51 ] to calculate probabilistic success rates across the gadgets.
Figure 8
More in this topic

AI Agent Security

Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.

See all 508 papers

Explore similar work

CardsList
Oct 2, 2026

Beware EviLLM: Enabling Vulnerability Injection via Large Language Models

Advances in large language models (LLMs) have enabled AI-driven code generation from natural language specifications, introducing new attack surfaces for injecting vulnerabilities into software. Prior work has studied this problem only in benign settings where vulnerabilities are introduced inadvertently, or under unconventional threat models where the LLM itself is malicious (backdooring) or the user is the attacker (jailbreaking). In this paper, we study a more realistic threat model: a third-party adversary, with capabilities comparable to existing cybercriminals, compromises the AI code generation pipeline to deliberately introduce vulnerabilities. We call this the EviLLM attack. We have implemented two instances of EviLLM, each of which only requires the underlying LLM to be accessed through a compromised account or browser, and can inject vulnerabilities from 13 CWE classes. As we show in our feasibility study, both attack vectors are already used to implement many existing cyberattacks. Our user study shows that 7 out of 8 and 10 out of 13 participants did not notice the vulnerabilities injected by the two instances of EviLLM, and 13 out of 21 participants "rarely" or "never" considered the risk of an attack like EviLLM.

Zeezoo Ryu, Simon Chung, Muhammad Faraz Karim +4

Georgia Institute of Technology · Snowflake

LLM Security
AskPDF
May 10, 2026

Restricting the Model, Missing the System: Measurement and Accountability in Offensive AI Governance

We show that the instruments used to measure AI offensive capability fail in two ways: (i) they overstate harm, and (ii) they credit the model with capability that belongs to the surrounding system. We argue that restricting access to a model is therefore necessary but not sufficient and that policy and procurement also need system-level, harm-grounded capability assessment. In June 2026, two frontier models were suspended under US export controls, reportedly prompted by a jailbreak that asked a model to read a codebase and fix its flaws. This finding measured an elicitation system of model, prompt, and task. We support our argument with a study of an open-source framework in which lightweight large language model (LLM) agents coordinate through shared memory and evolutionary optimization, providing two pieces of evidence. First, jailbreak metrics overstate harm: over 225 swarm-generated attacks per target, LLM-as-judge scoring rated Claude Sonnet 4 compromised in 40% of attacks, yet manual verification found actionable harmful content in none, against a 45.8% Effective Harm Rate for GPT-4o. Second, scaffolded evaluations misattribute capability: on a planted-vulnerability target, a full pipeline built around a 1.2B-parameter model recovers 9 of 9 weaknesses, while the same model without the hand-crafted components recovers 0 of 9 by crash verification and 2 of 9 by cited source line. Offensive capability is a property of model, scaffold, and evaluation protocol together. The duty to assess it should lie with whoever controls the system, shapes its behaviour, and can foresee what it will do. In our setting, that is the party who builds the harness around the model, a role that current regulation does not clearly cover.

Michael Alexander Riegler, Finn Schwall, Annika Willoch Olstad +4

AI Safety Department SimulaMet and OsloMet Oslo, Norway · Department of Computer Science NTNU Trondheim, Norway

AI Agent SecurityAI Safety EvaluationLLM Jailbreak Attacks
AskPDF
May 26, 2026

Lessons from Penetration Tests on Large-Scale Agent Systems

As AI systems gain increasing autonomy and execution capability, the number of discovered security vulnerabilities continues to rise. However, many of these vulnerabilities are not fundamentally novel, but instead reflect recurring classes of weaknesses long observed in prior computing systems. Execution-capable AI agents are effectively unbounded, self-modifying programs that interact extensively with multiple layers of the computing stack. This broad interaction surface imposes a significant security burden on developers, who must reason about and secure complex cross-layer behaviors. Prior research has primarily focused on vulnerabilities in open-source agents and agent frameworks. In contrast, it remains unclear whether proprietary agent systems -- developed under stricter coding standards and formal review processes -- exhibit similar security weaknesses. In this paper, we present findings from two penetration tests conducted in 2025 against proprietary agent products and evaluate whether the security posture of AI agents has improved since these assessments.

Kevin Eykholt, Dhilung Kirat, Xiaokui Shu +3

IBM Research

Software SecurityAI Agent Security
AskPDF