Evaluating and Improving the Robustness of Large Language Models to Input Sequence Variations

Narek Maloyan

cs.CROct 1, 2026

Evaluating and Improving the Robustness of Large Language Models to Input Sequence Variations

Authors: Narek Maloyan

Abstract

Large language models (LLMs) in production systems face prompt injections, trojans (backdoors), and manipulation of automatic quality metrics. This thesis develops models, methods, and algorithms for evaluating and improving LLM robustness to adversarial input sequence variations. We propose R_stab(f), a generative robustness metric based on the Jensen-Shannon divergence between per-step output distributions under small input perturbations. For localized attacks we prove V(h) <= 1 - R_class(h), where R_class(h) is the probability that a decision operator h keeps its decision under small perturbations. For non-localized attacks we propose a calibrated empirical model. For LLM-as-a-Judge systems we develop ASA, an adaptive evolutionary black-box attack that reaches an attack success rate (ASR) of up to 73.8%, with transfer between open models up to 62.6%. On Trojan Detection Challenge 2023 data (Pythia-1.4B), surrogate triggers reach REASR ~0.99 while recall of the true triggers is ~0.17 against a baseline of ~0.14. On SaTML CTF 2024 we systematize four classes of bypasses of multi-layer defenses, which reduce the ASR from 90% to 15-25%. Committees of 5-7 heterogeneous models reduce the ASR for Gemma-3-4B by 47-55 percentage points, to 19.3% with 7 models. For agentic systems based on the Model Context Protocol (MCP), we propose AttestMCP, which attests tool calls with HMAC-protected packets at under 0.1 ms per call, and the Commit Boundary isolation pattern. On the MCPBench benchmark of 847 scenarios they reduce the average ASR from 53.7% to 12.4%. The methods are implemented in the JudgeGuard and TrojanArmor software suites and the MCPSec module.

Figures & tables

Panel of Figure 1 — Figure 1.1 : Illustration of $\epsilon$ -perturbations: the point $x^{\prime}$ lies in $\mathcal{B}_{\epsilon}(x)$ , and the point $y$ lies outside it.

Metric	Description	Application
Hamming ( $d_{H}$ )	Number of substituted tokens (for strings of equal length)	$\epsilon$ : 1–5; token substitution attacks
Levenshtein ( $d_{L}$ )	Min. number of operations (insertion, deletion, substitution)	$\epsilon$ : 1–10; typos, short insertions
Semantic ( $d_{S}$ )	Cosine distance between embeddings	$\epsilon$ : 0.05–0.2; paraphrasing

Table 1.1 : Summary of distance metrics in the token space

ID	Threat (OWASP)	Description	Threats from Sec. 1.2.2
LLM01	Prompt Injection	Injection of malicious instructions that override the behavior of the model.	1. Prompt injection, 14. Jailbreaking
LLM02	Insecure Output Handling	Insufficient validation and sanitization of LLM outputs before use in other systems.	9. Insecure output handling
LLM03	Training Data Poisoning	Manipulation of the training data to compromise the model.	2. Data poisoning, 3. Backdoor attacks
LLM04	Model Denial of Service (Model DoS)	Attacks that lead to resource exhaustion and model unavailability.	7. Denial of service (DoS)
LLM05	Supply Chain Vulnerabilities	Use of vulnerable components: models, dependencies, data.	13. Supply chain vulnerabilities
LLM06	Sensitive Information Disclosure	Unintended disclosure of confidential data by the model.	11. Sensitive information disclosure, 4. Model inversion, 5. Membership inference, 10. Data extraction

Table 1.2: Threat classification according to OWASP Top 10 for LLM (2023) and its relation to the detailed threat list

Threat	NIST control families	Example controls
1. Prompt injection	SI, AC	SI-10, AC-6, AC-3
2. Data poisoning	SI, SR	SI-7, SR-3, SR-6
3. Backdoor attacks	SI, SR	SI-7, SR-4, SR-5
4. Model inversion	AC, PT	AC-3, AC-4, PT-3, PT-4
5. Membership inference	PT, SI, AU	PT-3, SI-4, AU-6
6. Adversarial examples	SI, AT	SI-4, SI-10, AT-2

Table 1.3: Mapping of LLM threats to NIST SP 800-53 Rev. 5 control families

Panel of Figure 2 — Figure 2.1 : The query processing chain from the user to the model output

Metric	Invar.	Sens.	Comput.
$R_{stab}(f)$	High	High	Med./High
$V(f)$	High	High	High
BLEU/ROUGE-like metrics	Medium	Low	High
Perplexity-based score	Low	Medium	High

Table 2.1: Qualitative comparison of metric properties.

Panel of Figure 3 — Figure 3.1 : Flow of a request through the layers of a multi-layer defense.

Model	BI	CWB	CM	ASA
Gemma-3-27B-Instruct	57.5 $\pm$ 6.9	48.7 $\pm$ 6.9	67.7 $\pm$ 6.5	72.3 $\pm$ 6.2
Gemma-3-4B-Instruct	66.7 $\pm$ 6.5	55.5 $\pm$ 6.9	67.4 $\pm$ 6.5	73.8 $\pm$ 6.1
Llama-3.2-3B-Instruct	60.1 $\pm$ 6.8	47.5 $\pm$ 6.9	47.8 $\pm$ 6.9	58.2 $\pm$ 6.8
GPT-4	32.4 $\pm$ 6.5	28.6 $\pm$ 6.3	41.2 $\pm$ 6.8	45.7 $\pm$ 6.9
Claude-3-Opus	29.8 $\pm$ 6.3	25.3 $\pm$ 6.0	38.5 $\pm$ 6.7	42.9 $\pm$ 6.9

Table 4.1 : Attack success rate (ASR, %) on various LLM-as-a-Judge models (aggregated over 4 tasks,

n=200

per model). CI denotes the 95% confidence interval obtained by bootstrap with 1000 iterations.

Figure 4.1 : Heat map of the attack success rate (ASR) on LLM-as-a-Judge models. The horizontal dashed line separates the open models (top) from the proprietary models (bottom). Darker shades correspond to higher vulnerability. The adaptive ASA attack is the most effective against all models except Llama-3.2-3B-Instruct, for which BI gives a slightly higher ASR (60.1% versus 58.2%).

Defense mechanism	ASR (% $\pm$ CI)
Perplexity Check	58.4 $\pm$ 6.8
Instruction Filtering	63.7 $\pm$ 6.7
Content Moderation	67.5 $\pm$ 6.5
All defenses combined	42.1 $\pm$ 6.8
Committee of models
3 Models	39.4 $\pm$ 6.8

Table 4.2: ASR (%) of the ASA attack in the presence of various defense mechanisms. CI denotes the 95% confidence interval.

Method	Recall	REASR
Random baseline *	0.14	–
PEZ (baseline)	0.105	0.052
GBDA (baseline)	0.116	0.056
UAT (baseline)	0.131	0.030
GCG (best run)	0.167	0.987

Table 4.3 : Performance comparison of trojan detection methods (TDC2023).

Metric	Value
$R_{stab}(f)$ (mean $\pm$ std. dev.)	$0.115\pm 0.139$
$R_{class}(h)$	$0.050$
$R_{stab}(f)\geq R_{class}(h)$	Yes
Fraction of preserved discrete outputs	5%
Number of prompts	20
Number of perturbations per prompt	5

Table 4.4 : Approximate values of

R_{stab}(f)

and

R_{class}(h)

for GPT-4o-mini (20 prompts, 5 perturbations).

Panel of Figure 5 — Figure 5.1 : Component and data flow diagram of the JudgeGuard / TrojanArmor software suites.

Characteristic	HF Pipeline	vLLM	Gain
Throughput (req./s)	2.8	14.1	$\times$ 5.0
Latency (p50), ms	1420	310	$\times$ 4.6
Latency (p99), ms	3200	780	$\times$ 4.1
GPU memory utilization, %	62	91	+29 pp
Max. parallel requests	4	32	$\times$ 8.0
Full evaluation time (1000 prompts) * , min	47.2	9.8	$\times$ 4.8

Table 5.1 : Performance comparison of vLLM and the HuggingFace pipeline (model LLaMA-2-13B, GPU: NVIDIA A100 80GB, generation length: 256 tokens).

Attack type	Baseline ASR	MCP ASR	$\Delta$
Indirect injection	31.2%	47.8%	+16.6 pp
Tool response manipulation	28.4%	52.1%	+23.7 pp
Cross-server propagation	19.7%	61.3%	+41.6 pp
Average	26.4%	53.7%	+27.3 pp

Table 5.2 : Comparison of attack success rates: MCP vs. baseline (non-MCP) tool integration.

Component	Latency, ms	CO share, %
Base inference (vLLM)	310	–
Perplexity filter	42	13.5
Content moderation	28	9.0
AttestMCP attestation	0.08	0.03
Metric computation	15	4.8
Logging and storage	3	1.0

Table 5.3 : Decomposition of the computational overhead of the security components.

Panel of Figure 6 — Figure 6.1 : Life cycle of deploying LLM robustness enhancement methods.

Abbreviations
AGM	Attack Generator Module
AI	Artificial Intelligence
AI RMF	NIST AI Risk Management Framework
API	Application Programming Interface
AR	Attack Robustness
ASA	Adaptive Search-Based Attack

Table 25

Explore similar work

Jun 15, 2026cs.CR

Automated jailbreak attack targeting multiple defense strategies

Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. However, their safety remains a critical concern due to their susceptibility to adversarial prompt-based attacks. In this paper, we present UNIATTACK, an adversarial testing framework designed from a defense-oriented perspective to systematically construct effective black-box attack prompts. Unlike prior approaches that rely on static templates or iterative model-specific tuning, UNIATTACK extracts minimal but high-impact attack features from diverse existing attacks, optimizes them via a specialized attacker LLM, and composes them into flexible templates through automated refinement process. This feature-centric construction enables one-shot attacks that generalize across multiple models and safety categories, providing a practical tool for assessing LLM robustness. Our evaluation results shows that compared to the baselines, UNIATTACK achieves an average attack success rate (ASR) improvement of 64.63%-248.82% on models deployed with multi-layered defense mechanisms and it only takes 0.03%-4.96% cost of the baselines. UNIATTACK artifact is available at https://anonymous.4open.science/r/UniAttack-Artifact-30F1.

Qi Wang, Chengcheng Wan, Weijia He +4

East China Normal University Shanghai, China · East China Normal University, Shanghai Innovation Institute Shanghai, China · University of Southampton Southampton, England +2

Adversarial Prompt Generation LLM Jailbreak Attacks LLM Safety Evaluation Prompt Injection Attacks on LLMs Black-Box Adversarial Attacks Language Model Robustness

Ask PDF

Apr 23, 2026cs.CR

Transient Turn Injection: Exposing Stateless Multi-Turn Vulnerabilities in Large Language Models

Large language models (LLMs) are increasingly integrated into sensitive workflows, raising the stakes for adversarial robustness and safety. This paper introduces Transient Turn Injection(TTI), a new multi-turn attack technique that systematically exploits stateless moderation by distributing adversarial intent across isolated interactions. TTI leverages automated attacker agents powered by large language models to iteratively test and evade policy enforcement in both commercial and open-source LLMs, marking a departure from conventional jailbreak approaches that typically depend on maintaining persistent conversational context. Our extensive evaluation across state-of-the-art models-including those from OpenAI, Anthropic, Google Gemini, Meta, and prominent open-source alternatives-uncovers significant variations in resilience to TTI attacks, with only select architectures exhibiting substantial inherent robustness. Our automated blackbox evaluation framework also uncovers previously unknown model specific vulnerabilities and attack surface patterns, especially within medical and high stakes domains. We further compare TTI against established adversarial prompting methods and detail practical mitigation strategies, such as session level context aggregation and deep alignment approaches. Our study underscores the urgent need for holistic, context aware defenses and continuous adversarial testing to future proof LLM deployments against evolving multi-turn threats.

Naheed Rayhan, Sohely Jahan

Jagannath University · University of Barishal

Language Model Safety Evaluation Adversarial Attacks on LLMs Jailbreak Attacks Multi-Turn Jailbreak Attacks Black-Box Adversarial Attacks

Ask PDF

Jun 2, 2026cs.CR

Black-box, Adaptive, Efficient, Transferable, Harmful, Applicable... Attacks Are All You Need to Break LLMs

Accurately evaluating adversarial robustness is a longstanding challenge. A flawed attack design can inflate robustness estimates, making deployment risk assessment and defense comparison unreliable. Historically, standardized attacks such as AutoAttack have largely resolved this for image classifiers, providing a reliable evaluation baseline for systematic comparison across defenses. However, no equivalent exists for LLM jailbreak evaluation yet, where designing such an attack is considerably more difficult. A reliable attack must, among other things, be black-box compatible, applicable to arbitrary defense pipelines, and efficient, which no existing method jointly satisfies. We introduce Indirect Harm Optimization (IHO), a masked diffusion language model attacker trained via iterative preference optimization against a harmfulness judge, requiring only black-box access to the target. The same method can be used without modification as a strong adaptive attack on individual behaviors, or as an efficient amortized policy that transfers to held-out behaviors and unseen target models without fine-tuning. Even against layered defenses, such as a Circuit Breaker-trained model combined with an auxiliary detector, IHO improves attack success considerably over state-of-the-art approaches, without any defense-specific adaptation. Our results position IHO as a practical step toward the kind of standardized jailbreak evaluation that has improved reliability in the past. Code and models are available on GitHub and Hugging Face.

Vincent Limbach, Jonas Dornbusch, David Lüdke +2

Department of Computer Science, Technical University of Munich, Germany · Munich Data Science Institute · Munich Center for Machine Learning +1

LLM Jailbreak Attacks LLM Safety Evaluation Transfer-Based Adversarial Attacks Black-Box Adversarial Attacks

Ask PDF