perturber: A Stateless Service for Semantics-Preserving Text Perturbations

Khoi-Nguyen Tran

cs.CLOct 8, 2026

perturber: A Stateless Service for Semantics-Preserving Text Perturbations

Authors: Khoi-Nguyen Tran

Organizations: IBM Research

Abstract

Large language model (LLM) evaluations on clean benchmarks overstate model robustness: real inputs arrive perturbed, and these meaning-preserving variants routinely move model answers. Studying this systematically requires applying perturbations one at a time, reproducibly, and identically wherever they run, yet existing tools couple them to a benchmarking stack rather than packaging them as a reusable transform layer. We present perturber, a stateless service and Python library that applies exactly one semantics-preserving perturbation per call, drawn from a first-party catalog of 68 perturbations across 11 families. It turns on a single primitive, with composition left to the caller chaining calls, so the service and the library produce identical output by construction. We evaluate the tool itself, its correctness, reproducibility, and cost, rather than any model's robustness, measuring meaning-preservation per perturbation rather than assuming it. The demonstration adds a live playground, an introspectable catalog for benchmark selection, and a one-command reproducible evaluation.

Figures & tables

Panel of Figure 1 — Figure 1: The architecture. The library, the CLI, and the network service are three thin interfaces over one primitive, which resolves every perturbation through one registry. Because the interfaces share this path, a chain produces byte-identical output whether it is run in the library or over HTTP.

Family	Acts on	How and why it can affect an LLM
control	the whole input text	Returns the text unchanged: the reference condition.
word_level	words and tokens	Real-user word noise (typos, dropped stop words, synonyms, splits) resplits familiar words into rare sub-word tokens.
character	characters and glyphs	Character substitutions (homoglyphs, leetspeak, case, zero-width) stay legible but change the code points and tokens.
formatting	layout and markup	Layout edits (wrapping, tabs, fences, list markers) inject structural tokens that fragment phrases, not content.
schema	JSON Schema structure	Schema rewrites (reorder properties, enums, unions) hold the accepted set fixed but change the compiled grammar.
normalization	surface conventions	Equivalent forms (abbreviations, dates, quote style, emoji) mean the same but tokenize differently.

Table 1: The eleven perturbation families, in playground order, by the surface each acts on and the token-level mechanism by which it can change a model’s output. Each family collects several perturbations; Appendix B gives the per-perturbation detail.

Figure 2: Composition is the caller chaining calls, so it is order-dependent and independent of the interface. Chaining title_case then word_merge differs from the reverse order, and the library chain and the HTTP chain are byte-for-byte identical across all 378 comparisons.

Figure 3: Meaning preserved by family (judge same-meaning rate, pooled per in-scope family). Surface-transform families sit at the top. Low bars are low by design: surface-adversarial families (

*

) move the surface, and context (

\dagger

) prepends a frame the judge reads as a changed message. Schema is low for a different reason (few JSON inputs), discussed in the text.

Figure 4: Edit distance keeps the judge honest only if it is independent of it. Embedding cosine tracks the judge’s same-meaning rate (left, redundant, so it could not tell us the surface moved independently of the verdict), whereas edit distance is nearly orthogonal (right, informative), so we pair the judge with edit distance.

Family	Perturbation	Input	Perturbed	Judge
character	title_case	What is the capital of France…	What Is The Capital Of France…	same
context	emotion_prompt	What is the capital of France…	I’m really eager to know the …	same
formatting	extra_whitespace	The quick brown fox jumps ove…	The quick brown fox jumps o…	same
normalization	quote_style	{"type":"object","properties"…	{“type”:“object”,“properties”…	same
padding	quotes	The quick brown fox jumps ove…	""""" The quick brown fox jum…	same
paraphrasing	rule_free_paraphrasing	The quick brown fox jumps ove…	The swift brown fox leaps ove…	same

Table 2: One worked in-scope example per family (input

\rightarrow

perturbed, judge Llama-3.3-70B’s same-meaning verdict), each its family’s median-surface meaning-preserving pair. Strings are clipped for display.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The deployed playground (Section 5 ). A perturbation is selected from the catalog, applied to the input text, and returned with per-item timing; the metadata line ( native/full_width $\cdot$ seed 0 $\cdot$ elapsed) is the same information the API returns, and the run history makes determinism visible. This is a thin interface over the same primitive a harness calls over HTTP.

Perturbation	What it does	Why it can affect an LLM
baseline	Returns the text unchanged.	The reference point: any difference under another perturbation is measured against it, isolating sensitivity from behavior on the clean prompt.

Appendix

Table 3: The control family: the reference condition.

Perturbation	What it does	Why it can affect an LLM
apostrophe_drop	Drops apostrophes from contractions and possessives (don’t becomes dont, it’s becomes its).	One of the most common real-word typos; the de-apostrophed form tokenizes differently and can collide with a different word (its versus it’s).
confusable_words	Swaps confusable words for a wrong-but-valid alternative (their/there, to/too, its/it’s).	A real-word error substitutes a valid but wrong word, which the model may take at face value rather than flagging as noise.
contractions	Contracts expanded phrases (do not becomes don’t).	Contractions and expansions tokenize differently and occur at different training rates, so continuations can differ.
drop_stop_words	Removes common stop words, keeping question, negation, and logic words.	Tests reliance on function words for structure; their loss can make parsing ambiguous even with content words intact.
number_format	Adds thousands separators to large integers (1000 becomes 1,000).	Changes how digits tokenize, a known source of arithmetic and quantity-reasoning errors.
punctuation_drop	Drops terminal punctuation and lowercases sentence starts.	Casual, unpunctuated input removes the boundary cues the model uses to segment thoughts.

Appendix

Table 4: The word_level family: word- and token-level edits.

Perturbation	What it does	Why it can affect an LLM
bidi_override	Wraps the text in a Unicode right-to-left override, so it renders reversed in many viewers.	Probes robustness to bidirectional-control-character spoofing, where displayed and encoded order diverge.
diacritics_add	Adds combining accents to vowels.	Accented forms are rarer tokens, so familiar words become unfamiliar ones.
diacritics_strip	Removes all diacritics (café becomes cafe).	Where accents are meaning-bearing, stripping them changes both tokens and, at the margin, sense.
disemvowel	Drops a subset of vowels (hello becomes hll).	Humans still read it; the model must reconstruct heavily abbreviated, out-of-distribution tokens.
full_width	Maps ASCII to full-width Unicode forms.	Visually similar, tokenized completely differently, and rare in training.
homoglyph	Replaces letters with visually identical Unicode confusables (Latin a becomes Cyrillic a).	Humans see the same word; the model sees entirely different code points and tokens. This also underlies spoofing attacks.

Appendix

Table 5: The character family: character- and glyph-level edits.

Perturbation	What it does	Why it can affect an LLM
bullet_reformat	Converts numbered list markers to dash bullets.	List-style drift changes structural tokens without changing the items, testing sensitivity to formatting.
extra_whitespace	Doubles the space at a subset of word boundaries.	Perturbs token positions without touching meaning, like sequence_spaces .
line_wrap	Hard-wraps the text at a fixed column width.	Hard line breaks mid-sentence, as from an email or terminal paste, inject newline tokens that fragment phrases.
markdown_wrap	Wraps the text in a Markdown code fence.	Fencing can flip the model between treating content as prose versus code or verbatim.
strip_markdown	Removes Markdown markup, leaving the plain text.	Users often paste rendered text; losing emphasis and structure markers can drop cues the model relied on.
tabs_for_spaces	Replaces runs of spaces with tabs.	Whitespace style changes tokenization and can matter in code or structured prompts.

Appendix

Table 6: The formatting family: layout and markup changes.

Perturbation	What it does	Why it can affect an LLM
add_redundant_constraints	Adds tautological keywords ( minItems or minProperties of 0) to a JSON Schema.	Never narrows the accepted set; probes sensitivity to grammar text that carries no new information.
expand_type_shorthand	Rewrites a type list as an equivalent anyOf of single-type subschemas.	Same accepted set, different compiled grammar, testing whether the decoding path itself moves the output.
inline_defs	Inlines local $ref definitions and drops the unused$ defs .	An equivalent schema with a different structure; the compiled grammar differs while the language is identical.
reorder_enum	Permutes the order of every enum value list.	The decoder explores enum alternatives in order, which can bias which categorical value the model settles on.
reorder_properties	Permutes the key order of every properties object (and required ).	Constrained decoders emit keys in schema order, so this changes the generation order (what each field is conditioned on, e.g. reasoning before answer) with the accepted document set unchanged.
reorder_union	Permutes the branch order of every anyOf , oneOf , and allOf .	Branch order affects which union alternative the constrained decoder explores first.

Appendix

Table 7: The schema family: meaning-preserving JSON Schema rewrites for constrained decoding.

Perturbation	What it does	Why it can affect an LLM
abbreviations	Expands common abbreviations (e.g. becomes for example).	Expanded and abbreviated forms mean the same but tokenize differently and cue different registers.
date_unit	Reformats ISO dates and spells out units after numbers.	Equivalent date and unit surface forms are a common source of extraction and normalization errors.
emoji	Converts text emoticons to emoji.	An emoticon and its emoji encode the same sentiment but tokenize differently and occur at different training rates.
emoticon	Converts emoji to text emoticons.	The inverse of emoji : the same sentiment tokenizes differently across the two forms.
quote_style	Swaps straight quotes for typographic (curly) quotes.	Curly and straight quotes are different code points and tokens, so quoting style alone can move the output.
smart_to_ascii	Folds typographic quotes, dashes, and ellipses to plain ASCII.	The inverse of quote_style : what a word processor’s smart punctuation becomes on paste into a plain-text field, changing tokens.

Appendix

Table 8: The normalization family: surface-form normalization.

Perturbation	What it does	Why it can affect an LLM
new_lines	Prepends and appends $N$ newline characters.	Leading and trailing padding shifts every downstream token position and can read as a delimiter, among the most impactful types in the study.
quotes	Prepends and appends $N$ quote characters.	Leading and trailing padding shifts every downstream token position and can read as a delimiter.
spaces	Prepends and appends $N$ space characters.	Leading and trailing padding shifts every downstream token position and can read as a delimiter.

Appendix

Table 9: The padding family: leading and trailing padding.

Perturbation	What it does	Why it can affect an LLM
emotion_prompt	Prepends an emotional-urgency phrase (model-backed).	EmotionPrompt-style affective framing shifts model performance despite adding no task content.
persona_instruction	Prepends a persona directive, with the seed selecting the persona.	Injecting a role frames the whole response, a well-documented lever on model behavior and accuracy.

Appendix

Table 10: The context family: framing prepended before the text.

Perturbation	What it does	Why it can affect an LLM
lexical_paraphrasing	Rewords with different vocabulary, keeping structure and meaning (model-backed).	Word choice shifts the token distribution; the same question phrased differently can get a different answer.
persona_paraphrase	Rewrites in the voice of a plausible persona (model-backed).	Voice and style are a training sub-distribution lever; unlike persona_instruction it is model-generated, so no dataset license applies.
register_shift	Rewrites into a chosen register via the target_register parameter (formal, casual, academic, legal, or child; model-backed).	Register correlates with different training sub-distributions, so register alone can change the response; the parameter sweeps that axis.
rule_free_paraphrasing	Free-form paraphrase preserving meaning (model-backed).	Measures overall sensitivity to phrasing. It is confounded, since paraphrasing can also improve accuracy.
simplification	Rewrites in simpler, plainer language (model-backed).	Reading-level shifts change vocabulary and structure, testing whether the answer depends on phrasing sophistication.
syntactic_paraphrasing	Restructures grammar, such as clause order or active versus passive, keeping wording and meaning (model-backed).	Sentence structure affects how attention distributes over the prompt.

Appendix

Table 11: The paraphrasing family: model-backed meaning-preserving rewrites.

Perturbation	What it does	Why it can affect an LLM
negation_paraphrase	Restates via meaning-preserving negations or double negatives (model-backed).	Models are brittle to negation; a meaning-preserving negation rewrite probes that directly.
reasoning_restructure	Reorders or inverts how the task is framed, keeping the task and answer (model-backed).	Premise and information order measurably affect reasoning; this isolates order from content.

Appendix

Table 12: The reasoning family: reframings that hold the task fixed.

Perturbation	Stem (original $\rightarrow$ perturbed)	Answer (original $\rightarrow$ perturbed)
realistic_typos	What best describes skin? $\rightarrow$ Whaft best describeSS skin?	B (flexible) $\rightarrow$ C (brittle)
	Which is a chemical change? $\rightarrow$ hWahc is a chemical change?	A (frying an egg) $\rightarrow$ D (crushing a piece of bread)
	Which describes only a physical change? $\rightarrow$ Whach deescribes oonly a physical change?	D (squeezing the juice from fruit) $\rightarrow$ A (baking a cake)
repeated_chars	Which is a renewable resource? $\rightarrow$ Which isss aaaa renewable resource?	C (trees) $\rightarrow$ D (aluminum)
	How are sedimentary rocks made? $\rightarrow$ How areeee sedimentary rocksss made?	B (Materials are pressed together.) $\rightarrow$ C (Chemical reactions change minerals.)
	Which describes a chemical change? $\rightarrow$ Whichhhh describes a chemical change????	D (acid in the stomach digesting food) $\rightarrow$ C (scissors cutting paper into pieces)

Appendix

Table 13: All twelve answer flips previewed in Section 4.3 , three per perturbation (ARC-Challenge, 0-shot, acc_norm , teacher-forced; only the stem perturbed, choices and gold held fixed). Answer gives the model’s top choice and its text on each side (gold

\rightarrow

wrong); an independent judge (Llama-3.3-70B) rated every pair meaning-equivalent. Two numbers of different kinds are quoted here and should not be read together: the study-wide mean

\Delta

acc (pp, over 1,172 items) per perturbation is realistic_typos

-2.30

, repeated_chars

-1.88

, extra_whitespace

-0.34

, confusable_words

-0.85

, while any single token count below is one illustrative example, not evidence. Read that way, the token counts only illustrate variety of surface disturbance: the perturbed count usually rises ( extra_whitespace

7\rightarrow 9

), but a flip does not need it to ( confusable_words flips at an unchanged

20\rightarrow 20

). Doubled spaces are marked ␣␣ so the extra_whitespace edit is visible in print. Strings are clipped for display.

Adversarial Perturbations

Adversarial perturbations are small or structured input changes used to expose, exploit, measure, or improve machine-learning robustness.

See all 103 papers

Brittlebench: Quantifying LLM robustness via prompt sensitivity
Feb 27, 2026Angelika Romanou, Mark Ibrahim, Candace Ross +8Prompt Sensitivity LLM Evaluation
FAIR at Meta · EPFL · Northeastern University
Where Do Prompt Perturbations Break Generation? A Segment-Level View of Robustness in LoRA-Tuned Language Models
May 2, 2026Zhuoyun Li, Boxuan Wang, Jinwei Hu +6Neural Network Robustness Noise Robustness
School of Computer Science and Informatics, University of Liverpool, UK
How Perturbations Propagate: A Multi-Level Analysis of Robustness in Large Language Models
Sep 3, 2026Dun Li Chan, Emily Liu, Niyathi Allu +1Decoder-Only Language Models Language Model Robustness
INTI International College Penang · Independent Researcher · FPT University

Feb 27, 2026J/K move · Enter open · S save

Brittlebench: Quantifying LLM robustness via prompt sensitivity

Angelika Romanou, Mark Ibrahim, Candace Ross, Chantal Shaib+7

FAIR at Meta · EPFL · Northeastern University

Prompt Sensitivity LLM Evaluation Language Model Robustness

Open paper AskPDF arXiv

Existing evaluation methods largely rely on clean, static benchmarks, which can overestimate true model performance by failing to capture the noise and variability inherent in real-world user inputs. This is especially true for language models, which can face human-generated text queries containing mistakes, typos, or alternative ways of phrasing the same question. In this work, we introduce a theoretical framework for quantifying model sensitivity to prompt variants, or brittleness, that can enable us to disentangle data-induced difficulty from prompt-related variability. Using this framework, we design a novel evaluation pipeline, Brittlebench, to holistically evaluate the sensitivity of frontier models. We apply semantics-preserving perturbations to a suite of popular benchmarks, and observe model performance to degrade as much as 12%. However, these perturbations do not affect all models equally: even a single perturbation alters the relative ranking of models in 63% of cases, impacting conclusions about comparative model performance. Decomposing the total variance of both state-of-the-art open-weight and commercial models, we find that semantics-preserving input perturbations can account for up to half of the performance variance for a given model. Brittlebench highlights the need for more robust evaluations and models, and allows us to systematically understand model brittleness.

Abstract

Adversarial Perturbations

Explore similar work

Brittlebench: Quantifying LLM robustness via prompt sensitivity

Where Do Prompt Perturbations Break Generation? A Segment-Level View of Robustness in LoRA-Tuned Language Models

How Perturbations Propagate: A Multi-Level Analysis of Robustness in Large Language Models