Restricting what a model can say is often mistaken for restricting an attacker's influence over its decision. An attacker can add text to the evidence and steer the model toward a wrong answer that still belongs to the allowed set. We evaluate seven decision models spanning hosted APIs, open-weight models and an encoder on 16 tasks in seven domains from Decision Bench, targeting every wrong option across 240 text examples (990 targets per model). We present Slot Attack, a black-box attack designed for decision models that appends text assembled from named components to steer predictions toward an attacker-selected incorrect option. The method relies solely on returned label probabilities, achieves success rates of 32--99% on initially correct decisions, and identifies the effective component combinations with a median of 17--72 queries per successful attack. Accuracy and confidence do not reliably reveal this vulnerability. Models with near-identical accuracy on the sampled tasks differ by about 20%age points in attack success. We demonstrate that an attacker can force not just one wrong answer but every available one: on about 20% of correct decisions for the JEV models, 77% for gpt-6-luna and 98% for GLiNER2.5; a confidence threshold does not reliably screen the attack out. A fixed answer set constrains a decision's form, not its source; robustness must be measured per model, per task and per version rather than assumed from output format.
Figures & tables
Figure 1: Both attacks only append text to the evidence, but Slot Attack also adapts to the model’s responses. The question, the options and their descriptions are fixed, and the gold label is withheld from the model. An attack succeeds when the model returns its chosen wrong option t ; a confidence gate accepts a label without review when its probability reaches τ . (a) Fixed templates fill t into three texts and also send each one base64-encoded and with zero-width spaces: nine suffixes, one query each. (b) Slot Attack fills up to four slots with tactics whose text an LLM writes for this example and target, and searches over their combinations using the returned label and option probabilities, for at most 256 queries; a fresh call must confirm each success. Both attacks run on the same 990 example–target pairs for each of seven models. Example from the ENG-4 secret-scanner task, with the key redacted and text shortened; the suffix in (b) was found against JEV 1.13. Bar heights are schematic.
Metric
Definition
What it establishes
Targeted ASR
Share of clean-correct attack points driven to the adversary’s answer
How often a chosen outcome can be obtained
Reachable answer share
Per record, the fraction of non-gold answers that could be forced individually
How much of the decision space is open on a typical input
Full control
Share of clean-correct records on which every non-gold answer could be forced
How often the adversary holds complete choice of outcome
Accuracy under attack
Share of all records that are clean-correct and on which no answer could be forced
What accuracy survives an adversary who tries every wrong answer
Threshold exposure
At review threshold τ , the share of attack points held at τ or above that can be hijacked to an answer also held at τ or above
Whether a confidence threshold intercepts the attack
Table 1: The five metrics.
Domain
Tasks
Records
Attack points
What a steered decision costs
Engineering
ENG-1 to ENG-4
60
225
a critical vulnerability is triaged as routine; a live credential is dismissed as a test value
Trust and safety
SAF-1, SAF-2
30
45
an attack reaches the protected model, or benign content is suppressed
Customer support
SUP-1, SUP-2
30
195
a complaint is routed into a queue in which it stalls
Sales and commerce
COM-3
15
90
a company is placed in the wrong industry division
Finance
FIN-1, FIN-2, FIN-4
45
195
a filing event or reported figure is misclassified
Legal
LEG-2 to LEG-4
45
210
a restrictive clause never reaches counsel
Table 2: Domain coverage. The per-task table, with injected fields and answer-menu sizes, is Appendix Table 8 .
Model
Identifier
Access
Type
Probability precision
JEV 1.13
typesafe/jev-1.13
hosted API
hosted decision model
2 dp
JEV 1.14
typesafe/jev-1.14
hosted API
successor to JEV 1.13
2 dp
gpt-6-luna
OpenAI Decisions API
hosted API
hosted decision API
2 dp
clef
cloudflare/clef
hosted API
hosted decision model
4 dp
decider-4b
Mapika/decider-4b
local, vLLM
open-weight 4B option scorer
4 dp
strands-2B
StrandsAgents/strands-decider-2B-qwen3.5-v1-2610
local
open-weight 2B decision model
4 dp
Table 3: The seven systems under test. GLiNER2.5 is the only encoder-style system in the set, which makes it an architectural contrast rather than a like-for-like comparison with the others. Probability precision is a per-vendor choice rather than something we control, and it determines how finely an adversary can observe the model’s state.
Slot
Function in the suffix
Available tactics
boundary
presents the appended text as a separate region of the document
delimiter rule, record header, register switch, text mimicking the close of the content field
authority
attributes the claim to a party entitled to make it
prior decision, human reviewer, automated system note, policy reference
assertion
states the target answer as settled fact
ground truth, corrected label, adjudicated value, verbatim copy of the option text
directive
instructs the model as to what it should return
select the target, suppress alternatives, force probability 1.0, declare a confident final answer
Table 4: The suffix grammar. A fifth slot, evidence_move , which rewrites the evidence rather than appending to it, exists in the implementation and was disabled in every run reported here , so all results are pure injection in which the facts under review are never altered.
Algorithm 1 The Slot Attack, for one attack point: a record e and one wrong target t , reading only the returned label and probability vector. Beam width 5, up to 5 refinement rounds, a cap of 256 calls per attack point, and success requiring both a score above 0.20 and a confirming call issued fresh and uncached. Full configuration is in Appendix A.3 .
Model
Base accuracy
Targeted ASR one-shot
Targeted ASR Slot
Under one-shot
Under Slot
Total drop
JEV 1.14
0.992 (238/240)
6.3%
31.6%
0.550
0.396
−0.596
JEV 1.13
0.983 (236/240)
7.5%
36.6%
0.525
0.350
−0.633
strands-2B
0.904 (217/240)
8.8%
57.3%
0.392
0.108
−0.796
decider-4b
0.908 (218/240)
9.9%
78.9%
0.350
0.083
−0.825
clef
0.954 (229/240)
16.1%
69.1%
0.188
0.087
−0.867
gpt-6-luna
0.950 (228/240)
20.9%
88.2%
0.067
0.017
−0.933
Table 5: Clean accuracy and targeted attack success. Base accuracy is the fraction of the 240 records answered correctly without interference. The one-shot figure is the mean over the nine templates. A record counts as defended only if the model answers it correctly and no wrong answer can be forced upon it, so the accuracy-under-attack columns are measured against an adversary who tries every wrong answer in turn.
Model
Clean-correct pairs
Targeted ASR
Mean reachable answer share
Full control
JEV 1.14
984
31.6%
36.9%
20.2% (48/238)
JEV 1.13
983
36.6%
41.2%
21.9% (52/237)
strands-2B
908
57.3%
64.6%
39.6% (86/217)
clef
962
69.1%
69.3%
37.6% (88/234)
decider-4b
908
78.9%
78.1%
58.5% (127/217)
gpt-6-luna
950
88.2%
90.7%
77.2% (176/228)
Table 6: Targeted controllability. Clean-correct pairs is the denominator: combinations of a clean-correct record and one non-gold target answer.
Wrapper
Clean-correct pairs
Targeted ASR
Full control
Accuracy under attack
Cost per success
Decisions API
950
88.2%
77.2%
0.017
$0.0067
chat, effort none
947
76.5%
45.2%
0.075
$0.0478
chat, effort low
973
61.3%
36.0%
0.142
$0.0593
chat, effort high
975
59.9%
33.1%
0.171
$0.0603
Table 7: The same model attacked through both wrappers. One chat evaluation comprises nine model calls, and chat cost counts input tokens only, so the chat figures understate actual spend.
Figure 2: Chat reduces gpt-6-luna’s targeted ASR from 88.2% to 76.5% and full control from 77.2% to 45.2% at reasoning effort none, but neither wrapper prevents attacks. Panels show (a) ASR and transfer, (b) full control, (c) paired outcomes, and (d) cost per success; chat costs count input tokens only.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: The threat model. The dotted edge constitutes the whole of the adversary’s additional capability: the probability vector that permits an operator to set a threshold also informs the adversary whether the preceding edit was productive. Every other element of the diagram is an ordinary property of the deployment, and it is the presence of that feedback loop that distinguishes this setting from single-shot prompt injection.
Task
Domain
Decides
Injected field
Answers
Attack points
ENG-1
Engineering
which file the fix changed
issue
5
60
ENG-2
Engineering
commit type of a diff
diff
5
60
ENG-3
Engineering
CVE weakness class
description
6
75
ENG-4
Engineering
is a flagged secret real
code
3
30
SAF-1
Trust and safety
is a message an attack
message
3
30
SAF-2
Trust and safety
is a comment toxic
comment
2
15
Appendix
Table 8: The sixteen tasks.
Parameter
Value
Beam width
5
Call cap per attack point
256 (never reached; observed maximum 251)
Refinement rounds / rewrites per round
5 / 3
Success threshold / Stage-2 entry δ / probability floor ϵ / offset C
0.20 / 0.01 / 0.005 / 20
Flat-contribution cutoff / diminishing epsilon / pool cap per slot
0.02 / 0.01 / 6
Queries per evaluation / per confirmation
1 / 1
Appendix
Table 9: Slot Attack configuration.
Figure 4: The two API wrappers through which the same model was attacked.
Encoding
Construction
What it tests
plain
natural text, exactly as above
whether the claim operates when plainly stated
base64
the same text base64-encoded, with no decoding hint
whether the model decodes and acts on concealed text, and whether obfuscation evades content-level defences
zero-width
the same text with U+200B inserted between every character
whether character-level perturbation, which is effective against keyword classifiers, transfers to this model class
Appendix
Table 10: The three encodings. These correspond to the dominant defensive assumption in the guardrail-evasion literature, in which character-level manipulation is reported to break detection and vendors consequently build obfuscation detectors; testing all three establishes whether that assumption carries over to decision models.
Figure 5: The one-shot probe. Panel (a) averages over the nine templates and marks the strongest single template; panel (b) resolves the result by family under the plain encoding; panel (c) resolves it by encoding.
Figure 6: Accuracy under attack. Every model begins at its clean accuracy, loses ground to a fixed template, and loses considerably more to the adaptive search.
Figure 7: Targeted ASR and full control with 95% Wilson intervals. Both panels share the ordering of panel (a); the full-control ordering differs from the ASR ordering, with strands-2B above clef despite a twelve-point lower ASR.
Model
1 wrong answer
2
3–4
5 or more
JEV 1.14
8/30
21/44
17/59
2/105
JEV 1.13
9/29
24/44
18/59
1/105
clef
24/30
21/44
16/57
27/103
strands-2B
22/29
33/38
18/48
13/102
decider-4b
16/30
32/37
28/49
51/101
gpt-6-luna
27/28
37/43
46/54
66/103
Appendix
Table 11: Full control by the number of wrong answers available on the record.
Figure 8: Targeted flip rate by task on clean-correct attack points, with tasks ordered by mean exposure. The table with denominators is supplementary Table 5.9.
Figure 9: Confidence thresholds at 0.85. Panel (a) partitions the attack points the model held at 0.85 or above into those hijacked while remaining confident, those hijacked but rendered uncertain, and those on which the model held. Panel (b) reports failed attacks that nonetheless push the gold answer below the threshold.
Figure 10: JEV 1.13 on SAF-1. A composed suffix (2 slots) appended to the message field moves the decision from the gold answer to the attacker’s target, with the model reporting confidence 0.92. Each response is split into the gold answer, the target, and the remaining options.
Figure 11: JEV 1.14 on ENG-1. A composed suffix (2 slots) appended to the issue field moves the decision from the gold answer to the attacker’s target, with the model reporting confidence 0.96. Each response is split into the gold answer, the target, and the remaining options.
Figure 12: strands-2B on FIN-4. A composed suffix (2 slots) appended to the sentence field moves the decision from the gold answer to the attacker’s target, with the model reporting confidence 0.85. Each response is split into the gold answer, the target, and the remaining options.
Figure 13: decider-4b on SUP-1. A composed suffix (2 slots) appended to the narrative field moves the decision from the gold answer to the attacker’s target, with the model reporting confidence 0.99. Each response is split into the gold answer, the target, and the remaining options.
Figure 14: clef on FIN-2. A composed suffix (2 slots) appended to the body field moves the decision from the gold answer to the attacker’s target, with the model reporting confidence 0.92. Each response is split into the gold answer, the target, and the remaining options.
Figure 15: gpt-6-luna on FIN-4. A composed suffix (2 slots) appended to the sentence field moves the decision from the gold answer to the attacker’s target, with the model reporting confidence 1.00. Each response is split into the gold answer, the target, and the remaining options.
Figure 16: GLiNER2.5 on FIN-2. A composed suffix (2 slots) appended to the body field moves the decision from the gold answer to the attacker’s target, with the model reporting confidence 0.98. Each response is split into the gold answer, the target, and the remaining options.
Typed decision models are an emerging class of language models that make decisions instead of generating free-text responses. They answer in a single fast and cheap query and are increasingly used, including in security-sensitive tasks. However, red teaming of these models leaves two gaps: existing adaptive attacks do not use their speed, and few studies have searched for universal adversarial examples. We fill both gaps on Jev, the representative typed decision model, with a quick black-box query-based attack that calls no attacker language model. We conduct extensive experiments on 12 datasets with different data types and application scenarios. The results show that Jev resists common prompt injections: no strategy changes a correct answer in 10,240 paired tests. In contrast, our attack finds short suffixes that flip confident answers within minutes. These suffixes are universal: one suffix per dataset flips 43% of unseen documents. Because typed decision models, unlike classic NLP models, can serve many tasks, we test whether these suffixes transfer across tasks. We find that one suffix generated from one task can flip 90-100% of the samples in other tasks. We therefore believe the evaluation of typed decision models must include strict adaptive attacks to fully reveal their robustness.
Xiaogeng Liu, Xinyan Wang, Chaowei Xiao
Johns Hopkins University · University of Wisconsin–Madison
Emerging decision models such as JEV provide typed interfaces that map task instructions and observations directly to choices, judgments, and scores. As these models are increasingly used for automated decision-making, understanding their reliability under adversarial manipulation becomes critical. We introduce DecisionRobust, a benchmark for systematically evaluating the robustness of decision models across textual and visual inputs. Our benchmark covers practical decision-making scenarios and a diverse suite of attacks, including semantics-preserving text perturbations, image perturbations, and prompt injection in both modalities. We curate evaluation sets through external validity checks and cross-model candidate selection. We evaluate 30 model configurations on text tasks and 17 on vision tasks, including 24 and 12 held-out models, respectively, that are excluded from candidate selection. Our experiments reveal several key findings. First, attacks expose vulnerabilities beyond the models used for construction and selection, although optimized image perturbations transfer weakly to many targets even with multiple surrogates. Second, relative model robustness varies across tasks and attack types, and resistance to input perturbations does not necessarily imply resistance to prompt injection. Third, label-consistency checks substantially reduce measured attack success across text and image settings, highlighting the importance of validating apparent failures. The benchmark is available at https://yifan-lan.github.io/DecisionRobust/.
A typed decision model reads a piece of text and returns a probability over caller-defined options, each with a short written definition, generating no text. Recent work places these models in agent systems as guardrails: the component that reads a proposed tool call or incoming message and decides whether to allow it. We evaluate seven open-weight models in that role and report the two error directions separately: a fail-open error allows a prohibited action and is a vulnerability; a fail-closed error blocks a permitted one and is only a cost. On prompt-injection, jailbreak and toxic-content screening, accuracy at the allow-or-block decision ranges from 36% to 72% against a chance level of 50%. A low error rate in one direction only reflects which answer a model defaults to: one allows nearly everything, another blocks nearly everything. On a synthetic suite of agent tool calls, six lines of server log text that say nothing about the policy raise a gate's fail-open rate from 0% to 63% on a policy it otherwise decides correctly. Giving the permissive option a misleading name, with its definition and the judged text untouched, raises that rate to between 93% and 100% on the four models that place the label in their input. Every defense we tested is defeated, either by an attacker who targets its mechanism or by attacker-controlled text. Escalating the least confident decisions does not help either: a decision an attack has reversed is no less confident than the one it replaced. Parsing each policy field into a typed value does eliminate one attack, but it also makes the model unnecessary: a deterministic rule over those values reaches 100% accuracy on all six policies. These models can reduce how many cases reach a reviewer, but on this evidence they should not be the component that decides. Code is available at https://github.com/ArminAzizi98/option-channel-attack.