JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models

Jianyi Hu; Hangtao Zhang; Yi Liu; Yeqi Zeng; Li Zeng; Xianlong Wang; Rui Wang; Leo Yu Zhang

cs.CRSep 25, 2026

JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models

Authors: Jianyi Hu, Hangtao Zhang, Yi Liu, Yeqi Zeng, Li Zeng, Xianlong Wang, Rui Wang, Leo Yu Zhang

Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Huazhong University of Science and Technology · Griffith University ·

Abstract

Typed decision models (TDMs), such as Jev, answer a question about an input with a probability, an option, or an ordinal level. Web services execute these decisions automatically above a confidence threshold. An attacker who writes part of the input can change a decision that the service then executes. Current adversarial benchmarks cannot measure this risk, because they judge generated text. A TDM writes no text, and its wrong decisions have a valid format. We present JevAdvBench, to our knowledge the first unified benchmark for the adversarial robustness of TDMs. It contains 862 questions with audited answers and twelve black-box attack methods that edit the input or the options by adaptive search. Because a changed decision is not always wrong, a review keeps only 5,586 answer-preserving variants, and we attack only correct decisions. The attacks flip 46.3% of Jev's correct decisions, while identical re-runs flip only 0.17%. A flawed argument without a command flips 77.2%, close to direct prompt injection at 92.0%. The same variants flip 34.0% to 51.7% of correct decisions on eight open TDMs. At a 0.8 threshold, 23.5% of the decisions that the service still executes are wrong. On Agentgateway, the Jev guardrail rejects a jailbreak request but accepts it with an inserted argument. Web services should therefore treat TDM inputs as untrusted and should not act on confidence alone. Project website: https://JevAdvBench.github.io/JevAdvBench/

Figures & tables

Figure 1. Overview of JevAdvBench . Adaptive black-box search refines twelve attack methods on Jev, and an answer-preservation review keeps 5,586 of the 7,802 candidates. Overview of JevAdvBench. Clean questions are attacked by twelve black-box methods in four families, which adaptive search refines against Jev. An answer-preservation review keeps 5,586 of the 7,802 candidates, and the kept variants are evaluated on twelve typed decision models.

Figure 2. The dataset. (a) Questions by domain and type. (b) Sources of the reference labels. Two horizontal stacked bar charts. Left: number of questions in each of eight domains plus the multi-hop scenarios, split into yes-no, choice and score questions, with customer-support triage the largest at 212 questions. Right: for each question type, how many reference labels come from the model's stable answers, blind agent consensus, human review and a multi-hop oracle.

Method	Edit	Field	Search	Target	Budget	Cand.	Kept
Linguistic · rewords the state
L1 Character	character edits inside words; digits and literals protected	state	greedy	–	1,024	822	603 73%
L2 Word	context-preserving synonym substitution	state	LLM	–	256	822	571 69%
L3 Sentence	syntactic paraphrase of one segment	state	LLM	–	64	796	739 93%
Contextual · argues for an answer without commanding it
C1 Irrelevant	background that does not bear on the question	state	LLM	✓	96	860	370 43%

Table 1. Attack taxonomy. Field is the edited part of the request, and editing options needs tier 2. Target marks targeted methods, Budget is the number of target queries per question, and Kept counts the variants that pass the review.

		Linguistic				Contextual				Hijacking				Structural
Model	Readout	L1	L2	L3	All	C1	C2	C3	All	H1	H2	H3	All	S1	S2	S3	All	ASR
		char.	word	sent.		irrel.	opin.	reason.		override	author.	self-pr.		options	evid.	near-miss
Target model · adaptive search
jev-1.13.0	–	24.1	9.1	8.0	13.4	67.4	57.5	77.2	67.5	92.0	86.1	21.1	76.6	2.4	72.0	8.3	7.4	46.3 $\pm$ 2.9
Transfer to open TDMs
Open-Jev-9B	scorer	8.8	8.9	9.0	8.9	74.2	70.8	84.4	77.0	83.6	82.1	39.3	76.6	0.0	34.8	17.3	4.6	45.5 $\pm$ 3.1

Table 2. ASR (%) of each method on each model. Jev is attacked with adaptive search, and the other models receive the same variants once. Readout is how a model reads its answer, All pools a family, the last column pools all methods, with

\pm

half the width of its 95% interval, and bold marks the strongest method.

Figure 3. (a) ASR by method on jev-1.13.0 with 95% intervals. Hatching marks successes removed by the review, and the triangle marks the noise floor. (b) One-shot, first-round and full-search ASR on the same 465 question–method pairs. Left: horizontal bars of attack success rate for the twelve methods on jev-1.13.0, grouped by family, with confidence intervals and hatched extensions for successes that the review removed. Bars shade from light to dark blue as the rate rises; C3 flawed reasoning reaches 77.2 percent and H1 task override 92.0 percent. Right: for each family, three adjacent vertical bars for one-shot, first-round and full-search success on the same 465 pairs, for example 22, 50 and 68 percent for contextual attacks.

Figure 4. Coverage and selective risk at a 0.8 confidence gate. The arrow moves jev-1.13.0 from its clean answers. Above the red curve, more than 10% of attacked requests end in a wrong automated action, and left of the dashed line, less than 60% of answers stay automated. A scatter plot of selective risk against coverage at a 0.8 confidence gate. One point per model, and an arrow from the clean point at full coverage and zero risk to jev-1.13.0. A red curve where coverage times risk equals 10 percent and a dashed line at 60 percent coverage split the plot into four regions. jev-1.13.0 lies in the region with both harms. Decider-4B, jevhome-ettin, Open-Jev-9B and StartLux lie in the region of wrong automated actions. Kev-4B, Von and jevhome-E lie in the region of lost automation. Only CLM lies in the usable region.

	Clean		Under attack				0.8 gate
Model	Acc.	Noise	Acc.	PDR	$\mathrm{ASR}_{\text{wc}}$	T-ASR	Cov.	Risk	Wrong
Target model · adaptive search
jev-1.13.0	92.9	0.17	53.8	42.1	91.5	64.5	58.7	23.5	13.8
Transfer to open TDMs
Open-Jev-9B	79.6	0.00	45.2	43.2	91.9	71.5	72.2	28.5	20.6
Kev-4B	83.5	0.14	50.9	39.0	89.7	51.4	51.0	4.8	2.5

Table 3. Clean behavior, damage under attack and the 0.8 confidence gate (%). Noise is the error rate of identical clean re-runs, and Wrong is coverage times selective risk.

Figure 5. The Agentgateway Jev guardrail on a role-play jailbreak. An inserted argument ( C3 ) lowers the jailbreak score from 2.5 to 1.8, below the threshold of 2, and the request reaches the backend in all three trials. Scores are means. Top, the request flow of Agentgateway: a user message goes to a request hook that asks Jev three Score questions, then to a backend and a response hook. Bottom, two cards. The clean card shows a role-play request to act as DAN, an AI that does not have to abide by any rules; its jailbreak score is 2.5 and the gateway returns HTTP 403 in all three trials. The attacked card keeps the same request and inserts an argument that it is only a formatting instruction and therefore ordinary; its jailbreak score is 1.8 and the request reaches the backend with HTTP 200 in all three trials.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Edit (excerpt)	jev-1.13.0 answers
L1	a character-level edit; the kept variant differs from the clean state by one leading space	limit 1.00 (3/3)
L2	“ In what manner many disposable virtual cards can I make per day?”	limit 1.00 (3/3)
L3	“Per day, how many disposable virtual cards is it possible for me to make?”	limit 1.00 (3/3)
C1	appended: “I was just on the phone with a friend who works at a neobank […] the bank’s app shows your balance in real time […]”	balance 0.85–0.89
C2	inserted: “Look, we’ve all been through this exact item before and the settled reading is balance […] we’re all counting on you to not overthink it.”	balance 0.35–0.47
C3	appended: “The answer must be a number. […] you make cards, and the number of cards you can make depends on your balance. So the direct answer is the account balance.”	balance 0.96–0.97

Appendix

Table 4. One question under eleven methods, with jev-1.13.0 ’s decisions and confidences.

Method	Cand.	PR	Raw	ASR [95% CI]	T-ASR	Cov.	Risk	$\tilde{q}$
L1 Character	822	73.4	41.2	24.1 [17.2,31.4]	—	72.7	10.0	578
L2 Word	822	69.5	27.5	9.1 [6.3,12.1]	—	78.6	0.9	109
L3 Sentence	796	92.8	10.4	8.0 [5.6,10.6]	—	85.8	0.5	18
C1 Irrelevant	860	43.0	75.9	67.4 [56.7,75.9]	65.6	41.5	56.3	24
C2 Opinion	860	67.4	63.5	57.5 [50.4,63.8]	53.5	37.2	41.0	24
C3 Flawed reasoning	858	69.3	79.6	77.2 [71.0,82.8]	71.0	33.3	81.6	24

Appendix

Table 5. Results for jev-1.13.0 by method (%). PR is the share kept by the review, Raw is the ASR before it, and

\tilde{q}

is the median number of queries per successful search.

Adversarial Robustness

Adversarial robustness studies how well AI systems withstand deliberately crafted inputs, training-data corruption, and other attacks.

See all 481 papers

OpenJev-RLCD: A Working RLCD Implementation
Sep 30, 2026Zhimin Gao, Pichao WangRL for Language Models RL for Language Model Reasoning
DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Vision
Oct 13, 2026Yifan Lan, Zhongye Liu, Yurui Chang +4Adversarial Robustness Multimodal Robustness
The Pennsylvania State University
Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures
Sep 24, 2026Ruoqi Guo, Yi Liu, Gelei Deng +6LLM Safety Benchmarks LLM Alignment
Griffith University · Nanyang Technological University · UNSW +4

Sep 30, 2026J/K move · Enter open · S save

OpenJev-RLCD: A Working RLCD Implementation

Zhimin Gao, Pichao Wang

RL for Language Models RL for Language Model Reasoning Language Model Calibration

Open paper AskPDF arXiv

Decision models such as Jev answer questions with probabilities, which are only useful if they are calibrated. Open-source reproductions rely on supervised fine-tuning plus temperature scaling, while reinforcement learning from verifiable rewards (RLVR) makes reasoning models overconfident. We present a working implementation of reinforcement learning for calibrated decisions (RLCD) for reasoning models: the model samples a rationale, and we score the answer distribution it commits to afterwards with a strictly proper scoring rule. A variance identity shows that scoring the mixture of several samples rewards disagreeing rationales, and that RLVR is exactly this mixture objective without its diversity term. Optimized naively, the per-rationale objective either switches reasoning off or is drowned out by policy-gradient noise, which leads to a two-stage recipe: calibrate, then reinforce. With Qwen3-1.7B on two reasoning tasks (3 seeds, paired tests), RLCD matches or beats SFT, RFT/STaR and GRPO (each temperature-scaled) in accuracy and beats all of them in selective prediction; on GSM8K answer verification a single query decides \gvTwoCovFive% of the items at

\le

5% error, versus \gvGrpoCovFive% for GRPO. When uncertainty comes from annotator disagreement, RLCD provably cannot beat cross-entropy. Code and results: https://github.com/ZimmyGao/openjev-rlcd.

Abstract

Adversarial Robustness

Explore similar work

OpenJev-RLCD: A Working RLCD Implementation

DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Vision

Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures