The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval

Jianbo Chen

cs.IROct 10, 2026

The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval

Authors: Jianbo Chen

Organizations: Tsinghua University Beijing, China

Abstract

Jev's documented interface evaluates typed questions independently against shared evidence. We audit request independence in seven open Jev-style checkpoints on 276 constructed three-passage states from 92 TREC retrieval queries. Paired interventions separately change requested questions, displayed evidence, and candidate eligibility. Request expansion induces strict binary flips in 9.1%/8.0% of Nimble's judgments and changes 21.4%/18.9% of selection-or-abstention actions in TREC DL2019/2020 under a threshold of 0.5 fixed before inference. Correct-action rates change by 0.0/+8.2 percentage points, respectively, with 95% query-bootstrap intervals of [-6.0,6.0] and [3.8,12.6]. Field masking traces all 70 request-set flips to the schema fields serialized into each scoring prompt. In Von, fixing a shared separator's position eliminates two eligibility-induced preference flips in selected diagnostic states. Laya-typed's default CUDA path produces five passage swaps without strict binary flips, and both yearly action-change intervals include zero; in an FP32 rerun, its probability movement vanishes at the model's four-decimal reporting resolution.

Figures & tables

Contrast	Changed	Fixed
Request set	One vs. three questions	State, target proposition
Joint request	One vs. three emitted answers	State, target proposition
State addition	Append grade-0 D	Target question, A/B/C passage text
Document isolation	Remove other passages	Target question, target passage text, label
Eligibility	A/B vs. A/B/C criteria	Full A/B/C state
Abstention	Add N=None criterion	State, passage labels

Table 1. What each paired comparison changes. State includes the query and displayed passages.

Decision model	Year	Request flips	State flips	Singleton accuracy	Action change
Laya general	2019	0.6 [0.0, 1.4]	10.8 [6.0, 16.5]	66.4 [63.5, 68.9]	0.0 †
Laya general	2020	0.8 [0.0, 1.9]	8.0 [4.4, 11.9]	63.9 [60.4, 67.3]	2.5 [0.6, 5.0]
Laya typed	2019	0.0 †	13.1 [8.3, 18.5]	66.1 [61.8, 70.1]	1.7 [0.0, 4.3]
Laya typed	2020	0.0 †	15.1 [10.3, 20.1]	64.8 [61.4, 67.7]	1.9 [0.0, 4.4]
Kev-0.8B	2019	0.0 †	9.4 [5.7, 13.1]	51.0 [45.6, 56.7]	0.0 †
Kev-0.8B	2020	0.0 †	9.6 [6.7, 12.8]	53.2 [47.8, 58.7]	0.0 †

Table 2. Native trained decision engines on the frozen retrieval states. Outcomes are percentages; brackets give 95% query-bootstrap intervals where shown. Request and state columns measure strict binary flips; accuracy is single-question (singleton) Noul accuracy; action change is the percentage of states whose selected passage or abstention status changes when questions are batched at threshold 0.5.

\dagger

: no observed events; the degenerate bootstrap interval is omitted. The one-sided 95% upper bound for a query having any corresponding event is 7.4%/5.5% (39/53 queries), a different estimand from the table’s event rates.

Figure 1. Changing the requested questions versus adding a passage to the state. Bars show strict-flip percentages; whiskers are 95% query-bootstrap intervals. Dots at zero mark zero observed flips, whose query-level upper bounds remain positive. Years and interventions are kept separate. Grouped bar chart of strict-flip percentages for the request-set and state-addition interventions across seven checkpoints and two years, with bootstrap whiskers; Nimble and Laya-general show nonzero request-set flips.

Figure 2. Empirical distributions of request-induced absolute probability movement over 828 pairs per Laya checkpoint (both years). BF16 is the native CUDA default; FP32 is a separate diagnostic run. The horizontal axis is linear near zero and logarithmic above

10^{-4}

. Dots show the mass at exactly zero: 17.5%/21.6% in BF16 and 100%/99.4% in FP32 for typed/general, where Laya-general’s FP32 maximum movement is 0.0001. Probabilities are native four-decimal outputs. Distributions of absolute request-induced probability movement over 828 paired judgments per Laya checkpoint, under the BF16 CUDA default and the FP32 diagnostic; nearly all FP32 mass sits exactly at zero.

Model	Year	Binary Choice	Score	Criteria rev.	Eligibility
Laya general	2019	16.0	46.4	5.1	9.4
Laya general	2020	10.1	45.7	6.3	8.2
Laya typed	2019	15.7	59.5	3.4	4.3
Laya typed	2020	23.7	57.0	5.0	5.7
Kev-0.8B	2019	15.4	37.9	6.0	0.0
Kev-0.8B	2020	21.2	35.4	6.9	0.0

Table 3. Native interface sensitivity (strict flips, %). Question order/keys keep state fixed; binary Choice and Score are compared with Noul. The last two columns test conditional A/B preference under criteria reversal and A/B-to-A/B/C expansion, respectively, keeping the full A/B/C state fixed. The first four outcome columns are percentages of judgments; the last two are percentages of states. Full query intervals and tie counts are in the artifact.

Model	Year	Accuracy	Brier	Overall	Zero	One	Two
Laya general	2019	66.4	0.216	54.7	92.3	28.2	43.6
Laya general	2020	63.9	0.236	44.7	83.0	20.8	30.2
Laya typed	2019	66.1	0.218	59.8	82.1	41.0	56.4
Laya typed	2020	64.8	0.222	52.2	83.0	26.4	47.2
Kev-0.8B	2019	51.0	0.285	52.1	7.7	61.5	87.2
Kev-0.8B	2020	53.2	0.278	53.5	17.0	58.5	84.9

Table 4. Task quality with full shared evidence: native singleton Noul above the rule, ordinary-LLM controls (explicit-output realization, independent-compiler arm) below it. Accuracy and correct actions are percentages; Brier is unscaled. Actions use threshold 0.5. Overall gives the correct-action rate across all states; Zero/One/Two give rates for states with that many judged answer-bearing passages. All-No obtains 66.7% accuracy; always abstaining obtains 33.3% correct actions. Calibration differs: Kev uses shipped temperatures, while Nimble uses unfitted

T=1

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 644 papers

JevOut: Natural Context Can Flip Decision Models
Sep 24, 2026Zixiang Xu, Zirui Song, Chiyu Zhang +4Adversarial Attacks on LLMs LLM Decision-Making
University of Southern California
Evaluating and Benchmarking the System One Model Jev
Sep 29, 2026Tobias Deußer, Rajkumar Ramamurthy, Lorenz Sparrenberg +1Multilingual Language Model Evaluation LLM Evaluation
University of Bonn, Bonn, Germany · Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany · Fraunhofer IAIS, Sankt Augustin, Germany
What Can't Jev Decide? Probing the Limits of One-Pass Decision Models
Oct 18, 2026Wang Yang, Xinpeng Li, Yanyan Zhang +8LLM Decision-Making LLM Reliability
Case Western Reserve University · NII LLMC (Japan) · Georgia Institute of Technology +1

Sep 24, 2026J/K move · Enter open · S save

JevOut: Natural Context Can Flip Decision Models

Zixiang Xu, Zirui Song, Chiyu Zhang, Xiuying Chen+3

University of Southern California

Adversarial Attacks on LLMs LLM Decision-Making Language Model Robustness

Open paper AskPDF arXiv

An ordinary-looking background detail can turn a correct model decision into a confident mistake. We demonstrate this fragility in four decision systems, including Jev, across seven datasets covering knowledge, reasoning, and tool routing. Within 64 accepted target evaluations per decision, we uncover short context additions that redirect 61.4%-73.2% of each system's initially correct decisions toward a wrong option fixed in advance. The additions supply background or procedural information rather than explicit answer-selection instructions, leaving the original question and choices intact. We construct them through probability-guided context optimization, which uses shifts in the option distribution to refine surrounding text under naturalness and answer-preservation constraints. Redirection affects initially confident decisions, often produces high-confidence wrong choices, and transfers across models. In blinded human evaluation, 91.6% of 250 sampled successful contexts are judged natural, answer-preserving, and free of decisive answer-changing evidence by a majority of three independent annotators. These findings expose a weakness in current decision models: context that looks entirely compatible with an input can redirect the choices that agents, routers, and evaluators rely on.

Abstract

LLM Auditing

Explore similar work

JevOut: Natural Context Can Flip Decision Models

Evaluating and Benchmarking the System One Model Jev

What Can't Jev Decide? Probing the Limits of One-Pass Decision Models