cs.CLOct 26, 2026
SaveUnderstanding Jev-Style System One Models: Does Structure Reduce Selection Bias?
Organizations: Foundation Model Department, Tencent, China
Abstract
Selection bias in large language models (LLMs) can persist despite instructions to ignore presentation order or prompts that elicit reasoning. Recent efforts to reproduce Jev-style System One behavior have produced a growing family of models that predict probabilities directly over supplied candidates. Yet systematic comparisons of selection bias across Jev versions, Jev reproductions, and generative LLMs remain limited in multi-option tasks. This leaves a central question: does Jev-style decision structure reduce selection bias? We address this question with a simple test that shuffles options while keeping question and candidate content fixed. Across two Jev versions, twelve Jev reproductions, and six generative LLMs, we trace individual answers through eight presentations and compare them with fixed-order repetition. On our primary collection of 1,700 questions from two benchmarks, Jev's accuracy varies by less than two percentage points, yet its cumulative failure set expands by 27--29% relative to the original order, with first failures continuing into the final presentation. This loss of initial successes is substantially larger than under fixed-order repetition, even for highly repeatable Jev reproductions. Nor does Jev show a general advantage over the LLM baselines in preserving correct choices. Aggregation offers limited protection: voting yields modest gains for Jev, while even unanimous answers are frequently wrong. These results expose a gap between repeatable decisions and decisions that remain correct across equivalent presentations, showing why stable accuracy alone cannot establish resistance to selection bias.
Figures & tables
Figure 1: Stable Jev accuracy hides an expanding failure set. Two Jev versions and six generative LLMs on 1,700 questions: (a) accuracy, (b) cumulative same-candidate agreement under eight original-order requests, (c) cumulative failed questions, and (d) retention of initial successes. Panels (a,c,d) use the original order (O), reversal (R), and six random permutations (S1–S6). Each failed question counts once; all requests retain the full menu.
Figure 2: High repeatability does not ensure order robustness. Twelve Jev reproductions and both Jev versions on the same 1,700 questions. Panels follow Figure 1 . The reproductions achieve 97.7–100% same-candidate agreement under original-order repetition but lose 23.2–64.2% of initial successes under reordering.
Figure 3: Accuracy hides changes in individual choices. Original-to-reversed transitions on paired questions: (a) correct to wrong, (b) wrong to correct, (c) wrong to a different wrong candidate, and (d) all changed choices. Opposing transitions can cancel in accuracy; changes between wrong candidates leave it unchanged.
Figure 4: Accuracy also hides turnover in Jev reproductions. Reversal transitions on each model’s fixed identity set, using the categories in Figure 3 . AutoTrust-JEV-9B loses 167 correct answers and recovers 170, while another 377 choices move between wrong candidates.
Figure 5: Jev’s fixed-order agreement includes repeated errors. Eight original-order requests: (a) mean accuracy, (b) agreement across all eight, (c) loss of initial successes, and (d) questions with the same wrong answer throughout. Panels use the same paired set as Figure 1 (b). Error bars show pointwise 95% bootstrap intervals.
Figure 6: High fixed-order agreement also repeats errors. Eight original-order requests, with metrics as in Figure 5 . Identity statistics use each model’s fixed selection set; correctness metrics use the full collection. Error bars show pointwise 95% bootstrap intervals.
Figure 7: Reordering exposes more Jev instability at equal call counts. Eight original-order requests (hatched) versus eight reordered presentations (solid): (a) changed choices, (b) loss of initial successes, (c) mean accuracy, and (d) plurality gain over mean accuracy. Error bars show pointwise 95% paired bootstrap intervals.
Figure 8: Reordering reveals failures rarely seen under repetition. Eight calls per condition, with metrics as in Figure 7 . Identity sets are fixed jointly across conditions; correctness metrics use the full collection. Error bars show pointwise 95% paired bootstrap intervals.
Figure 9: Additional wrong candidates drive most of Jev’s wrong-pair growth. (a) Original errors , (b) newly failed questions , (c) extra wrong candidates beyond the first per affected question , and (d) their sum . Counts are distinct question–wrong-option pairs and can exceed the number of questions. Every request retains the full menu.
Figure 10: Additional distractors dominate wrong-pair growth in Jev reproductions. (a) Original wrong pairs , (b) newly observed questions with an actual wrong candidate , (c) extra wrong candidates , and (d) all distinct wrong pairs . Extra candidates account for 70.8–89.5% of growth.
Figure 11: Jev’s relative stability depends on the task and baseline. By benchmark: (a) eight-presentation mean accuracy, (b) changed choices, (c) loss of initial successes, and (d) extra wrong pairs beyond the first per affected question, per 100 questions. Error bars show pointwise 95% bootstrap intervals (Section 2.6 ).
Figure 12: Lower turnover need not mean better retention. Benchmark profiles of Jev reproductions, with metrics as in Figure 11 . AutoTrust-JEV-27B-v1.1 changes fewer choices than either Jev version on every benchmark, yet loses more initial successes than Jev 1.14 on LEXam. Error bars show pointwise 95% bootstrap intervals.
Figure 13: Jev changes more choices on harder questions. SuperGPQA difficulty groups: (a) mean accuracy, (b) changed choices, (c) loss of initial successes, and (d) changed choices restricted to ten-option questions. Error bars show pointwise 95% bootstrap intervals. Panel (d) holds menu size fixed; question content still differs.
Figure 14: Hard questions expose greater order sensitivity in Jev reproductions. SuperGPQA profiles, as in Figure 13 . Every reproduction changes more choices on hard than easy questions, including within ten-option menus. Error bars show pointwise 95% bootstrap intervals; content and subject composition differ.
Figure 15: Harder questions amplify Jev’s loss of correct answers. Retention across eight presentations for ten-option SuperGPQA questions: easy, middle, hard, and all difficulties. Each curve starts at 100% and retains a question only while every observed answer remains correct. Panels share a scale; question content differs across groups.
Figure 16: Correct choices by Jev reproductions are more fragile on hard questions. Ten-option SuperGPQA retention by difficulty. AutoTrust-JEV-27B-v1.1 loses 18.2% of initial successes on easy questions and 44.9% on hard questions. Panels share a scale, whose range differs from Figure 15 .
Figure 17: Subject profiles separate correctness changes from wrong-answer switches. By SuperGPQA discipline: (a) mean accuracy, (b) changed choices, (c) both correct and wrong answers, and (d) always wrong with changing candidates. Rates in (c,d) sum to (b). Error bars show pointwise 95% bootstrap intervals. Other disciplines pools five groups with fewer than 30 questions each; subject and difficulty vary together.
Figure 18: Jev reproductions switch between wrong answers within subjects. SuperGPQA discipline profiles, using the decomposition in Figure 17 . Correctness changes and always-wrong switches partition each model’s changed-choice rate. Error bars show pointwise 95% bootstrap intervals; subject and difficulty vary together.
Figure 19: Larger LEXam menus accompany more varied wrong choices. Option-count groups contain 125 questions each; metrics follow Figure 11 . Lines connect model estimates, with pointwise 95% bootstrap intervals. Groups contain different questions, so the comparison does not isolate an effect of option count.
Figure 20: LEXam exposes both varied choices and menu limitations. Profiles include 4-, 8-, 16-, and 32-option questions, using the metrics in Figure 19 . Error bars show pointwise 95% bootstrap intervals. Groups contain different questions.
Figure 21: Jev loses initial successes in both option-count groups. Retention through each presentation for (a) 437 questions with fewer than eight options and (b) 1,263 with at least eight. Each curve is normalized by the model’s initially correct answers in its group.
Figure 22: Jev reproductions lose correct choices even with at least eight options. Groups and metric follow Figure 21 .
Figure 23: Jev’s instability has no uniform trend across menu sizes. All observed option counts: (a) mean accuracy, (b) changed choices, (c) loss conditional on initial success, and (d) newly failed questions after O per 100 questions, counted once. Error bars show pointwise 95% bootstrap intervals. Benchmark and difficulty composition differ across groups.
Figure 24: Jev reproductions fail across the full range of menu sizes. Four to 32 options; metrics follow Figure 23 . Error bars show pointwise 95% bootstrap intervals. Benchmark and question content differ across groups.
Figure 25: Voting offers limited Jev correction; unanimity still includes errors. Across eight orders: (a) mean accuracy, (b) plurality gain in percentage points, (c) unanimous coverage, and (d) accuracy among unanimous answers. Error bars show pointwise 95% paired source-group bootstrap intervals, stratified by benchmark. Plurality ties use uniform tie-breaking in expectation.
Figure 26: Voting and unanimity provide limited safeguards for Jev reproductions. Metrics follow Figure 25 , with accuracy and coverage over all 1,700 questions. Unanimous accuracy ranges from 30.2% to 66.3% at differing coverage. Error bars show pointwise 95% paired bootstrap intervals; plurality ties use uniform tie-breaking in expectation.
Figure 27: Jev’s higher MMLU-Pro accuracy still leaves correct answers vulnerable. (a) Accuracy on all 300 questions; (b) cumulative failed questions; (c) loss of initial successes after eight original-order or reordered requests; (d) coverage and accuracy of answers unanimous across reordered presentations. Panels (b–d) use each model’s fixed selection set defined in Section 6 .
Figure 28: Perfect repetition does not protect Jev reproductions from reordering. MMLU-Pro results for twelve Jev reproductions with both Jev versions as references, using the panels in Figure 27 . All panels use the complete 300-question sample.
Figure 29: Four candidates still leave correct Jev choices vulnerable. GPQA Diamond: (a) accuracy, (b) cumulative failed questions, (c) initial-success loss under repetition and reordering, and (d) distinct wrong question–candidate pairs. All panels use the full 198 questions.
Figure 30: Perfect repetition coexists with lost successes on four-option questions. GPQA Diamond results for Jev reproductions, with both Jev versions as references. Panels and denominators follow Figure 29 .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Contribution | Question answered | Examples and boundaries |
|---|---|---|
| Model construction | How is a decision model obtained, and what does its inference compute? | Jeff, SII, LDM, AutoTrust; CLM/FunDM model recipes. Jev can be a target behavior, teacher, or baseline. |
| Downstream workflow | What does the model control in task execution or in learning another agent? | Jev-Mem, Ultrafast, LAVOIR interaction; run selectors and frozen-Jev RL assistance. The model need not be proprietary Jev. |
| Evaluation and analysis | Which capabilities, system outcomes, or usage patterns are supported by evidence? | JEVal benchmark, agent integration studies, coherence tests, ecosystem surveys. Evaluation is distinct from workflow invention. |
Table 1: Three levels of contribution. A work may contribute at more than one level.
Figure 31: Four readouts for candidate probabilities. (a) Vocabulary rows, (b) complete suffixes, (c) joint-context option states, and (d) separately encoded states and actions. Suffix token boundaries are illustrative. Across the mechanism figures, green denotes model computation, gold supervision or external feedback, slate operations, and gray data or memory. Labels specify training status. Sources and objectives appear in Appendix C.3 .
Figure 32: Candidate reuse and interaction are separate choices. (a) Independent scoring, (b) set-dependent request refinement, and (c) request-conditioned candidate blocks. Arrows show dependencies; dashed outlines mark representations reusable with fixed inputs and weights. Odds statements fix request text, candidate text, and calibration.
Figure 33: Four sources of teacher targets. (a) Averaged endpoint probabilities, (b) sampled-answer votes, (c) full-depth predictions, and (d) AutoTrust service outputs, excluding its other target sources. Vectors in (a,b) are illustrative; (b) assumes four parseable samples. Dotted gold arrows carry fixed targets; dashed arrows mark updated components. Student inference omits teacher computation.
Figure 34: Three paths for learning signals. (a) Direct differentiation, (b) output-space score-function credit supplementing soft cross-entropy, and (c) direct answer-readout gradients with second-stage rationale-policy credit. Here is a baseline, a fixed target, and stop-gradient. Schematics omit centering, normalization, and auxiliary regularizers; they compare mechanisms, not measured estimator advantages.
Figure 35: Direct outputs permit different execution paths. (a) Kev cache reuse, (b) Clef joint schema readout, (c) LDM routing to untuned-base-model reasoning, and (d) StartLux fusion with the same weights on both paths. and are the two largest readout probabilities. Arrows show inference flow, without parameter updates.
| Model / recipe | Target and update | Readout and inference | Public evidence |
|---|---|---|---|
| Inner Jev / Jeff ( Geuter et al., 2026 ) | Frozen baseline; adapted model uses mixed-label CE, LoRA and residual head; temperature fitting | Up to 16 letter labels; one forward per question; released server runs questions serially | Paper, weights, serving code and OpenDecisions |
| LLM-as-Jev ( Li and Wagle, 2026 ) | Frozen suffix scoring; optional candidate-tree NLL with KL anchors on human labels | Original architecture; short multi-token suffix scoring; optional inherited thinking | Paper describing scoring and optional adaptation |
| CLM | Bidirectional InfoNCE on matched state–action pairs; frozen backbone, learned projections | Independently encoded state and candidates; softmax of scaled cosine scores; reusable embeddings | Technical blog, heads, inference and fine-tuning code |
| SII-Decision ( Duan et al., 2026 ) | Cached reasoning-endpoint distributions; CE, reference KL and ordinal loss | 255 native single-token labels; one prefill; no rationale; | Paper, training/inference code, four models and target datasets |
| LDM ( Chen et al., 2026b ) | Hard-label variants; SPRD matches concentrated teacher votes; SPRD-C also checks label agreement | Direct readout; optional untuned-base reasoning below validation margin’s 30th percentile | Paper; repository states “Code coming soon” at the survey cutoff |
| FunDM ( Panchal, 2026 ) | Teacher-to-student KL; train intermediate head, freeze backbone | Early-layer decision head; remaining backbone layers can be skipped | Paper and implementation |
Table 2: Model construction, readout, and public artifacts. Rows distinguish training stages and model variants.
Figure 36: Decisions intervene at different workflow stages. (a) Online action selection, (b) completed-run selection, and (c) repeated clarification, handoff, or decision. Arrows show runtime flow. Panel (b) scores completed runs rather than controlling their individual actions; CLM includes mean score aggregation. Colors follow Figure 31 ; sources appear in the corresponding subsections.
Figure 37: Jev-Mem controls memory writing and retrieval ( Jiang et al., 2026 ) . (a) Writing preserves observations and judges relations. (b) Retrieval iteratively acquires evidence before answer synthesis. Four graph views share memory nodes. Arrows span multiple operations and calls; colors follow Figure 31 .
| Project / primary source | Documented mechanism | Scope and distinctions |
|---|---|---|
| Frozen readout and execution recipes | ||
| SemIf / former OpenJev | Restricted answer-label probabilities; workload-specific temperature fitting | Runtime code; explicitly an interface pattern, not TypeSafe’s training recipe |
| open-alternative-jev | Multiple packed readout positions; exact-logit and top- logprob backends | Runtime variants differ; some use constrained one-token generation |
| Cygnet | Frozen Gemma-4-12B-it; label probabilities, fitted temperature and large-menu grouping | One output token is emitted; no weight adaptation |
| deck31b | Frozen Gemma-4-31B-it for candidate selection | Uses pretrained weights without decision-specific adaptation |
| reflex | Frozen Qwen3.5-4B with averaging over two option orders | Tested adapters degraded out-of-distribution judgments; the default model remains frozen |
Table 3: Additional implementations and documented mechanisms.
More in this topic
Language Model Bias Evaluation
Language model bias evaluation tests how models’ outputs, decisions, and representations vary across social groups, cultures, languages, and contexts.
See all 491 papers