Cross-Primitive Consistency in Jev

Xu Liu; Guikun Chen; Kanzhi Wu; Wenguan Wang

cs.CVOct 28, 2026

Cross-Primitive Consistency in Jev

Authors: Xu Liu, Guikun Chen, Kanzhi Wu, Wenguan Wang

Organizations: Zhejiang University · vivo

Abstract

Jev enables efficient decision-making with large language models (LLMs) through typed querying primitives, allowing the same semantic state to be expressed in different formats, such as binary and multiple-choice queries. However, the consistency of decisions across primitives has not been systematically evaluated. This work introduces PrimaCons, a four-axis benchmark for assessing cross-primitive consistency while preserving the semantics of the query. Evaluations across three Jev sizes reveal consistency violations across all four axes. Our analysis shows that these violations follow systematic patterns and that cross-primitive discrepancies predict decision failure with a ROC-AUC of 0.91. To mitigate these inconsistencies, we propose the Cross-Primitive Residual Calibrator (CPRC), which attaches a lightweight residual network to a frozen backbone and jointly optimizes predictive accuracy and cross-primitive consistency. Under a leave-one-domain-out protocol, CPRC reduces cross-primitive inconsistency by 58.57% and improves accuracy by +2.97%., demonstrating its effectiveness in both in-domain and out-of-domain settings. Our findings establish cross-primitive consistency as a key dimension for evaluating Jev and provide a practical approach to improving the reliability of its decisions. Code and benchmark is available at https://github.com/67L1/PrimaCons-CRPR.

Figures & tables

Figure 1: An example of cross-primitive inconsistency.

Model	$G_{n^{+},n^{-}}$ $\downarrow$	$G_{n^{+},c}$ $\downarrow$	$G_{n^{-},c}$ $\downarrow$
JevK5-2B	0.1858	0.2471	0.1127
JevK5-4B	0.1360	0.1132	0.0689
JevK5-9B	0.0892	0.0972	0.0686

Table 1: Axis I consistency.

G_{n^{+},n^{-}}

G_{n^{+},c}

, and

G_{n^{-},c}

are the probability gaps over the three pairwise combinations of Positive Noul (

n^{+}

), Reversed Noul (

n^{-}

), and Choice (

c

Figure 3: Axis I polarity complementarity and error signal. The x-axis is the Positive Noul and the y-axis is the Reversed Noul. The color bar denotes the residual. The inset shows the ROC curve of predicting decision error from the residual (AUC $=0.91$ ).

Model	$D_{C}$ $\downarrow$	$D_{M}$ $\downarrow$	$D_{F}$ $\downarrow$
JevK5-2B	0.2945	0.4837	0.4890
JevK5-4B	0.0870	0.5329	0.6323
JevK5-9B	0.1300	0.3650	0.5107

Table 2: Axis II consistency.

D_{C}

D_{M}

, and

D_{F}

denote

TV

between Choice and Noul at the coarse (

C

), medium (

M

), and fine (

F

) granularity levels, respectively.

Figure 4: Axis II label hierarchy consistency residual. (a) shows $e^{CM}$ (x-axis) and $e^{CF}$ (y-axis). (b) displays the residual between the Medium probability and the sum of its Fine probabilities, with the black dots denoting the average residual values (all below $0$ ).

Model	$G_{A}$ $\downarrow$	$K$ (%) $\downarrow$	Acc. Gap (%) $\downarrow$
JevK5-2B	0.1928	36.54	9.66
JevK5-4B	0.2614	26.69	10.83
JevK5-9B	0.2130	17.18	11.72

Table 3: Axis III consistency .

G_{A}

is the abstention probability gap,

K

is the Choice/Noul hard-label conflict rate, and Acc. Gap measures the difference in accuracy between the two primitives.

Figure 5: Axis III abstention consistency. The scatter plot compares Choice $a^{C}$ (x-axis) and Noul abstention probability $a^{N}$ (y-axis). The dashed lines denote the $0.5$ decision boundaries, and the color gradient represents the absolute probability gap $|a^{C}-a^{N}|$ .

Model	$\overline{TV}(p^{\mathrm{r}},p^{\mathrm{d}})$ $\downarrow$	$\overline{TV}(p^{\mathrm{r}},p^{\mathrm{n}})$ $\downarrow$	$\overline{TV}(p^{\mathrm{d}},p^{\mathrm{n}})$ $\downarrow$
JevK5-2B	0.0917	0.4386	0.4716
JevK5-4B	0.1013	0.2405	0.2694
JevK5-9B	0.0439	0.2227	0.2450

Table 4: Axis IV consistency.

\overline{TV}

measures the Total Variation among the raw Choice (

p^{\mathrm{r}}

), description-augmented Choice (

p^{\mathrm{d}}

), and canonical Noul (

p^{\mathrm{n}}

) primitives.

Figure 6: Axis IV label-description consistency distribution. The four panels correspond to the AG News text categories. The cross symbols ( $\times$ ) denote the mean coordinates of the distributions.

Model	Axis I				Axis II				Axis III
Model	$G_{n^{+},n^{-}}\!\!\downarrow$	$G_{n^{+},c}\!\downarrow$	$G_{n^{-},c}\!\downarrow$	Acc	$D_{C}\downarrow$	$D_{M}\downarrow$	$D_{F}\downarrow$	Acc	$G_{A}\downarrow$	$K$ (%) $\downarrow$	$AG\downarrow$	Acc
JevK5-2B	0.1832	0.2442	0.1106	65.03	0.2923	0.4806	0.4867	77.67	0.1884	36.15	8.77	77.81
JevK5-2B ( Cprc )	0.0869	0.0962	0.0643	79.00	0.0666	0.1296	0.1573	98.48	0.2260	18.37	6.92	87.96
JevK5-4B	0.1361	0.1132	0.0688	71.77	0.0870	0.5330	0.6325	95.20	0.2619	26.74	11.03	85.63
JevK5-4B ( Cprc )	0.0577	0.0619	0.0511	82.80	0.0842	0.2815	0.3545	99.65	0.2018	14.84	9.90	93.58
JevK5-9B	0.0861	0.0931	0.0654	71.22	0.1292	0.3618	0.5085	93.77	0.2103	17.27	11.51	91.79

Table 5: In-Domain consistency and accuracy.

AG

denotes the Accuracy Gap (%).

Model	Consistency ( $\overline{TV}\downarrow$ )
Model	$\overline{TV}(p^{\mathrm{r}},p^{\mathrm{d}})$	$\overline{TV}(p^{\mathrm{r}},p^{\mathrm{n}})$	$\overline{TV}(p^{\mathrm{d}},p^{\mathrm{n}})$
JevK5-2B	0.0825	0.4391	0.4736
JevK5-2B (Cprc)	0.0988	0.1819	0.2369
JevK5-4B	0.0907	0.2407	0.2696
JevK5-4B (Cprc)	0.0663	0.1719	0.2083
JevK5-9B	0.0447	0.2251	0.2477

Table 6: Out-of-Domain consistency.

Model	Raw Labels	Desc Labels
JevK5-2B	84.90	86.80
JevK5-2B (Cprc)	87.42	87.90
JevK5-4B	84.08	86.97
JevK5-4B (Cprc)	84.12	87.05
JevK5-9B	88.02	89.03
JevK5-9B (Cprc)	89.75	90.07

Table 7: Out-of-Domain accuracy.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks
Oct 8, 2026Xing Li, Qingcheng Chang, Jinzhong Ning +5LLM Evaluation Multiple-Choice Question Answering
Dalian Maritime University · Dalian University of Technology
General Decision Models: Benchmarking and Insights Beyond Jev
Oct 2, 2026Feiyu Duan, Jiayu Lin, Jia Wang +6LLM Decision-Making Uncertainty Calibration
Shanghai Innovation Institute · Fudan University · Tongji University +2
What Can't Jev Decide? Probing the Limits of One-Pass Decision Models
Oct 18, 2026Wang Yang, Xinpeng Li, Yanyan Zhang +8LLM Decision-Making LLM Reliability
Case Western Reserve University · NII LLMC (Japan) · Georgia Institute of Technology +1

Oct 8, 2026J/K move · Enter open · S save

Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks

Xing Li, Qingcheng Chang, Jinzhong Ning, Changfeng Xu+4

Dalian Maritime University · Dalian University of Technology

LLM Evaluation Multiple-Choice Question Answering

Open paper AskPDF arXiv

Jev is a "System One" model that returns a choice among given options instead of generating text. We study how such a specialized decision model compares with general-purpose large language models (LLMs). We evaluate Jev on 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding, and compare it with 19 LLMs in three tiers: frontier, representative, and small. Jev is competitive with frontier LLMs on knowledge and commonsense benchmarks and obtains the best score on MMLU-Redux and ARC-Challenge. Outside mathematics, it also outperforms most representative LLMs and all small LLMs. However, it falls behind on mathematical word problems: on MathQA, it is 17.7 points below the frontier median and scores lower than all 19 LLMs. These results indicate that a specialized decision model can match general-purpose LLMs on decisions that rely mainly on knowledge, but not on decisions that require multi-step calculation.

Abstract

LLM Evaluation

Explore similar work

Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks

General Decision Models: Benchmarking and Insights Beyond Jev

What Can't Jev Decide? Probing the Limits of One-Pass Decision Models