FastJEV: Understanding Redundancy for Compact JEV Inference

Jie Ma; Jie Gao; Yihang Liu; Zhike Qiu; Junle Li; Chongyi Zhuang; Jiayi Ji; Xiaoshuai Sun

cs.CVOct 8, 2026

FastJEV: Understanding Redundancy for Compact JEV Inference

Authors: Jie Ma, Jie Gao, Yihang Liu, Zhike Qiu, Junle Li, Chongyi Zhuang, Jiayi Ji, Xiaoshuai Sun

Organizations: Xiamen University

Abstract

JEV models make multimodal decisions by directly scoring candidates. Although the common context is encoded once, candidate evaluation can still repeat matching token histories, duplicate inference states, and execute the full backbone. In this paper, we study these sources of redundancy and present FastJEV for compact candidate evaluation. We jointly organize history reuse and state storage, since sharing computation requires preserving states for later branches. We first introduce shared context anchoring to reuse recurrent initial states and omit unused final recurrent caches. We extend this reuse through candidate prefix sharing, retaining the intermediate states needed by subsequent branches. To further reduce the depth of these paths, we apply decision guided pruning based on relative score changes measured on a small unlabeled set. Our method retains full context encoding and all candidates without additional training. We evaluate FastJEV across three OmniJev model sizes on five public benchmarks and reconstructed LIBERO-10 offline questions. At the selected pruning budgets, the complete method reduces candidate depth by 43.75% to 45.83%, while retaining 93.66% to 97.52% of the original task scores on average across the six evaluation sets. Through controlled experiments, we show how candidate overlap and branching structure affect the execution cost of history reuse. In our implementation, candidate prefix sharing can reduce repeated computation while increasing latency. These findings motivate designing sharing granularity and execution schedules together for efficient JEV inference.

Figures & tables

Figure 1 : Overview of FastJEV , illustrated with an ERQA example ( Team, 2025 ) on OmniJev ( Xu et al., 2026 ) . (a) Shared context anchoring reuses the encoded context and shares recurrent initial state storage. (b) Candidate prefix sharing merges matching token histories and retains continuation caches at forks. (c) Decision guided pruning bypasses selected candidate layers using a fixed mask determined through offline sensitivity assessment. (d) The original scoring procedure produces candidate logits in their original order and an abstention logit. Full context encoding and all candidate tokens are retained. Phrase boxes group consecutive tokens, and layer positions are schematic.

Figure 2 : Decision sensitivity across layers in OmniJev. Each point is the mean relative score perturbation

J_{\ell}

over 32 multiple-choice calibration inputs when a single candidate block is skipped. The six lowest-scoring layers are highlighted for the default pruning budget. Layer indices start at one, and the vertical axis uses a logarithmic scale. Context encoding retains all layers.

0.8B (24 layers)
			Accuracy (%)
Method	Pruned layers	Pruning (%)	AI2D	SQA	POPE	BLINK	ERQA	LIBERO	Mean ret. (%)
OmniJev	-	0.00	62.27	69.01	84.76	43.98	28.75	55.45	100.00
FastJEV	6	25.00	62.37	69.31	87.79	43.40	29.50	55.65	100.97
FastJEV	8	33.33	62.37	68.86	88.17	42.92	31.00	56.38	101.85
FastJEV	11	45.83	58.19	66.63	88.50	41.45	28.00	54.92	97.52

Table 1: Decision quality of OmniJev and FastJEV, grouped by model scale. All FastJEV rows use T+S+P. Pruning applies to candidate evaluation, while context encoding uses all layers. Dataset columns report accuracy (%). Mean retention averages six accuracy ratios to the matched Original. Bold pruning and retention entries mark the largest tested budget with mean retention of at least 90%. SQA denotes ScienceQA-IMG, and LIBERO denotes reconstructed LIBERO-10 offline QA.

	AI2D			LIBERO-10 QA
Mode	Acc. (%) $\uparrow$	ms $\downarrow$	MiB $\downarrow$	Acc. (%) $\uparrow$	ms $\downarrow$	MiB $\downarrow$
C	84.23	132.51	9454.1	64.96	324.58	10381.8
T	84.20	175.74	9525.0	64.89	838.51	9689.8
S	84.23	130.82	9261.6	64.96	319.20	9508.0
P	84.36	125.42	9453.0	63.90	296.14	10356.8
T+S	84.20	174.98	9419.0	64.89	833.27	9477.9

Table 2: Factorial comparison with 4B using a common recurrent kernel. C copies initial states, retains final caches, and executes full candidate depth. P skips six candidate blocks when enabled. C differs from the released Original in Table 1 . LIBERO latency is per frame. Memory denotes the total allocated peak. Bold marks the best value in each column.

Figure 3 : Structural compactness and execution cost with 4B. (a) Logical candidate token reduction across the six complete evaluation sets. SQA-IMG denotes ScienceQA-IMG, and LIBERO QA is our reconstructed offline set. (b) Observed live state storage for four inputs, with T and six-layer P fixed. Copy / Keep and Share / Drop follow Table 4 . (c) Full-request latency and total peak GPU memory of T+S+P relative to S+P across all 96 artificial workload conditions. Colors indicate candidate length, and marker shapes indicate branching structure. Ratios below one indicate lower cost. Artificial workloads evaluate execution cost only.

Table 6

Workload			Latency (ms) $\downarrow$			Peak memory (MiB) $\downarrow$
Candidates	Tokens	Shared (%)	S+P	T+S+P	Speedup $\uparrow$	S+P	T+S+P
2	8	0	110.54	140.13	0.789	9436.6	9549.0
8	32	50	129.95	153.10	0.849	9646.7	9449.2
64	128	0	1511.67	1595.58	0.947	24742.3	12466.2
64	128	75	1511.75	552.62	2.736	24615.1	11260.5

Table 5: Selected artificial 4B workloads with fixed source images and questions. Sharing is the fraction of candidate content held in common, excluding mandatory boundary tokens. Each condition uses four source inputs and five timing rounds. Speedup is S+P latency divided by T+S+P latency. Bold numbers mark the lower latency or peak memory in each pair. Artificial candidates are used to analyze execution cost.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

0.8B
	Accuracy (%) $\uparrow$		Latency (ms) $\downarrow$		Peak memory (MiB) $\downarrow$
Dataset	Original	FastJEV	Original	FastJEV	Original	FastJEV
AI2D	62.27	62.37	98.09	105.54	1930.9	1934.8
ScienceQA-IMG	69.01	69.31	100.17	106.60	1907.9	1890.8
POPE	84.76	87.79	91.26	75.30	1832.1	1877.8
BLINK	43.98	43.40	119.35	123.10	2020.2	2075.1

Appendix

Table 7: Complete accuracy and inference cost at the default six-block budget across three OmniJev model sizes. FastJEV uses all three components (T+S+P). Bold marks the best value in each pair at the displayed precision. Latency is per question, except for LIBERO-10 QA, which is per frame of eight questions. Peak GPU memory includes model weights and uses the maximum PyTorch allocated peak over the measured inputs. LIBERO-10 QA is our reconstructed offline set.

Figure 5 : Selected correct FastJEV answers on AI2D, ScienceQA-IMG, and POPE using OmniJev 4B. FastJEV uses shared context anchoring, candidate prefix sharing, and pruning of 14 candidate layers. Each panel places the image on the left and the question and answer comparison on the right. GT denotes the evaluation label. Bars show up to three highest probabilities from each model without renormalization. Checks and crosses indicate agreement with GT.

Figure 6 : Selected correct FastJEV answers on BLINK, ERQA, and reconstructed LIBERO-10 offline QA under the same OmniJev 4B configuration. All input views are shown, including both robot cameras. Answers display the selected candidate text. Appended choice lists and response-format instructions are omitted from displayed questions. The LIBERO ground truth uses the frozen state-derived label. Bars retain the original model probabilities and are not normalized over the displayed subset.

Calibration inputs	0.8B	2B	4B
8	$\{16,17,18,19,20,21\}$	$\{15,17,18,19,21,22\}$	$\{21,23,24,25,26,27\}$
16	$\{16,17,18,19,20,21\}$	$\{15,17,18,19,21,22\}$	$\{21,23,24,25,26,27\}$
32	$\{16,17,18,19,20,21\}$	$\{15,17,18,19,21,22\}$	$\{21,23,24,25,26,27\}$
64	$\{16,17,18,19,20,21\}$	$\{15,17,18,19,21,22\}$	$\{21,23,24,25,26,27\}$
128	$\{16,17,18,19,20,21\}$	$\{15,17,18,19,21,22\}$	$\{21,23,24,25,26,27\}$

Appendix

Table 8: Selected candidate blocks across calibration set sizes for OmniJev v1.1. Each entry lists the six blocks selected for pruning. All three calibration draws yield the same set in every entry. Layer indices start at one and are listed in ascending order. Context encoding retains all blocks.

Explore similar work

Sep 30, 2026cs.AI

More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models

Direct-decision models turn text into low-latency structured labels and scores, making them attractive for classification and automatic evaluation. Yet reliability requires more than accuracy: a model must also use the ordinal decision scale supplied by the user faithfully. We analyze JEV~1.13 and three open KEV models. Our investigation begins with ANLI, where JEV assigns 38.8% of all predictions and 51.3% of errors to Neutral despite 74.95% accuracy, nearly balanced gold labels, and balanced candidate positions. Across 36 ordinal datasets, final decisions use only 67--76% of the effective gold support, versus 87--102% on four nominal tasks. Randomizing candidate order weakens but does not remove this compression. Holding items and source scores fixed while balancing gold support and positions, we refine scales from

K=2

14

; utilization falls for every model and reaches 26--75% at

K=14

, although candidate probabilities remain broad for most models. Targeted BA-LoRA post-training raises gold-relative utilization from roughly 47% to 86% on eight supervised scales at both KEV sizes, showing that the compression is learned and modifiable rather than an immutable architectural limit. We call this ordinal scale-utilization bias: decision-stage candidate-space compression distinct from accuracy, gold imbalance, fixed position, and candidate count alone. The code and data are available at https://github.com/Glax147/jev_ordinal_scale_bia

Tianxiang Gao, Jinzhe Li, Zhiyuan Li +2

School of Artificial Intelligence, Jilin University · College of Software, Jilin University · International Center of Future Science, Jilin University +1

LLM Evaluation Ordinal Regression Language Model Bias Evaluation

Ask PDF

Sep 27, 2026cs.LG

JET: Justification Evaluation in Transformer

JET uses pretrained language and vision-language models to select among a finite set of answers without additional training. It evaluates candidate likelihoods directly and shares computation across candidates. Experiments on desktop CPUs and consumer GPUs assess decision accuracy and execution cost. Qwen3.6-35B-A3B achieves 87.48% accuracy on the full MMLU test set and 3.69 requests per second on a separately timed MMLU subset. The accuracy-throughput comparison covers model, hardware, and reasoning choices, with Jev as an external reference. Controlled execution experiments show 2.18-2.23-fold speedups from prefix reuse and cache management, and a 30.8% reduction in process time from input preparation optimizations, with unchanged outputs. Optional reasoning has a task-dependent accuracy-throughput trade-off. These results support local decision inference from existing models.

Shenghao Ding

Yet Another AI

Efficient VLM Inference LLM Inference Acceleration Efficient Language Model Inference

Ask PDF

Sep 22, 2026cs.AI

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

LLM-as-a-judge scales evaluation, but reasoning judges are slow and costly. We study JEV-as-a-Judge: evaluation with JEV, a decision-only judge that returns label probabilities instead of text, and whose confidence decides whether to accept its verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges, with blinded human adjudication, JEV comes within three points of GPT-6 wherever a verdict can be read off the text, at 0.36% of its fee and a 0.15-second median latency, and falls behind where the verdict must be derived, as in math, code, and logic. Its confidence marks this boundary. With a threshold frozen in advance, accepting confident verdicts and escalating the rest is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs at 41% of its fee, and in a pre-specified live test on two new workloads the cascade matches GPT-6's accuracy exactly. Confidence routing weakens on style-adversarial pairs and reference-free prose; we close with a simple recipe for validating thresholds locally.

Yubo Li, Yidi Miao, Ramayya Krishnan +1

Carnegie Mellon University

LLM-as-a-Judge Language Model Generation Evaluation Selective Prediction LLM Uncertainty Estimation Automated Evaluation

Ask PDF