JET: Justification Evaluation in Transformer

Shenghao Ding

cs.LGSep 27, 2026

JET: Justification Evaluation in Transformer

Authors: Shenghao Ding

Organizations: Yet Another AI

Abstract

JET uses pretrained language and vision-language models to select among a finite set of answers without additional training. It evaluates candidate likelihoods directly and shares computation across candidates. Experiments on desktop CPUs and consumer GPUs assess decision accuracy and execution cost. Qwen3.6-35B-A3B achieves 87.48% accuracy on the full MMLU test set and 3.69 requests per second on a separately timed MMLU subset. The accuracy-throughput comparison covers model, hardware, and reasoning choices, with Jev as an external reference. Controlled execution experiments show 2.18-2.23-fold speedups from prefix reuse and cache management, and a 30.8% reduction in process time from input preparation optimizations, with unchanged outputs. Optional reasoning has a task-dependent accuracy-throughput trade-off. These results support local decision inference from existing models.

Figures & tables

Panel of Figure 1 — Figure 1: End-to-end sampler-free workflow, executed locally on CPU, GPU, or a mixed placement. Candidate tokens are supplied rather than sampled. Optional text-only reasoning, omitted from the principal path above, generates a trace before scoring and freezes it into the shared prefix; that mode includes sampling. Invalid requests produce errors.

Model	Quantization	Correct	MMLU (%)	Wall (s)	Req./s
Qwen3.5-0.8B	Q8_0	4,615	32.87	3,015.29	4.66
Qwen3.5-2B	Q8_0	6,807	48.48	4,219.81	3.33
Qwen3.5-4B	Q8_0	9,334	66.47	9,832.76	1.43
Qwen3.5-9B	Q8_0	10,332	73.58	13,307.56	1.06

Table 1: Full MMLU test set on Arc A770, 14,042 requests per model, thinking disabled. Complete-process wall time includes model loading and shutdown. One run per model, with no repeat-run variance estimate.

Q8_0 model	Before (s)	After runs (s)	Mean (s)	Speedup
Qwen3.5-0.8B	242.54	108.06 / 109.10	108.58	$2.23\times$
Qwen3.5-2B	313.34	143.41 / 143.44	143.43	$2.18\times$

Table 2: Prefix-reuse and cache-management comparison on 541 requests. The speedup uses the mean of the two new complete-process times.

JET model	Quantization	Correct	MMLU (%)	Wall (s)	Req./s
Qwen3.5-0.8B	Q8_0	4,552	32.42	1,434.16	9.79
Qwen3.5-2B	Q8_0	6,818	48.55	1,443.47	9.73
Qwen3.5-4B	Q8_0	9,323	66.39	1,986.14	7.07
Qwen3.5-9B	Q8_0	10,344	73.66	2,466.26	5.69
Qwen3.5-27B	Q4_K_M	11,948	85.09	6,353.75	2.21
Qwen3.6-35B-A3B	Q4_K_M	11,601	82.62	3,370.71	4.17

Table 3: Full MMLU test set on RTX 4090 with Vulkan, 14,042 requests per model. Wall time includes model loading and shutdown; throughput is requests divided by wall time. One run per model, with no repeat-run variance estimate.

System / model	MMLU (%)	Evaluation basis
Full-MMLU evaluations
JET / Qwen3.5-27B, RTX 4090	85.09	Full MMLU test set; Vulkan; local execution
JET / Qwen3.6-35B-A3B, RTX 4090	82.62	Full MMLU test set; Vulkan; local execution
Jev 1.13	89.06	Full MMLU test set; native decision interface
Published larger-model references
Llama 3.1 70B Instruct [ 9 ]	83.6	5-shot; subject-macro accuracy

Table 4: MMLU reference comparison. Protocols differ across groups; the table provides numerical context, not a matched ranking. External results are not measurements of those models running in JET.

Panel of Figure 2 — Figure 2: Full-MMLU accuracy and throughput with reasoning disabled. Blue: Arc A770 (Vulkan); teal: RTX 4090 (Vulkan); violet: RTX 4090 (CUDA); green outlined: Qwen3.5-4B LoRA (CUDA); orange: Jev 1.13 hosted API. Each JET point combines accuracy and complete-process throughput from the same 14,042-question run. Jev uses full-MMLU accuracy and a separately reported API rate. Each metric uses a common horizontal scale.

Explore similar work

Evaluating and Benchmarking the System One Model Jev
Sep 29, 2026Tobias Deußer, Lorenz Sparrenberg, Rafet SifaMultilingual Language Model Evaluation LLM Evaluation
University of Bonn, Bonn, Germany · Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany · Fraunhofer IAIS, Sankt Augustin, Germany
Visual Jev: Accurate and Efficient Decisions from Shared Visual Context
Sep 22, 2026Guanxu Yu, Yuhang YaoVisual Question Answering Efficient Inference
Independent Research · Carnegie Mellon University
Scaling Evaluation-time Compute with Reasoning Models as Evaluators
Mar 25, 2025Seungone Kim, Ian Wu, Jinu Lee +8LLM Evaluation Language Model Generation Evaluation
CMU · UIUC2 · KAIST AI3 +3

Sep 29, 2026, cs.CLJ/K move · Enter open · S save

Evaluating and Benchmarking the System One Model Jev

Tobias Deußer, Lorenz Sparrenberg, Rafet Sifa

University of Bonn, Bonn, Germany · Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany · Fraunhofer IAIS, Sankt Augustin, Germany

Multilingual Language Model Evaluation LLM Evaluation Language Model Calibration

Open paper AskPDF arXiv

Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed options, a position on a rubric, or the probability that a statement is true, with probabilities the vendor describes as calibrated. Such models target small decisions in information access pipelines, such as routing queries, checking grounding, moderating content, or rating against a rubric. We evaluate Jev (jev-1.13.0) zero-shot on 37 datasets spanning classification, routing, natural language inference, reading comprehension, commonsense reasoning, moderation, legal clause analysis and rubric scoring, with one frozen template per dataset and full evaluation splits: 346,009 requests for under USD 10. For reference, we score Qwen3.8-27B and Gemma-4-E4B on identical requests via their exact next-token probabilities over the options. Jev reaches 95-99% accuracy on IMDB, SST-2, HellaSwag and ARC and 86.7% on Belebele across 122 languages. It beats Qwen on 27 of 37 datasets, with none of Qwen's nine leads outside the bootstrap intervals, and Gemma on all 37. All three models degrade on low-resource languages, fine-grained or noisy labels, and rubric-based quality judgments. Jev's choice probabilities are well calibrated and support selective prediction. Binary probabilities rank well but are poorly placed relative to a fixed 0.5 threshold; thresholds tuned on training data raise micro-F1 on UNFAIR-ToS from 0.50 to 0.75. Jev answers MMLU's calculation-heavy questions more accurately than other MMLU questions (94% vs. 91%), whereas both open models, and all three on C-Eval, find them harder. Rotating the options leaves Jev's accuracy unchanged and withholding the question drops it to near chance, ruling out shallow memorization but not memorized question-answer pairs. We release the code, harness and all raw responses.

Abstract

Explore similar work

Evaluating and Benchmarking the System One Model Jev

Visual Jev: Accurate and Efficient Decisions from Shared Visual Context

Scaling Evaluation-time Compute with Reasoning Models as Evaluators