Industry applications often demand low-latency classification, yet current large language model (LLM) approaches remain poorly suited for latency-critical applications. Existing prompting and constrained decoding produce verbose, multi-token outputs that require expensive token-by-token generation, while encoder-based models achieve faster inference but sacrifice task flexibility. We propose Koa-decision, a framework for low-latency atomic actions -- fast, single-step decisions such as classification, semantic endpointing, Boolean checks, and scoring -- formulated as constrained generation with single-token outputs. By introducing atomic label tokens and applying supervised fine-tuning, our method reduces classification to a deterministic one-step decoding problem. Across standard benchmarks, Koa-decision delivers competitive accuracy with consistently low and stable latency. On a production intent-routing benchmark, Koa-decision reaches 85.5% accuracy -- competitive with the strongest frontier models (Claude-4.8-Opus, Gemini-Pro-3.1) and ahead of GPT-5 and Gemini-2.5-Pro -- while answering in about half a second, several-fold faster than every frontier model (up to ~7.5x at the median) under identical serving conditions. Against the dedicated single-token system Jev/TypeSafe, Koa-decision is competitive on accuracy and faster at the median, while also handling multimodal inputs and multi-label outputs that single-label text systems do not.
Figures & tables
Figure 1: Overview of our framework. Inputs are processed by an LLM that directly outputs an atomic special token for the target class. Unlike baseline prompting with multi-token decoding, the framework performs classification in a single O(1) decode step (independent of output length) and supports zero-shot adaptation.
Model
Acc.
Base VLMs (no adaptation, zero-shot)
Mistral-3-24B (Base)
35.17
Gemma-3-4B (Base)
36.65
Gemma-3-27B (Base)
43.29
Frontier VLMs (zero-shot, +video)
GPT-4o
42.06
Table 1: MIntRec 2.0 accuracy (%), all models given text + video. Base VLMs and frontier VLMs are evaluated zero-shot under the same natural-language prompt (list the 30 intent names, reply with one). The strongest zero-shot model (Gemini-Pro-3.1, 55.9%) still trails our fine-tuned Gemma-3-27B (62.7%); even our 4B Koa-decision model (55.2%) surpasses every frontier VLM except Gemini-Pro-3.1.
Model
SST-2
Amazon
Intent Routing
Voice
Encoder / few-shot baselines (task-specific)
BERT (linear head)
70.87
77.04
–
–
RoBERTa (linear head)
61.35
79.16
–
–
LM-BFF
90.14
83.76
–
–
Ours — Koa-decision single-token classifiers (zero-shot on every set)
Gemma-4-26B (FT, Koa-decision)
95.76
96.60
81.43
94.33
Table 2: Text classification accuracy (%) on SST-2 (872 examples), Amazon Reviews (5k), a production intent-routing benchmark (587 utterances, with a label inventory that evolves over time), and a voice turn-taking / endpointing benchmark (353 examples, binary). Every LLM row—ours and the frontier models—is evaluated zero-shot; no model in this table was trained on any of these datasets. Our Koa-decision single-token classifiers sit inside the frontier band on every set (within ∼ 1–2 points of the best model) and lead the smaller/deployable tier, at a fraction of the serving latency (Figure 2 ). Encoder / few-shot baselines are task-specific and are not applicable to the intent-routing benchmark (its label space evolves over time, requiring retraining) or to voice endpointing. Jev is not run on the intent-routing set (internal data). Bold = best per column.
Figure 2: End-to-end per-request serving latency on the intent-routing set (587 utterances), median (p50) with a whisker to the p95 tail. Orange = our Koa-decision single-token classifier; slate = frontier LLMs. Every model is accessed through the same serving interface and issued the same requests one at a time in round-robin order, so all models see identical load and their times are directly comparable. Our classifier answers in 0.53 s (p50) — ∼ 1.9 × faster than the quickest frontier model (GPT-4.1-mini) and ∼ 7.5 × faster than the slowest (Gemini-Pro-3.1) — and has a near-flat tail (p95 = 0.63 s) where frontier models fan out by 0.5–1.7 s.
Figure 3: End-to-end per-request serving latency on the voice turn-taking / endpointing benchmark (353 examples), under the same round-robin, concurrency-1, shared serving-interface protocol as Figure 2 . Orange = our Koa-decision single-token classifier; teal = the external single-token system Jev/TypeSafe; slate = frontier LLMs. Koa-decision answers in 0.53 s (p95 0.63 s), matching its latency on the intent-routing and SST-2 tasks almost to the millisecond, and the external single-token system Jev/TypeSafe is essentially tied (0.55 s, p95 0.69 s), while every frontier model must decode over a running dialogue transcript and lands at 0.98–2.88 s at the median.
Figure 4: Accuracy vs. median serving latency on the intent-routing benchmark (log-scaled latency; lower-left of the accuracy axis omitted). All points use the same round-robin serving protocol. Our single-token classifier (orange star) occupies the fast, high-accuracy corner; the orange curve traces the Pareto frontier (Koa-decision, GPT-4.1, Claude-4.8-Opus), and the dashed arrow marks the nearest frontier point of comparable accuracy (GPT-4.1), which runs at ∼ 1.9 × our latency. No model is both faster and more accurate than Koa-decision.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: End-to-end per-request serving latency on SST-2 (872 examples), median (p50) with a whisker to the p95 tail, measured under the same round-robin, concurrency-1, shared serving-interface protocol as Figure 2 . Orange = our Koa-decision single-token classifier; teal = the external single-token system Jev/TypeSafe; slate = frontier LLMs. The two single-token systems (Koa-decision 0.53 s, Jev 0.55 s) are tied at the front and well ahead of every frontier model, isolating the speedup to the single-token formulation itself.
Dataset
Model
Acc. (%)
TweetTopic
GPT-4o
77.50
Mistral-3-24B (Base)
78.13
Mistral-3-24B (FT, Koa-decision)
81.86
GPT-OSS-20B (FT, Koa-decision)
77.42
Gemma-4-26B (FT, Koa-decision)
79.43
Banking77
GPT-4o
74.68
Appendix
Table 3: Evaluation results on TweetTopic and Banking77 across three Koa-decision backbones (Mistral-3-24B, GPT-OSS-20B, Gemma-4-26B). All Koa-decision variants are competitive with GPT-4o while retaining single-token decoding; on Banking77 the GPT-OSS-20B and Gemma-4-26B backbones both surpass GPT-4o.
Model
en
de
es
fr
hi
th
Avg
GPT-4o
79.03
78.26
77.02
79.22
74.39
78.28
77.70
Mistral-3-24B (Base)
81.08
79.29
81.65
80.18
77.45
70.81
78.41
Mistral-3-24B (FT, Koa-decision)
70.63
69.71
74.05
71.12
67.55
59.31
68.73
GPT-OSS-20B (FT, Koa-decision)
71.99
70.62
71.84
71.73
68.90
68.46
70.59
Gemma-4-26B (FT, Koa-decision)
75.06
70.75
73.95
76.45
73.07
72.73
73.67
Appendix
Table 4: Multilingual evaluation on the MTOP intent classification benchmark (6 languages) across three Koa-decision backbones.
Figure 6: Performance of Gemma-3 models across datasets.
Batch
Base P50
Koa-decision P50 (improv.)
Base P95
Koa-decision P95 (improv.)
2
0.1106
0.0545 ( 0.0561 ↓ )
0.1175
0.0875 ( 0.0300 ↓ )
4
0.0554
0.0327 ( 0.0227 ↓ )
0.0580
0.0427 ( 0.0153 ↓ )
8
0.0282
0.0205 ( 0.0077 ↓ )
0.0295
0.0222 ( 0.0073 ↓ )
16
0.0146
0.0108 ( 0.0038 ↓ )
0.0148
0.0110 ( 0.0038 ↓ )
32
0.0090
0.0068 ( 0.0022 ↓ )
0.0098
0.0069 ( 0.0029 ↓ )
64
0.0269
0.0056 ( 0.0213 ↓ )
0.0365
0.0053 ( 0.0312 ↓ )
Appendix
Table 5: Amazon Reviews — Per-sample latency (seconds) vs. batch size for Gemma-3-27B.
Run
Seed
Accuracy (%)
1
42
61.93
2
43
59.17
3
44
60.90
4
45
61.49
5
46
63.40
6
47
61.24
Appendix
Table 6: Accuracy under random permutations of control-token assignments during inference (Gemma-3-27B, full MIntRec 2.0 test set). Accuracy stays on par with the fixed-mapping result (62.72%, Table 1 ).
Figure 7: Illustration of a fine-tuning training instance in our classification datasets. Each sample includes the structured messages field and optional image_path .
Figure 8: Illustration of the class-index prompting baseline evaluated on MIntRec2.0.
Figure 9: Prompt template for SST-2 movie reviews.
Figure 11: Illustration of an inference prompt from DBpedia (Company category).
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
Current LLM systems are increasingly equipped with a code interpreter that executes generated code to obtain results. This works serially: the model first generates the complete code, then an interpreter executes it. This sequential workflow leaves the executor idle during generation and the generator idle during execution, resulting in unnecessary end-to-end latency. Our key observation is that an LLM, unlike a human developer, emits code tokens left to right and does not backtrack over what it has already written. This makes it possible to start executing a piece of code while later tokens are still being generated. We formalize this parallel execution paradigm, modeling it as a three-stage pipeline of generation, detection, and execution, and derive closed-form latency bounds that characterize its speedup potential and operating regimes. We then present EAGER, a concrete implementation featuring AST-based chunking, dynamic batching with gated execution, and early error interruption. We evaluate EAGER across four benchmarks, seven LLMs, and three execution environments. The overlap mechanism hides almost all execution behind generation, reducing the non-overlapped portion of execution time by up to 99.8% and cutting end-to-end latency by up to 37.3% on error-free runs.
Zhensu Sun, Zhihao Lin, Zhi Chen +4
Singapore Management University · Beihang University
Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.
Jev-style decision models have gained surging traction by making decision-making dramatically faster and more efficient, returning categorical probability distributions over predefined options without generating free-form text. However, because Jev's architecture and training methodologies remain proprietary, how to replicate these capabilities effectively remains an open question. In this work, we investigate the extent to which general-purpose LLMs already possess this capability out of the box, and when fine-tuning is actually necessary. We present LLM-as-Jev, an architecture-preserving framework offering both a training-free inference recipe and a fine-tuning objective with KL-divergence anchors. Across Qwen3.5-4B and Qwen3-0.6B, we find that modern LLMs are inherently effective decision models: off the shelf, frozen models match or exceed existing training-free readouts on the same 4B backbone, scale to arbitrary option counts, and natively handle multimodal decisions. They also seamlessly inherit the backbone's thinking mode: while Jev was introduced as a System 1 model, "Thinking Jev" leverages budgeted test-time reasoning to boost accuracy on challenging tasks without training. Fine-tuning provides targeted gains for smaller models and complex routing, while our KL anchors prevent behavioral degradation in general text generation.