Industry applications often demand low-latency classification, yet current large language model (LLM) approaches remain poorly suited for latency-critical applications. Existing prompting and constrained decoding produce verbose, multi-token outputs that require expensive token-by-token generation, while encoder-based models achieve faster inference but sacrifice task flexibility. We propose Koa-decision, a framework for low-latency atomic actions -- fast, single-step decisions such as classification, semantic endpointing, Boolean checks, and scoring -- formulated as constrained generation with single-token outputs. By introducing atomic label tokens and applying supervised fine-tuning, our method reduces classification to a deterministic one-step decoding problem. Across standard benchmarks, Koa-decision delivers competitive accuracy with consistently low and stable latency. On a production intent-routing benchmark, Koa-decision reaches 85.5% accuracy -- competitive with the strongest frontier models (Claude-4.8-Opus, Gemini-Pro-3.1) and ahead of GPT-5 and Gemini-2.5-Pro -- while answering in about half a second, several-fold faster than every frontier model (up to ~7.5x at the median) under identical serving conditions. Against the dedicated single-token system Jev/TypeSafe, Koa-decision is competitive on accuracy and faster at the median, while also handling multimodal inputs and multi-label outputs that single-label text systems do not.
Figures & tables
Figure 1: Overview of our framework. Inputs are processed by an LLM that directly outputs an atomic special token for the target class. Unlike baseline prompting with multi-token decoding, the framework performs classification in a single O(1) decode step (independent of output length) and supports zero-shot adaptation.
Model
Acc.
Base VLMs (no adaptation, zero-shot)
Mistral-3-24B (Base)
35.17
Gemma-3-4B (Base)
36.65
Gemma-3-27B (Base)
43.29
Frontier VLMs (zero-shot, +video)
GPT-4o
42.06
Table 1: MIntRec 2.0 accuracy (%), all models given text + video. Base VLMs and frontier VLMs are evaluated zero-shot under the same natural-language prompt (list the 30 intent names, reply with one). The strongest zero-shot model (Gemini-Pro-3.1, 55.9%) still trails our fine-tuned Gemma-3-27B (62.7%); even our 4B Koa-decision model (55.2%) surpasses every frontier VLM except Gemini-Pro-3.1.
Model
SST-2
Amazon
Intent Routing
Voice
Encoder / few-shot baselines (task-specific)
BERT (linear head)
70.87
77.04
–
–
RoBERTa (linear head)
61.35
79.16
–
–
LM-BFF
90.14
83.76
–
–
Ours — Koa-decision single-token classifiers (zero-shot on every set)
Gemma-4-26B (FT, Koa-decision)
95.76
96.60
81.43
94.33
Table 2: Text classification accuracy (%) on SST-2 (872 examples), Amazon Reviews (5k), a production intent-routing benchmark (587 utterances, with a label inventory that evolves over time), and a voice turn-taking / endpointing benchmark (353 examples, binary). Every LLM row—ours and the frontier models—is evaluated zero-shot; no model in this table was trained on any of these datasets. Our Koa-decision single-token classifiers sit inside the frontier band on every set (within ∼ 1–2 points of the best model) and lead the smaller/deployable tier, at a fraction of the serving latency (Figure 2 ). Encoder / few-shot baselines are task-specific and are not applicable to the intent-routing benchmark (its label space evolves over time, requiring retraining) or to voice endpointing. Jev is not run on the intent-routing set (internal data). Bold = best per column.
Figure 2: End-to-end per-request serving latency on the intent-routing set (587 utterances), median (p50) with a whisker to the p95 tail. Orange = our Koa-decision single-token classifier; slate = frontier LLMs. Every model is accessed through the same serving interface and issued the same requests one at a time in round-robin order, so all models see identical load and their times are directly comparable. Our classifier answers in 0.53 s (p50) — ∼ 1.9 × faster than the quickest frontier model (GPT-4.1-mini) and ∼ 7.5 × faster than the slowest (Gemini-Pro-3.1) — and has a near-flat tail (p95 = 0.63 s) where frontier models fan out by 0.5–1.7 s.
Figure 3: End-to-end per-request serving latency on the voice turn-taking / endpointing benchmark (353 examples), under the same round-robin, concurrency-1, shared serving-interface protocol as Figure 2 . Orange = our Koa-decision single-token classifier; teal = the external single-token system Jev/TypeSafe; slate = frontier LLMs. Koa-decision answers in 0.53 s (p95 0.63 s), matching its latency on the intent-routing and SST-2 tasks almost to the millisecond, and the external single-token system Jev/TypeSafe is essentially tied (0.55 s, p95 0.69 s), while every frontier model must decode over a running dialogue transcript and lands at 0.98–2.88 s at the median.
Figure 4: Accuracy vs. median serving latency on the intent-routing benchmark (log-scaled latency; lower-left of the accuracy axis omitted). All points use the same round-robin serving protocol. Our single-token classifier (orange star) occupies the fast, high-accuracy corner; the orange curve traces the Pareto frontier (Koa-decision, GPT-4.1, Claude-4.8-Opus), and the dashed arrow marks the nearest frontier point of comparable accuracy (GPT-4.1), which runs at ∼ 1.9 × our latency. No model is both faster and more accurate than Koa-decision.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: End-to-end per-request serving latency on SST-2 (872 examples), median (p50) with a whisker to the p95 tail, measured under the same round-robin, concurrency-1, shared serving-interface protocol as Figure 2 . Orange = our Koa-decision single-token classifier; teal = the external single-token system Jev/TypeSafe; slate = frontier LLMs. The two single-token systems (Koa-decision 0.53 s, Jev 0.55 s) are tied at the front and well ahead of every frontier model, isolating the speedup to the single-token formulation itself.
Dataset
Model
Acc. (%)
TweetTopic
GPT-4o
77.50
Mistral-3-24B (Base)
78.13
Mistral-3-24B (FT, Koa-decision)
81.86
GPT-OSS-20B (FT, Koa-decision)
77.42
Gemma-4-26B (FT, Koa-decision)
79.43
Banking77
GPT-4o
74.68
Appendix
Table 3: Evaluation results on TweetTopic and Banking77 across three Koa-decision backbones (Mistral-3-24B, GPT-OSS-20B, Gemma-4-26B). All Koa-decision variants are competitive with GPT-4o while retaining single-token decoding; on Banking77 the GPT-OSS-20B and Gemma-4-26B backbones both surpass GPT-4o.
Model
en
de
es
fr
hi
th
Avg
GPT-4o
79.03
78.26
77.02
79.22
74.39
78.28
77.70
Mistral-3-24B (Base)
81.08
79.29
81.65
80.18
77.45
70.81
78.41
Mistral-3-24B (FT, Koa-decision)
70.63
69.71
74.05
71.12
67.55
59.31
68.73
GPT-OSS-20B (FT, Koa-decision)
71.99
70.62
71.84
71.73
68.90
68.46
70.59
Gemma-4-26B (FT, Koa-decision)
75.06
70.75
73.95
76.45
73.07
72.73
73.67
Appendix
Table 4: Multilingual evaluation on the MTOP intent classification benchmark (6 languages) across three Koa-decision backbones.
Figure 6: Performance of Gemma-3 models across datasets.
Batch
Base P50
Koa-decision P50 (improv.)
Base P95
Koa-decision P95 (improv.)
2
0.1106
0.0545 ( 0.0561 ↓ )
0.1175
0.0875 ( 0.0300 ↓ )
4
0.0554
0.0327 ( 0.0227 ↓ )
0.0580
0.0427 ( 0.0153 ↓ )
8
0.0282
0.0205 ( 0.0077 ↓ )
0.0295
0.0222 ( 0.0073 ↓ )
16
0.0146
0.0108 ( 0.0038 ↓ )
0.0148
0.0110 ( 0.0038 ↓ )
32
0.0090
0.0068 ( 0.0022 ↓ )
0.0098
0.0069 ( 0.0029 ↓ )
64
0.0269
0.0056 ( 0.0213 ↓ )
0.0365
0.0053 ( 0.0312 ↓ )
Appendix
Table 5: Amazon Reviews — Per-sample latency (seconds) vs. batch size for Gemma-3-27B.
Run
Seed
Accuracy (%)
1
42
61.93
2
43
59.17
3
44
60.90
4
45
61.49
5
46
63.40
6
47
61.24
Appendix
Table 6: Accuracy under random permutations of control-token assignments during inference (Gemma-3-27B, full MIntRec 2.0 test set). Accuracy stays on par with the fixed-mapping result (62.72%, Table 1 ).
Figure 7: Illustration of a fine-tuning training instance in our classification datasets. Each sample includes the structured messages field and optional image_path .
Figure 8: Illustration of the class-index prompting baseline evaluated on MIntRec2.0.
Figure 9: Prompt template for SST-2 movie reviews.
Figure 11: Illustration of an inference prompt from DBpedia (Company category).
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.