Inference for Reasoning Models: Practical Lessons from Evaluation and Serving
Abstract
Running a language model involves more than its trained weights. The same model can receive very different benchmark scores, produce different answers, or cost more to serve depending on how it is evaluated and run. In one experiment in this book, changing only the maximum response length moved accuracy on a math benchmark from 0.08% to 93.18%. This book explains how inference settings shape the results we report and the systems we deploy. We begin with evaluation, including generation limits, answer extraction, scoring methods, and quantization. We then examine why outputs can change across batches, attention backends, and caching, even when sampling is disabled. The next chapters cover serving performance, including batching, memory capacity, latency, throughput, and cost. We also study structured outputs, tool calls, reasoning budgets, and long-context tasks, showing what these controls and benchmarks do and do not establish. The book concludes with practical guidance for recording inference runs, checking evaluations, estimating capacity, and comparing systems fairly. Written for researchers and engineers working with language models, it aims to make inference results easier to understand, reproduce, and trust.
Figures & tables
| Cap | Strict acc. | Truncation | prior | Interpretation under this run |
|---|---|---|---|---|
| tokens | percent | percent | pp | |
| n/a | Fully cap-limited | |||
| Fully cap-limited | ||||
| Fully cap-limited | ||||
| Almost fully cap-limited | ||||
| Capability and truncation mixed |
| Extraction rule | Accuracy | Correct | Observed behavior |
|---|---|---|---|
| percent | items | ||
| Last number in the response | Recovered later final values in some responses | ||
| Number after the literal #### marker | Missed responses whose final value did not sit at the marker | ||
| Loose phrase/equals-pattern heuristic | Selected intermediate arithmetic in some responses |
| Model / condition | Gap | Interpretation under this run |
|---|---|---|
| pp | ||
| Qwen3-14B (E3) | Large path incompatibility here | |
| Llama-3.1-8B (H2) | Material interaction on replication | |
| Mistral-7B (H2) | Small interaction; family-specific |
| Order ID | Example permutation | Accuracy (percent) |
|---|---|---|
| 0 | [0, 1, 2, 3, 4] | 35.8 |
| 1 | [3, 4, 0, 1, 2] | 51.0 |
| 2 | [3, 2, 1, 4, 0] | 48.0 |
| 3 | [4, 1, 3, 0, 2] | 69.8 |
| 4 | [4, 2, 3, 0, 1] | 68.9 |
| Task | BF16 | AWQ-INT4 | Diff. | Interpretation under this run |
|---|---|---|---|---|
| percent | percent | pp | ||
| MMLU | Small negative difference | |||
| GSM8K | No evidence of a loss | |||
| HumanEval | Largest drop; 5 of 164 items |
| Task | BF16 | AWQ | Diff. | Scope conclusion |
|---|---|---|---|---|
| percent | percent | pp | ||
| MMLU | Qwen’s MMLU delta did not reproduce | |||
| GSM8K | No measured degradation here | |||
| HumanEval | Qwen’s code-specific loss absent |
| Regime | Cap pair | Agreement | Correctness transitions |
|---|---|---|---|
| tokens | percent | ||
| Below transition plateau | 256 / 1,024 | 197 improved, 0 worsened, 315 neutral (n = 512); the smaller cap was still truncation-confounded | |
| Plateau more headroom | 1,024 / 4,096 | 0 improved, 1 worsened; 7 answers changed (n = 1,319); no measured accuracy gain |
| Cap | Strict acc. | Mean used | vs 1,024 | Interpretation |
|---|---|---|---|---|
| tokens | percent | tokens | pp | |
| Still below the usable range | ||||
| Near the plateau | ||||
| 0.00 | Plateau reference | |||
| 0.00 | No measured accuracy gain | |||
| No gain; 1 answer flipped wrong |
| Model | Arch. | Distinct compl. | Repeats | Interpretation |
|---|---|---|---|---|
| Qwen3-8B | dense | 1 | no fork in this sweep | |
| Qwen3-14B | dense | 2 | dense models vary too | |
| Qwen3-30B-A3B | MoE | 2 | at least the 8B control | |
| gpt-oss-20b | MoE | 3.8 (3–4) | non-Qwen MoE replication |
| Comparison | Text div. | Score, two paths | Interpretation |
|---|---|---|---|
| percent | percent | ||
| Qwen3-8B, cap 256 | 52.0 | 53.1 / 54.7 | cap-confounded accuracy |
| Qwen3-8B, cap 512 | 55.1 | 90.2 / 92.6 | persisted above the knee |
| Qwen3-8B reruns | 0.86 ± 0.16 | not reported | rerun noise is far smaller |
| Llama-3.1-8B | 39.1 | not reported | second-family replication |
| Mistral-7B | 43.0 | not reported | third-family replication |
| Workload | Prefix condition | Speedup | Engine |
|---|---|---|---|
| ratio | |||
| Identical prompts | shared inside one batch | 1.00 | grouped batch |
| Across batches | ~1.5k-token prefix | 1.40 | RL rollout study |
| E16 shared prefix | long prefix, 200 prompts | 1.74 | vLLM 0.11.0 |
| E42 repeated query | ~3k prefix, 40 queries | 5.4–8.3 | vLLM 0.23 |
| Model | Text div. | Repetition | Interpretation |
|---|---|---|---|
| percent | |||
| Qwen3-8B | 51.5 ± 0.9 | five repeats | hardened estimate |
| Llama-3.1-8B | 41.5 | single run | present in second family |
| Mistral-7B | 30.0 | single run | present in third family |
| GPU memory | Cache on | Cache off | Speedup |
|---|---|---|---|
| fraction | ms / query | ms / query | ratio |
| 0.90 | 14.7 | 0 84.1 | 5.70 |
| 0.55 | 15.6 | 0 84.1 | 5.40 |
| 0.35 | 15.8 | 0 89.2 | 5.65 |
| 0.25 | 15.5 | 129.1 | 8.34 |
| Model setup | Text div. | Accuracy | C W |
|---|---|---|---|
| percent | pp | percent | |
| Qwen3-8B, full | 46.9 ± 3.1 | −0.10 ± 0.66 | 1.5 ± 0.7 |
| Llama-3.1-8B, full | 43.3 ± 2.5 | +0.90 ± 1.43 | 2.0 ± 1.1 |
| Qwen3.5-9B, hybrid | 33.9 ± 2.7 | +0.00 ± 0.55 | 1.1 ± 0.5 |
| Reference path | Changed path | Interpretation |
|---|---|---|
| correct | correct | stable correct result |
| correct | wrong | harmful transition |
| wrong | correct | beneficial transition |
| wrong | wrong | stable error |
| Contract | Required equality | Acceptance test | Typical use |
|---|---|---|---|
| Statistical | Metric distribution | Repeat in interval | research claims |
| Task-level | Score in tolerance | Paired impact test | production validation |
| Bitwise replay | Identical tokens | Exact equality | bisection, audit |
| Execution change | Text-stability evidence | Task-impact evidence | Defensible conclusion |
|---|---|---|---|
| Batch context | Greedy-path changes possible; exact counts prompt-sensitive | Not measured in a paired study | Belongs in exact-replay metadata; task impact open |
| Tested MoE paths | Greater batch-context variability; routes not logged | Not measured | Replay needs added controls; routing claim bounded |
| Attention backend | 39–55% divergence, three families; same-backend under 1% | Both paths above 90% on one clean slice; no repeated paired study | Report the backend; quality equivalence not established |
| Prefix cache | 34–47% divergence, five slices, three models | Net GSM8K change near zero; C W 1.1–2.0% | No consistent net loss on this task; local acceptance test still required |
| Execution path | Batching policy | Output | vs sequential |
|---|---|---|---|
| tok/s | ratio | ||
| Hugging Face | one request at a time | ||
| Hugging Face | fixed batch of 32 | ||
| vLLM 0.11 | engine-managed batch |
| Serving policy | Output | Cost | Relative | Latency* |
|---|---|---|---|---|
| tok/s | $/M out tok | ratio | s | |
| Static B=1 | 139.6 | 7.96 | 126.7 | 1.83 |
| Static B=8 | 947 | 1.17 | 18.6 | 1.92 |
| Static B=32 | 3,377 | 0.329 | 5.2 | 2.13 |
| Static B=128 | 10,230 | 0.109 | 1.7 | 2.82 |
| Continuous | 17,686 | 0.063 | 1.0 | 6.54 |
| Offered | Achieved | Served | Median | p99 | p99 : median |
|---|---|---|---|---|---|
| req/s | req/s | percent | TTFT, ms | TTFT, ms | ratio |
| 4 | 3.77 | 94 | 67.2 | 79.7 | 1.19 |
| 16 | 12.79 | 80 | 73.8 | 88.2 | 1.20 |
| 64 | 31.46 | 49 | 84.0 | 123.8 | 1.47 |
| Model | Accuracy | Tokens | Output | Cost |
|---|---|---|---|---|
| percent | per correct | tok/s | $ per 1,000 correct | |
| Qwen3-8B | 91.99 | 259.3 | 13,106 | 0.0248 |
| Qwen3-14B | 94.73 | 235.0 | 8,750 | 0.0323 |
| Qwen3-32B | 94.53 | 239.9 | 2,236 | 0.1293 |
| Active context | Engine KV pool | Derived bound |
|---|---|---|
| tokens per request | token slots | concurrent sequences |
| 2,048 | 245,088 | 119 |
| 8,192 | 245,088 | 29 |
| 16,384 | 245,088 | 14 |
| 32,768 | 245,088 | 7 |
| Model | Token budget | TTFT | ITL |
|---|---|---|---|
| tokens | ms, mean ± SD | ms, mean ± SD | |
| Qwen3-8B | 512 | 103.2 ± 2.8 | 24.4 ± 0.4 |
| Qwen3-8B | 2,048 | 71.7 ± 1.7 | 23.5 ± 0.1 |
| Qwen3-8B | 8,192 | 70.9 ± 1.9 | 23.3 ± 0.2 |
| Qwen3-32B | 512 | 4505 ± 144 | 41.3 ± 0.2 |
| Qwen3-32B | 2,048 | 9378 ± 141 | 40.9 ± 0.2 |
| Intervention | Measured under the tested node | Condition where it may help |
|---|---|---|
| Disaggregation | Net-negative; transfer-bound | Multiple nodes, fast KV fabric |
| Head-of-line fix | Only 1.02–1.07 inflation | Schedulers without batching |
| Faster storage | Warm 11.07 vs 10.90 s; cold 34.7 vs 130 MB/s | Cold, storage-bound first touch |
| Regime | Shared FS | Local | Ratio | What dominates |
|---|---|---|---|---|
| cold raw read | 34.7 MB/s | 130 MB/s | 3.75 | storage throughput |
| cold engine init | 201.2 s | 138.5 s | 1.45 | weight fetch |
| warm engine init | 11.07 s | 10.90 s | 1.02 | runtime warmup |
| Lever | Best gain | Bounded cost | First production check |
|---|---|---|---|
| Continuous batching | 126.7 | p99 tail | offered vs achieved |
| Engine vs plain HF | 20.5 | not comparable | use batched HF |
| KV-budget planning | prevents OOM | concurrency loss | capacity / context |
| Chunked prefill | % TTFT | 2.08 on 32B | tune per model |
| P/D disaggregation | none here | KV transfer cost | needs a fast fabric |
| Model | Freeform | Asked | Forced | Request | Decoder |
|---|---|---|---|---|---|
| percent | percent | percent | pp | pp | |
| Qwen3-8B | 90.9 | 95.1 | 93.9 | −4.2 | +1.2 |
| gpt-oss-20b | 73.8 | 79.5 | 75.1 | −5.7 | +4.4 |
| Llama-3.1-8B | 83.9 | 80.5 | 55.2 | +3.4 | +25.3 |
| Batch | Model | Unconstrained | Constrained | Factor |
|---|---|---|---|---|
| tok/s | tok/s | ratio | ||
| 8 | Qwen3-8B | 1,155 | 908 | 1.27 |
| 8 | Llama-3.1-8B | 1,121 | 1,063 | 1.05 |
| 32 | Qwen3-8B | 4,387 | 3,685 | 1.19 |
| 32 | Llama-3.1-8B | 4,460 | 4,156 | 1.07 |
| Model | Format | AST acc. | ||
|---|---|---|---|---|
| native | with schema | native | with schema | |
| Qwen3-8B | 0.845 | +0.155 | 0.520 | +0.055 |
| Llama-3.1-8B | 1.000 | +0.000 | 0.555 | +0.000 |
| Mistral-7B | 1.000 | +0.000 | 0.545 | +0.000 |
| Model | Requested | Actual | Adherence | Accuracy |
|---|---|---|---|---|
| tokens | tokens, mean | error | percent | |
| Qwen3-8B | 256 | 433 | +0.69 | 92.0 |
| Qwen3-8B | 512 | 667 | +0.30 | 90.0 |
| Qwen3-8B | 1024 | 735 | −0.28 | 89.3 |
| Qwen3-30B-A3B | 256 | 397 | +0.55 | 97.3 |
| Qwen3-30B-A3B | 512 | 479 | −0.06 | 94.0 |
| Context | Four-needle retrieval | Common-words aggregation | Six-step tracking |
|---|---|---|---|
| tokens | trials correct | trials correct | trials correct |
| 2k | 30/30 | 29/30 | 30/30 |
| 16k | 30/30 | 29/30 | 9/30 |
| 28k | 30/30 | 27/30 | 10/30 |
| Context | Qwen3-8B | Qwen3-14B | Qwen3-32B |
|---|---|---|---|
| tokens | trials correct | trials correct | trials correct |
| 2k | 30/30 | 30/30 | 30/30 |
| 16k | 9/30 | 30/30 | 30/30 |
| 28k | 10/30 | 30/30 | 30/30 |
| Task and checkpoint | Last pass / first fail | Reported range |
|---|---|---|
| Tracking, Qwen3-8B | 2k / 16k | Crossing bracketed between the 2k pass and the 16k decline |
| Tracking, Qwen3-14B | 28k / not observed | At least 28k under the tested protocol |
| Tracking, Qwen3-32B | 28k / not observed | At least 28k under the tested protocol |
| Checkpoint | Attention | KV capacity | Derived bound |
|---|---|---|---|
| token slots | 16k sequences | ||
| Qwen3-8B | full | 221,056 | 13 |
| Qwen3.5-9B | hybrid | 841,632 | 52 |
| Context | Qwen3-8B | Qwen3.5-9B | Ratio |
|---|---|---|---|
| tokens | tok/s | tok/s | full / hybrid |
| 8k | 93,115 | 35,114 | 2.65 |
| 16k | 140,975 | 37,055 | 3.80 |
| 28k | 179,737 | 34,835 | 5.16 |
| Axis | Comparison control | Allowed conclusion |
|---|---|---|
| Six-step tracking | Different checkpoints and engine versions | Qwen3.5-9B performed better under the tested protocols |
| KV capacity | Same engine, hardware, memory setting | Qwen3.5-9B exposed 3.81 more KV capacity |
| Prefill throughput | Same engine and workload | Qwen3-8B was faster in this micro-benchmark |
| Prefix-cache impact | Same framework, five disjoint slices | Qwen3.5-9B drifted least; nonzero flips remained |
| Checkpoint | Parameters | GSM8K | Throughput | Memory |
|---|---|---|---|---|
| total / active | percent | tok/s | GB resident | |
| Qwen3-4B | 4B dense | 92.58 | 12,767 | 17 |
| Qwen3-30B-A3B | 30B / 3B | 93.75 | 4,618 | 70 |
| Checkpoint | MMLU | GSM8K | HumanEval | Pattern |
|---|---|---|---|---|
| pp | pp | pp | ||
| Qwen3-14B dense | −0.70 | +0.61 | −3.04 | Code-only loss |
| Qwen3-30B-A3B MoE | −2.6 | −1.9 | −1.8 | Broad degradation |
| Mistral-7B dense | +4.9 | +2.6 | +3.0 | Did not replicate |
| Measurement | Result | What is directly established |
|---|---|---|
| Top-1 expert change | 31.0% | The top expert differed frequently |
| Top- set change | 86.8% | At least one expert differed in most cases |
| Router gate quantized | no | Route changes followed activation drift |
| Routing entropy | 3.94 / 3.61 | AWQ routing was somewhat more concentrated |
| Claim | Label | Allowed conclusion |
|---|---|---|
| Curves differ by operation | task profile | Operations differ in sensitivity |
| Larger Qwen3 held to 28k | matched family | The 14B and 32B held the range |
| Qwen3.5-9B led on tracking | boundary | Higher under the tested protocols |
| Qwen3.5-9B: 3.81 KV capacity | controlled | It exposed a larger pool |
| Qwen3-8B led on prefill | controlled | It was faster on this benchmark |
| Qwen MoE AWQ rerouted | mechanistic | Expert selections differed under AWQ |
| Axis | Measurement contract | Part V result |
|---|---|---|
| Retrieval | Fixed placement, exact scoring | Correct through 28k |
| Aggregation | Distributed evidence, task metric | 90% or above through 28k |
| Chained reasoning | Workload-matched task and SLO | 8B declined by 16k; larger held |
| KV capacity | Same engine, memory setting, concurrency | Qwen3.5-9B exposed 3.81 more |
| Throughput | Same engine, precise workload metric | Qwen3-8B led on batch-8 prefill |
| MoE quantization | Exact checkpoint, quantizer, route logging | AWQ lost broadly and rerouted |
| Layer | Anchor fields | What it lets you reconstruct |
|---|---|---|
| Identity | Model and tokenizer revision, prompt and dataset hashes | What was evaluated, and where the evidence sits |
| Substrate | Container digest, engine version, driver, GPU topology | The stack the run executed on |
| Execution path | Precision, backend, cache, scheduler, decoding | The numerical and scheduling path |
| Evaluation | Extractor, scorer, metric, exclusions, uncertainty | How outputs became the reported number |
| Stage | Records to fix before launch |
|---|---|
| Identity | Checkpoint revision, tokenizer, chat-template hash, dataset revision and split, item IDs, preprocessing, rendered prompt |
| Execution | Engine and version, CUDA and framework, attention backend, precision, quantization, KV dtype, cache state, batch policy, scheduler |
| Decoding | Mode, temperature, thinking-mode state, generation budget, stop rules, seed policy, request order |
| Stage | Records and artifacts |
|---|---|
| Scoring | Raw outputs and item IDs; extractor or verifier and its version; item-level judgments; spot-check sample; uncertainty from repeats, slices, or seeds |
| Comparison | Shared and differing conditions per arm; paired transitions; comparison-design label, common-prompt or native-template |
| Release bundle | Manifest; item IDs and prompt-render hashes; raw outputs; item-level scores and scorer revision; aggregates with uncertainty; engine logs; canonical data; artifact hashes; evidence-status record |
| Calculation | Required inputs | Status |
|---|---|---|
| Engine block report | Measured | |
| , full attention only | Reserved memory, layers, KV heads, head dim, precision | Analytical estimate |
| Pool, prompt plus reserved output | Derived bound | |
| \/\text{M tok}=(r_{$}/T_{out})\times 10^{6}$ | GPU count, price, output throughput | Derived estimate |
| \/\text{success}=\text{cost}/N_{\mathrm{successful}}$ | Total measured cost, successful items | Derived estimate |
| Stage | Risk tested | What it settled |
|---|---|---|
| H1 | Repeatability and false precision | Which headline magnitudes were stable under repeats |
| H2 | Cross-family scope | Which patterns held beyond one model family |
| H3 | Quote safety under truncation | Backend divergence persisted above the truncation knee |
| H4 | Engine confound closure | The hybrid speed win did not survive a same-engine rerun |
| H5 | Methodology extensions | Real-arrival latency and a harder long-context task |
| Quantity | Preliminary | Hardened | Status |
|---|---|---|---|
| Backend text divergence | ~52% | ~52% | Confirmed |
| Prefix-cache divergence | 40% | 51.5 ± 0.9% | Increased |
| E6b helps / hurts | 197 / 0 | 198 / 1.4 | Stable |
| FP8-KV serving delta | pp | pp | Reduced |
| MoE completion counts | 2 vs 4 | 1 vs 2 | Not stable |
| Finding | Replication depth | Supported claim |
|---|---|---|
| Text divergence | Qwen, Llama, Mistral | Material in all three; state per-model ranges |
| MoE completion diversity | Qwen3-30B-A3B, gpt-oss-20b | A two-family pattern, not a fixed ratio |
| MMLU harness gap | Large on Qwen, smaller on Llama, near zero on Mistral | Model-dependent; validate per model |
| Qwen AWQ task split | Not reproduced on Mistral AWQ | Checkpoint- and toolchain-specific |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Experiment | Acceptance type | Replication target, and what may differ |
|---|---|---|
| E1 truncation | curve | Accuracy rises as truncation falls to a plateau; the exact knee may differ |
| E2 extractor | exact on frozen bytes | Identical bytes receive scorer-dependent results; the spread differs on new outputs |
| E5 exemplar order | directional, scoped | Order materially affects the tested prompt; the exact 34-point range may differ |
| E7 batch context | directional | A batch context may fork the greedy path; the exact fork token differs |
| E11 backend | statistical, control | A/B divergence materially exceeds the same-backend repeat; the percentage differs |
| E16/E41 cache | statistical | Cache can change text; task impact measured separately; magnitude differs |
| Location | Primary claim or function | Status |
|---|---|---|
| Ch. 1 | Generation-budget truncation curve (E1) | canonical measured |
| Ch. 2 | Scorer and extractor spread on frozen outputs (E2) | canonical; verifier wording guarded |
| Ch. 3 | Harness and model interaction, exemplar order (E3/H2; E5) | harness hardened; order scoped |
| Ch. 4 | Checkpoint- and task-dependent quantization (E8; H2; E9) | model and toolchain scoped |
| Ch. 5 | Budget plateau and sampled short/long (E6a; H1-E6b) | separate deterministic and sampled contracts |
| Ch. 6 | Batch-context sensitivity of greedy decoding (E7/H1) | qualitative canonical; exact count fragile |
LLM Inference
Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.
See all 315 papers