Answering While You Type: Optimizing Time-to-Answer with Streaming Generation

Nikita Dragunov; Matvey Mikhalchuk; Anton Razzhigaev

cs.CLOct 19, 2026

Answering While You Type: Optimizing Time-to-Answer with Streaming Generation

Authors: Nikita Dragunov, Matvey Mikhalchuk, Anton Razzhigaev

Organizations: FusionBrain Lab

Abstract

Work on efficient large-language-model inference has concentrated almost entirely on reducing the cost of decoding once a request is complete. The quantity a user experiences, however, is the interval between submitting a request and receiving a usable response, and that interval can be shortened without making decoding itself faster. While a request is being composed it is already partially specified, yet conventional deployments begin no computation until submission. We propose and study streaming generation, a decoding regime in which the model begins producing a response from an incomplete request and revises it as the remaining input arrives, so that only a small residue of the answer remains to be computed once the request is submitted. We evaluate the regime on autoregressive and diffusion language models, characterise the conditions under which it reduces user-visible latency, and analyse the mechanism responsible.

Figures & tables

Benchmark	base	stream	base	stream	mean	median	mean	median
	Quality (%)		Forwards after final token		Speedup, forwards ( $\times$ )		Speedup, wall clock ( $\times$ )
HumanEval (pass@1)	54.3	54.3	16	3	5.10	4.67	3.86	3.68
MBPP (pass@1)	55.1	57.1	12	3	3.76	3.33	3.21	2.90
CanItEdit (pass@1)	19.7	23.7	42	27	3.91	1.36	2.85	1.25
DS-1000 (pass@1)	3.9	1.6	18	11	2.12	1.43	1.76	1.32
CoEdIT (F1)	72.9	71.8	26	19	1.82	1.07	1.63	1.03

Table 1: ReFusion-8B. base : decoding the complete request with the benchmark’s default configuration. stream : the procedure of Section 3 . Both come from one run over the same examples. Forward counts are medians. Means exceed medians on every benchmark because the distribution carries a long right tail (Figure 1 ).

		Quality (%)		Speedup, forwards ( $\times$ )		Speedup, wall clock ( $\times$ )
Model	Benchmark	base	stream	mean	median	mean	median
Gemma-3-12B	MBPP (pass@1)	75.8	65.8	75.89	74.00	63.73	67.80
	HumanEval (pass@1)	84.1	75.0	63.05	7.98	49.13	7.67
	CanItEdit (pass@1)	41.0	30.5	53.95	1.44	20.84	1.40
	DS-1000 (pass@1)	0.0	10.2	2.38	1.04	2.23	1.03
	CoEdIT (F1)	39.0	39.2	7.89	1.05	7.07	1.05

Table 2: Autoregressive models with no adaptation, per benchmark. average is the macro-average over the six benchmarks. The gain concentrates on the two specification-first coding benchmarks, as it does for the diffusion model in Table 1 , and means run far above medians for the same reason.

Figure 1: Per-example residual speedup on ReFusion-8B, one panel per benchmark, on a logarithmic axis. Orange marks the median, dotted red the mean, dashed black the

1\times

line at which streaming costs what the baseline costs.

Figure 2: Per-example residual speedup on the three autoregressive models, on the same axes as Figure 1 . Orange marks the pooled median, dotted red the pooled mean, dashed black the

1\times

line.

Autoregressive
Model	Benchmark	quality	median speedup
Qwen3-8B	MBPP	$67.2\to 59.2$	$1.09\to 1.50$
Qwen3-8B	HumanEval	$52.4\to 59.8$	$2.12\to 3.44$
Llama-3.1-8B	average	$41.6\to 41.7$	$1.23\to 1.18$
Gemma-3-12B	average	$42.1\to 43.3$	$14.42\to 15.20$
Diffusion (ReFusion-8B)

Table 3: Adaptation to incomplete requests, before

\to

after.

Streaming Inference

See all 59 papers

Executing as You Generate: Hiding Execution Latency in LLM Code Interpreters
Apr 1, 2026Zhensu Sun, Zhihao Lin, Zhi Chen +4LLM Inference Acceleration Pipeline Parallelism
Singapore Management University · Beihang University
Micro Language Models Enable Instant Responses
Apr 21, 2026Wen Cheng, Tuochao Chen, Karim Helwani +3On-Device Language Model Inference Small Language Models
Paul G. Allen School of Computer Science & Engineering, University of Washington · Meta AI
Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
Apr 3, 2026Cornelius Kummer, Lena Jurkschat, Michael Färber +1Prompt Compression LLM Inference Efficiency
ScaDS.AI Dresden/Leipzig, CIDS, TU Dresden

Apr 1, 2026J/K move · Enter open · S save

Executing as You Generate: Hiding Execution Latency in LLM Code Interpreters

Zhensu Sun, Zhihao Lin, Zhi Chen, Chengran Yang+3

Singapore Management University · Beihang University

LLM Inference Acceleration Pipeline Parallelism LLM Code Generation

Open paper AskPDF arXiv

Current LLM systems are increasingly equipped with a code interpreter that executes generated code to obtain results. This works serially: the model first generates the complete code, then an interpreter executes it. This sequential workflow leaves the executor idle during generation and the generator idle during execution, resulting in unnecessary end-to-end latency. Our key observation is that an LLM, unlike a human developer, emits code tokens left to right and does not backtrack over what it has already written. This makes it possible to start executing a piece of code while later tokens are still being generated. We formalize this parallel execution paradigm, modeling it as a three-stage pipeline of generation, detection, and execution, and derive closed-form latency bounds that characterize its speedup potential and operating regimes. We then present EAGER, a concrete implementation featuring AST-based chunking, dynamic batching with gated execution, and early error interruption. We evaluate EAGER across four benchmarks, seven LLMs, and three execution environments. The overlap mechanism hides almost all execution behind generation, reducing the non-overlapped portion of execution time by up to 99.8% and cutting end-to-end latency by up to 37.3% on error-free runs.

Abstract

Streaming Inference

Explore similar work

Executing as You Generate: Hiding Execution Latency in LLM Code Interpreters

Micro Language Models Enable Instant Responses

Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference