SEIS: Self-Evolving Inference Systems

Zhen Xu; Jingyu Liu; Zongze Li; Tahseen Rabbani; Ce Zhang

cs.AIOct 3, 2026

SEIS: Self-Evolving Inference Systems

Authors: Zhen Xu, Jingyu Liu, Zongze Li, Tahseen Rabbani, Ce Zhang

Organizations: University of Chicago

Abstract

Inference systems determine how fast and how cheaply language models can be served, so making them faster has direct practical value. However, prior work focuses mostly on optimizing certain parts such as kernels or memory within the large system. In this work, we take a holistic approach and apply agentic self-evolution to optimize the whole system end-to-end. Our SEIS (Self-Evolving Inference Systems) autonomously optimizes the entire mini-sglang engine without human intervention through iterative sessions with inherited experiences and code changes. Serving Qwen3-0.6B on H100, the resulting engine reaches 3.27X the throughput of the original mini-sglang implementation and beats SOTA engines like vLLM, TensorRT-LLM, and SGLang in the single-request workload. The correctness of the optimized inference engine by SEIS is tested in terms of numerical difference and downstream accuracy on math and long-context retrieval tasks. The code and session histories show that the speedup comes from redesigning the whole engine and that building on earlier sessions beats independent attempts. These results suggest that agentic self-evolution can optimize a complex system end-to-end. The evaluation also has to evolve with the engine, and letting agents evolve it is a natural next step.

Figures & tables

Figure 1: Qwen3-0.6B on H100. (a) Natural-request throughput: median and range of three repetitions; the factor is relative to vanilla. (b) Latency means and central 50%/80% intervals over 100 requests. (c) Observed accuracy on GSM8K (660 items) and retrieval (270); dotted lines mark vanilla. Production settings are selected on development data (Appendix E).

Figure 2: The SEIS protocol. (a) A fixed agent chooses and implements optimizations to an inherited engine. Development gates determine which code is retained; passing candidates may be slower. Code and reports carry to the next session. Post-validation evaluates task accuracy and natural-length speed outside optimization feedback. (b) Branching: sessions start independently from the same engine

E_{0}

, and the best result is kept. (c) Branching Chain: each session inherits the code and reports of the previous session in its chain. (d) Sync Branching Chain: every

N

sessions, all chains exchange code and reports, and each agent decides what to adopt. Nodes represent sessions.

Figure 3: Performance and execution changes in the best engine. (a) Matched workloads: median throughput ratios over three repetitions; whiskers show run ranges with the vanilla median fixed. Percentages indicate the additional throughput from speculation. (b,c) Vanilla and the full evolved engine on a separate CUDA profile with 128 forced output tokens, including prefill: summed kernel time and kernel invocations per output token. Fused operations belong to their encompassing kernel group; these groups are not independent speedup attributions. Overlapping kernel intervals are counted separately, so their sum is not wall-clock time.

Figure 4: Organization of the evolved decode implementation. The policy selects single-token decoding or verification over

M

token positions. The runtime replays CUDA graphs, verifies drafts, emits tokens, and manages KV state; attention kernels read and update the cache. Appendix F shows the detailed execution paths.

		Opus				GPT
condition	chain	dev.	held-out	GSM8K $\Delta$	retrieval $\Delta$	dev.	held-out	GSM8K $\Delta$	retrieval $\Delta$
Chain	A	4.94 †	6.11	$-$ 0.15	$-$ 1.48	2.43	2.37	$-$ 1.36	$-$ 0.74
	B	2.88	2.71	+0.91	$-$ 28.89	1.84	1.82	$-$ 1.21	+1.11
	C	3.54	3.31	+0.91	+0.37	2.13	2.07	$-$ 1.36	$-$ 2.22
Sync-3	A	3.38	3.12	+1.52	$-$ 0.37	2.96	2.76	+0.30	0.00
	B	3.39	3.15	+1.06	0.00	2.95	2.84	$-$ 0.15	+0.74

Table 1: Final engines of all 24 Qwen chains. Speedups are factors; accuracy differences are percentage points on the 660-item GSM8K test split and retrieval. Bold marks a slowdown or a difference outside

\pm

3 points;

\dagger

marks speculation. Passing these task tests does not exclude the functional limitations in Section 6 .

Figure 5: Branching Chain versus Branching in the Chain, Sync-3, and Sync-9 conditions. (a) Retained first-session outcomes; rejected candidates retain the 1.00

\times

vanilla engine. (b) Median retained development speedup across the nine chains per agent, with the range across chains. Dashed curves show expected best-of-k selection without replacement from that agent’s nine observed Branching sessions, for k

\leq

Figure 6: Synchronization in Opus Sync-9 (Sync Branching Chain,

N=9

). (a) Retained-engine development speedups. The filled diamond marks B’s build; open diamonds mark A/C’s adoption after the second sync. (b) Cumulative development-suite measurements from A’s session-19 report: policy and state repairs, sixteen-row verification with its teacher-forcing fix, then output handling, launch bindings, and policy-cost retuning. The dashed reference measures the inherited engine with speculation off. Each checkpoint includes multiple implementation changes.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Opus
condition	chain	development	held-out	GSM8K $\Delta$	retrieval $\Delta$
Chain	A	4.94	6.11	$-$ 0.15	$-$ 1.48
Chain	B	2.88	2.71	+0.91	$-$ 28.89 !
Chain	C	3.54	3.31	+0.91	+0.37
Sync-3	A	3.38	3.12	+1.52	$-$ 0.37
Sync-3	B	3.39	3.15	+1.06	0.00

Appendix

Table 2: All Qwen endpoints: the last development-gate-passing engine of each chain. Speedups are factors relative to vanilla. Accuracy differences are percentage points on the 660-item GSM8K test split and retrieval; ! marks an observed difference outside

\pm

3 points.

Figure 7: Retained development speedup along every Qwen chain, with one common vertical scale. Chain colors and diamond markers match Figure 6 : filled diamonds indicate speculative-decoder builds and open diamonds adoptions. Crosses show rejected candidates at their measured candidate scores; rejected or unscored sessions leave the retained state unchanged. Vertical dashed lines mark syncs.

reference	verify rows	positions	mean KL	top-1 (%)	PPL ratio
vanilla	8	112	0.001605	98.21	1.0048
vanilla	16	352	0.001172	98.86	1.0001
optimized ordinary	8	112	0.000354	100.00	1.0003
optimized ordinary	16	352	0.000280	100.00	1.0007

Appendix

Table 3: Matched-prefix checks of the best engine’s actual verification paths. Metrics are pooled over the indicated positions. Reference paths decode one token at a time.

condition	passing	rejected	unscored	median minutes
Opus
Chain	60	0	0	109.2
Sync-3	57	2	1	83.2
Sync-9	58	0	2	115.0
Chain-128k	58	0	2	120.0
GPT

Appendix

Table 4: Main Qwen session records. Each condition has sixty indexed sessions. Passing and rejected counts refer to development gates; an unscored session contributes no candidate verdict. Durations describe recorded session work, not a total monetary-cost comparison.

Figure 8: Descriptive evolution statistics. (a) Cumulative distributions of all changes between consecutive passing engines within each chain. (b) Changes after sync and in other eligible sessions; horizontal marks are medians. Change axes are symmetric-log, linear within

\pm

0.05. (c) Three endpoints per large-window condition with min–max spans; open markers indicate observed task-accuracy differences outside

\pm

3 points. Sessions and synchronized chains are dependent observations.

engine	natural tokens/s [range]	speedup	fixed-256 tokens/s [range]	speedup
vanilla	594 [583–594]	1.00 $\times$	599 [599–599]	1.00 $\times$
evolved, speculation off	1,831 [1,725–1,837]	3.08 $\times$	1,866 [1,846–1,876]	3.12 $\times$
evolved, speculation on	1,942 [1,833–1,943]	3.27 $\times$	2,317 [2,317–2,318]	3.87 $\times$

Appendix

Table 5: Matched ablation on 100 GSM8K prompts. Throughput is the median of three repetitions; brackets show the minimum and maximum, not a confidence interval. Each speedup divides the median throughput by the vanilla median under the same length rule.

engine	output tokens	mean length	mean latency (ms)	median latency (ms)
vanilla	17,541	175.41	295.47	257.55
evolved, speculation off	17,614	176.14	96.19	90.54
evolved, speculation on	17,319	173.19	89.20	86.75

Appendix

Table 6: Natural-request output lengths and latency on the matched 100-request workload. Outputs are identical across repetitions within each configuration. Latency statistics use the repetition with median aggregate throughput; configurations generate their own answers.

Figure 9: Workload dependence. Points report geometric-mean speedup over recorded shapes with the indicated length. (a) Opus Sync-9 A before speculation and at its endpoint; the checkpoints differ in several implementation changes. (b) GPT Chain-128k B loses performance when longer inputs exceed its CUDA-graph coverage and trigger eager execution. The dashed line is vanilla throughput; the shaded region indicates a slowdown. Each length group contains its observed combinations of prompt types and other sequence lengths.

framework	selected settings	GSM8K $\Delta$	retrieval $\Delta$
vLLM	BF16; no speculation	+0.61	$-$ 0.74
SGLang	BF16; FlashInfer; no speculation	+0.30	$-$ 0.37
TensorRT-LLM	BF16; PyTorch backend; n-gram 5	+0.76	$-$ 1.11

Appendix

Table 7: Selected production configurations and final accuracy differences in percentage points relative to vanilla. Decode CUDA graphs are enabled. SGLang uses eager prefill; TensorRT-LLM n-gram decoding uses a private per-request pool and disables overlap scheduling.

Figure 10: Execution paths of the best engine. Teal denotes ordinary single-token decoding; coral denotes multi-token verification. Rows represent token positions within one request. Both paths use the optimized model computation and shared KV storage. Dashed arrows denote KV reservation and commitment; accepted tokens return through the existing generation interface. The schematic groups repeated Transformer operations.

condition	chain	development	omitted output rows	EOS-dependent execution	other observed changes
Chain	A	2.43 $\times$	59,872	—	FP8 KV on four layers
Chain	B	1.84 $\times$	51,910	deferred multi-step execution	—
Chain	C	2.13 $\times$	—	—	skips layer 23’s attention at selected context lengths
Sync-3	A	2.96 $\times$	19,840	—	fitted RMS denominator
Sync-3	B	2.95 $\times$	53,072	—	—
Sync-3	C	2.89 $\times$	53,072	—	—

Appendix

Table 8: Changes in GPT endpoints that restrict computation or depend on evaluation mode. Development speedups describe the complete endpoints and are not speedups attributable to the individual listed change.

	Opus				GPT
chain	dev.	held-out	GSM8K $\Delta$	retrieval $\Delta$	dev.	held-out	GSM8K $\Delta$	retrieval $\Delta$
A	5.66 †	7.02	$-$ 1.36	0.00	2.80	2.64	$-$ 3.48	$-$ 11.11
B	5.82 †	6.53	$-$ 1.67	0.00	2.53	2.47	$-$ 2.88	$-$ 0.74
C	2.63	2.59	$-$ 1.06	$-$ 0.74	2.50	2.21	$-$ 0.91	$-$ 11.11

Appendix

Table 9: Llama Chain endpoints, all at session 20. Speedups are factors relative to the Llama vanilla engine; accuracy differences are percentage points on the 660-item GSM8K test split and retrieval. Bold marks a difference outside

\pm

3 points;

\dagger

marks speculation.

engine	session	tokens/s	speedup over vanilla
vanilla mini-sglang	—	629	1.00 $\times$
Opus A	20	1,820	2.90 $\times$
Opus B	20	1,966	3.13 $\times$
Opus C	20	1,657	2.64 $\times$
GPT A	6	1,333	2.12 $\times$

Appendix

Table 10: Archived natural-length Llama throughput on 100 GSM8K requests, with EOS enabled and prefill included. Factors use unrounded throughput. The GPT entry is an earlier checkpoint; its three endpoints were not timed under this protocol.

Efficient Language Model Inference

Efficient language model inference reduces the compute, memory, latency, and energy needed to generate useful outputs.

See all 321 papers

RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems
Sep 11, 2026Ziyue Yang, Yuting Jiang, Lei Qu +1LLM Inference Large Language Model-Guided Optimization
Shanghai Xingyunzhili Artificial Intelligence Institute, Shanghai, China · Microsoft Research, Redmond, WA, USA
Compute Where it Counts: Self Optimizing Language Models
May 11, 2026Yash Akhauri, Mohamed S. AbdelfattahLLM Inference Efficiency RL for Language Models
Cornell University
Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference
Sep 13, 2026Tian JinAutoregressive Decoding Efficient Language Model Reasoning
Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY

Sep 11, 2026J/K move · Enter open · S save

RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems

Ziyue Yang, Yuting Jiang, Lei Qu, Peng Cheng

Shanghai Xingyunzhili Artificial Intelligence Institute, Shanghai, China · Microsoft Research, Redmond, WA, USA

LLM Inference Large Language Model-Guided Optimization LLM Inference Acceleration

Open paper AskPDF arXiv

AI is beginning to make substantive contributions to LLM inference optimization. Existing AI optimizations are predominantly profiling-based. Profiling-bound feedback confines the search to the capabilities and performance of an existing software stack, preventing a fundamentally better architecture of LLM inference systems from being identified. To enable the AI-driven LLM inference system architecting loop, we argue that a general workload representation, a verifiable mutation space, and an implementation-independent evaluator are required. We present the RoofLang domain-specific language (DSL) that provides these features. In our evaluation, RoofLang reveals that DeepSeek V4-series models could achieve 3.5-39.5

\times

higher peak decode throughput than other representative models. This gap is disproportionate to their total parameter counts and arises largely from compact KV-cache designs that support larger batches and reduce memory traffic. A persistent optimizer agent further discovered several new architectures that improved both throughput and interactivity of DeepSeek V4 Pro on NVIDIA B300 by 6.23-50.1%.

Abstract

Efficient Language Model Inference

Explore similar work

RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems

Compute Where it Counts: Self Optimizing Language Models

Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference