Hibana: Fast Agentic Inference on Unified-Memory Devices
Authors: Yahya Emara, Chi-Chih Chang, Anthony Fei, Ahmed F. AbouElhamayed, Chong-Yan Chen, Hung-Yueh Chiang, Kai-Chiang Wu, Mohamed S. Abdelfattah
Organizations: Cornell University · Department of Computer Science, University of Illinois Urbana-Champaign · Independent Researcher · National Yang Ming Chiao Tung University
Local AI promises capable personal assistants that run on users' own devices, keeping data and computation under their control. Open-weight models that fit on local devices can now handle practical tasks such as coding. Yet current inference engines fall short. Popular engines such as vLLM and SGLang target datacenter GPUs with dedicated host memory, while local engines underuse the memory hierarchy of unified-memory devices. We present Hibana, an inference engine for unified-memory devices that co-optimizes model representation, memory management, and decoding strategy. Model surgery, the process of editing a trained model's architecture to suit its hardware, changes how the model computes, shrinking its memory footprint and data movement. Unified-memory management preserves each agent's state across turns, so long-running agents can resume instantly instead of recomputing their state. Adaptive speculation generates several tokens per step. Hibana codesigns these features around the shared memory capacity and limited bandwidth of unified-memory systems-on-chip to support local multi-agent workloads. On a single DGX Spark serving Qwen and Nemotron models of up to 125B parameters, Hibana completes up to 3.4× more agentic tasks per hour and delivers up to 3.9× higher decoding throughput than other established frameworks such as vLLM, NVIDIA Flare, llama.cpp, and TensorFold. Code is available at https://github.com/sharar-project/hibana.
Figures & tables
Figure 1: Open-weight models’ capability and serving memory. Left Qwen3.8-Flash-Next scores above GPT-5.5 and fits on one DGX Spark. ( Right. ) The smallest model at GPT-5.5’s level fits on one DGX Spark ∼ 4 months after GPT-5.5 was released.
Figure 2: Hibana serving Qwen3.8-Flash-Next on one DGX Spark. (Top.) Hibana’s model surgery, unified-memory management, and adaptive speculation (Section 3 ). (Bottom.) (a) decode speed for a single user, (b) peak decode throughput at each engine’s best batch size, and (c) agentic tasks completed per hour with 16 agents. Speedups are over the strongest baseline.
Figure 3: Expert selection in one MoE layer for six tokens with k=2 . Bars show each expert’s routing mass summed over the tokens that rank it in their top two. (a) Standard routing reads eight expert matrices. (b) Hibana guarantees each token’s top-1 expert (step 1), fills the rest of a budget of four with the expert carrying the most routing mass (step 2), and reads four expert matrices.
Tokens/s
Accuracy
Selection
g
16 req.
64 req.
(pp)
Standard
10
330
702
–
No cap
5
412
765
−0.3
Cap 64
3
488
818
−0.5
Cap 64
2
550
865
−0.8
Cap 64
1
597
984
−1.8
Table 1: Expert-sharing ablation on Qwen3.8-Flash-Next. g is the number of each token’s top experts always kept under a cap of 64 experts per layer. The first row is standard top-10 routing, and the second keeps each token’s top five. Accuracy is the paired change on GSM8K, HumanEval, GPQA, and RULER v2 32K relative to the first row. Hibana uses g=2 at these batch sizes.
Figure 4: Agreement with the FP8 head’s token by the FP8 head’s entropy, over 24,460 draft steps of Qwen3.8-Flash-Next. Legend values are overall agreement.
Tokens
Three-bit (ms)
CUTLASS NVFP4 (ms)
Speedup
512
4.90
6.68
1.36×
1k
5.53
7.22
1.30×
2k
6.84
8.24
1.20×
4k
9.90
10.53
1.06×
Table 2: Latency of one Flash-Next MoE layer, Hibana’s three-bit kernel vs. FlashInfer’s CUTLASS NVFP4 fused-MoE kernel.
Figure 5: KV-cache tiers on separate versus unified memory. (a) On a server or PC, host DRAM is separate memory, so a host tier that keeps a copy of the GPU’s KV adds capacity. (b) On GB10 the CPU and GPU share one memory. Hibana keeps no host copy and moves KV between the GPU’s KV pool and the SSD through about 1 GiB of staging, the same at any context length: restores stream through a double-buffered read buffer, and backups wait in a write queue. Only the staging slice is drawn to scale.
Policy
Requests
Speedup
Adaptive drafting
1
1.16×
4
1.05×
Adaptive verification
8
1.13×
16
1.28×
Table 3: Decode speedup of adaptive drafting over a fixed draft length (MTP on Qwen3.8-Flash-Next) and of adaptive verification over verifying the full draft (DFlash2 on Qwen3.8-27B).
Figure 6: (a) SPEED-Bench throughput on one DGX Spark as concurrency grows. Each point is one concurrency level, labeled with its number of requests. Up and to the right is better. (b) Agentic task throughput on AgentPerf-Local replay as the number of concurrent agents grows. Missing points mark configurations the baseline could not run.
Figure 7: Qwen3.8-Flash-Next serving 192 agent conversations whose combined context reaches twice GPU KV capacity. Top: live session context, the tokens of all unfinished conversations. Middle: share of each prompt served from cache. Bottom: median TTFT, log scale. 32 agents send turns back to back, with thinking off and no tool time. Curves are 200-turn rolling medians; the hatched region is warm-up. Baseline runs were stopped after about 3.3k turns.
Figure 8: Decode throughput on Qwen3.8-Flash-Next at one request as each optimization is added. Percentages are the change from the previous step. The baseline runs NVFP4 experts without speculation. The last step adds an NVFP4 KV cache.
GPQA
LCB
IFBench
AIME25
AIME26
Exact
90.7
87.7
85.3
100.0
100.0
+ 3-bit weights
90.4 ( − 0.3)
85.4 ( − 2.3)
83.3 ( − 2.0)
99.2 ( − 0.8)
99.2 ( − 0.8)
+ Expert sharing
90.4 ( − 0.3)
85.4 ( − 2.3)
83.3 ( − 2.0)
99.2 ( − 0.8)
99.2 ( − 0.8)
+ Relaxed verif.
89.1 ( − 1.6)
84.6 ( − 3.1)
81.7 ( − 3.6)
99.2 ( − 0.8)
99.2 ( − 0.8)
Table 4: Task accuracy (%) on Qwen3.8-Flash-Next as each lossy optimization is added. The last row is the full fast recipe. Parentheses give the change from the exact baseline.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Accuracy (%)
Accepted tokens per verify step
Acceptance
GPQA
LCB v6
IFBench
AIME26
Avg. †
GPQA
LCB v6
IFBench
AIME26
Capped ‡
Exact
87.4
83.7
82.0
99.6
84.4
2.25
2.15
2.34
2.30
7
Relaxed, s=0.1
89.7
84.6
85.7
100.0
86.7
2.40
2.32
2.55
2.43
6
Relaxed, s=0.25
88.9
84.3
84.8
99.2
86.0
2.60
2.49
2.78
2.63
7
Relaxed, s=0.5
89.1
84.6
81.7
99.2
85.1
2.86
2.76
3.35
2.89
26
Relaxed, s=1
87.7
85.7
76.3
97.5
83.2
3.25
3.19
3.72
3.21
55
Appendix
Table 5: Qwen3.8-Flash-Next with its MTP drafter (3 drafted tokens per step), three-bit weights (§ 3.1.3 ) and expert sharing, under each acceptance rule. Under relaxed verification, an eligible draft y (in the target’s top 2 and within 1.0 logit of its top token) is accepted with probability min(1,esp(y)) ; other drafts get the exact test. 248K-token cap, thinking on, T=1.0 , top- p 0.95, top- k 20
GPQA
LCB
IFBench
AIME25
AIME26
Avg.
Nemotron-3.5-Super
Exact
87.2
86.5
75.2
95.8
94.6
87.9
3-bit
86.7
82.6
71.3
94.2
95.8
86.1
Fast
87.7
79.7
70.2
97.1
94.6
85.8
Qwen3.8-Flash-Next
Exact
90.7
87.7
85.3
100.0
100.0
87.9
Appendix
Table 6: Task accuracy (%) at a 248k-token budget. Fast is the full fast profile with relaxed verification. Avg. is over all five tasks, except for Flash-Next, where it leaves out the saturated AIME sets.
Requests
Two Sparks vs. one
1
1.56×
4
1.59×
16
1.79×
32
1.80×
Appendix
Table 7: Decode throughput of Qwen3.8-Flash-Next on two DGX Sparks relative to one. At 16 and 32 requests, Hibana uses the fast profile’s optimizations with a larger batch limit and shorter drafts.
Figure 9: Where each engine resumes on one turn of a SWE-bench trace, drawn to scale.
Figure 10: Share of each prompt served from cache on every turn of four AgentPerf-Local tasks, Qwen3.8-Flash-Next, one agent. Gray: the prompt’s length. Top: traces as published, where each SWE-bench request ends with a turn note that the next turn deletes. Bottom: the same traces with the note removed. The first turn of each task is omitted. Flare runs its recipe defaults; Table 8 omits llama.cpp.
Served from cache (%)
Re-read per turn (tokens)
Task
Kind
Turns
vLLM
Hibana
TensorFold
Flare
vLLM
Hibana
TensorFold
Flare
django-14500
SWE-bench
40
44
95
0
26
6,018
473
10,718
7,893
django-15851
SWE-bench
28
36
93
0
56
5,267
499
8,210
3,595
sphinx-10614
SWE-bench
40
49
96
0
26
6,482
507
12,708
9,345
council-votes
OpenClaw
21
93
92
93
93
2,483
2,520
2,490
2,483
council-budget
OpenClaw
20
91
91
91
91
1,991
2,020
1,998
1,991
Appendix
Table 8: Context reuse per task, traces as published, Qwen3.8-Flash-Next, one agent. Re-read is the prompt tokens an engine recomputes per turn, excluding each task’s first turn.
School of Computer Science and Engineering, Sun Yat-sen University · Sun Yat-sen University School of Computer Science and Engineering Guangzhou, China · China Mobile Internet Company Ltd +1