Hibana: Fast Agentic Inference on Unified-Memory Devices
Authors: Yahya Emara, Chi-Chih Chang, Anthony Fei, Ahmed F. AbouElhamayed, Chong-Yan Chen, Hung-Yueh Chiang, Kai-Chiang Wu, Mohamed S. Abdelfattah
Organizations: Cornell University · Department of Computer Science, University of Illinois Urbana-Champaign · Independent Researcher · National Yang Ming Chiao Tung University
Local AI promises capable personal assistants that run on users' own devices, keeping data and computation under their control. Open-weight models that fit on local devices can now handle practical tasks such as coding. Yet current inference engines fall short. Popular engines such as vLLM and SGLang target datacenter GPUs with dedicated host memory, while local engines underuse the memory hierarchy of unified-memory devices. We present Hibana, an inference engine for unified-memory devices that co-optimizes model representation, memory management, and decoding strategy. Model surgery, the process of editing a trained model's architecture to suit its hardware, changes how the model computes, shrinking its memory footprint and data movement. Unified-memory management preserves each agent's state across turns, so long-running agents can resume instantly instead of recomputing their state. Adaptive speculation generates several tokens per step. Hibana codesigns these features around the shared memory capacity and limited bandwidth of unified-memory systems-on-chip to support local multi-agent workloads. On a single DGX Spark serving Qwen and Nemotron models of up to 125B parameters, Hibana completes up to 3.4× more agentic tasks per hour and delivers up to 3.9× higher decoding throughput than other established frameworks such as vLLM, NVIDIA Flare, llama.cpp, and TensorFold. Code is available at https://github.com/sharar-project/hibana.
Figures & tables
Figure 1: Open-weight models’ capability and serving memory. Left Qwen3.8-Flash-Next scores above GPT-5.5 and fits on one DGX Spark. ( Right. ) The smallest model at GPT-5.5’s level fits on one DGX Spark ∼ 4 months after GPT-5.5 was released.
Figure 2: Hibana serving Qwen3.8-Flash-Next on one DGX Spark. (Top.) Hibana’s model surgery, unified-memory management, and adaptive speculation (Section 3 ). (Bottom.) (a) decode speed for a single user, (b) peak decode throughput at each engine’s best batch size, and (c) agentic tasks completed per hour with 16 agents. Speedups are over the strongest baseline.
Figure 3: Expert selection in one MoE layer for six tokens with k=2 . Bars show each expert’s routing mass summed over the tokens that rank it in their top two. (a) Standard routing reads eight expert matrices. (b) Hibana guarantees each token’s top-1 expert (step 1), fills the rest of a budget of four with the expert carrying the most routing mass (step 2), and reads four expert matrices.
Tokens/s
Accuracy
Selection
g
16 req.
64 req.
(pp)
Standard
10
330
702
–
No cap
5
412
765
−0.3
Cap 64
3
488
818
−0.5
Cap 64
2
550
865
−0.8
Cap 64
1
597
984
−1.8
Table 1: Expert-sharing ablation on Qwen3.8-Flash-Next. g is the number of each token’s top experts always kept under a cap of 64 experts per layer. The first row is standard top-10 routing, and the second keeps each token’s top five. Accuracy is the paired change on GSM8K, HumanEval, GPQA, and RULER v2 32K relative to the first row. Hibana uses g=2 at these batch sizes.
Figure 4: Agreement with the FP8 head’s token by the FP8 head’s entropy, over 24,460 draft steps of Qwen3.8-Flash-Next. Legend values are overall agreement.
Tokens
Three-bit (ms)
CUTLASS NVFP4 (ms)
Speedup
512
4.90
6.68
1.36×
1k
5.53
7.22
1.30×
2k
6.84
8.24
1.20×
4k
9.90
10.53
1.06×
Table 2: Latency of one Flash-Next MoE layer, Hibana’s three-bit kernel vs. FlashInfer’s CUTLASS NVFP4 fused-MoE kernel.
Figure 5: KV-cache tiers on separate versus unified memory. (a) On a server or PC, host DRAM is separate memory, so a host tier that keeps a copy of the GPU’s KV adds capacity. (b) On GB10 the CPU and GPU share one memory. Hibana keeps no host copy and moves KV between the GPU’s KV pool and the SSD through about 1 GiB of staging, the same at any context length: restores stream through a double-buffered read buffer, and backups wait in a write queue. Only the staging slice is drawn to scale.
Policy
Requests
Speedup
Adaptive drafting
1
1.16×
4
1.05×
Adaptive verification
8
1.13×
16
1.28×
Table 3: Decode speedup of adaptive drafting over a fixed draft length (MTP on Qwen3.8-Flash-Next) and of adaptive verification over verifying the full draft (DFlash2 on Qwen3.8-27B).
Figure 6: (a) SPEED-Bench throughput on one DGX Spark as concurrency grows. Each point is one concurrency level, labeled with its number of requests. Up and to the right is better. (b) Agentic task throughput on AgentPerf-Local replay as the number of concurrent agents grows. Missing points mark configurations the baseline could not run.
Figure 7: Qwen3.8-Flash-Next serving 192 agent conversations whose combined context reaches twice GPU KV capacity. Top: live session context, the tokens of all unfinished conversations. Middle: share of each prompt served from cache. Bottom: median TTFT, log scale. 32 agents send turns back to back, with thinking off and no tool time. Curves are 200-turn rolling medians; the hatched region is warm-up. Baseline runs were stopped after about 3.3k turns.
Figure 8: Decode throughput on Qwen3.8-Flash-Next at one request as each optimization is added. Percentages are the change from the previous step. The baseline runs NVFP4 experts without speculation. The last step adds an NVFP4 KV cache.
GPQA
LCB
IFBench
AIME25
AIME26
Exact
90.7
87.7
85.3
100.0
100.0
+ 3-bit weights
90.4 ( − 0.3)
85.4 ( − 2.3)
83.3 ( − 2.0)
99.2 ( − 0.8)
99.2 ( − 0.8)
+ Expert sharing
90.4 ( − 0.3)
85.4 ( − 2.3)
83.3 ( − 2.0)
99.2 ( − 0.8)
99.2 ( − 0.8)
+ Relaxed verif.
89.1 ( − 1.6)
84.6 ( − 3.1)
81.7 ( − 3.6)
99.2 ( − 0.8)
99.2 ( − 0.8)
Table 4: Task accuracy (%) on Qwen3.8-Flash-Next as each lossy optimization is added. The last row is the full fast recipe. Parentheses give the change from the exact baseline.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Accuracy (%)
Accepted tokens per verify step
Acceptance
GPQA
LCB v6
IFBench
AIME26
Avg. †
GPQA
LCB v6
IFBench
AIME26
Capped ‡
Exact
87.4
83.7
82.0
99.6
84.4
2.25
2.15
2.34
2.30
7
Relaxed, s=0.1
89.7
84.6
85.7
100.0
86.7
2.40
2.32
2.55
2.43
6
Relaxed, s=0.25
88.9
84.3
84.8
99.2
86.0
2.60
2.49
2.78
2.63
7
Relaxed, s=0.5
89.1
84.6
81.7
99.2
85.1
2.86
2.76
3.35
2.89
26
Relaxed, s=1
87.7
85.7
76.3
97.5
83.2
3.25
3.19
3.72
3.21
55
Appendix
Table 5: Qwen3.8-Flash-Next with its MTP drafter (3 drafted tokens per step), three-bit weights (§ 3.1.3 ) and expert sharing, under each acceptance rule. Under relaxed verification, an eligible draft y (in the target’s top 2 and within 1.0 logit of its top token) is accepted with probability min(1,esp(y)) ; other drafts get the exact test. 248K-token cap, thinking on, T=1.0 , top- p 0.95, top- k 20
GPQA
LCB
IFBench
AIME25
AIME26
Avg.
Nemotron-3.5-Super
Exact
87.2
86.5
75.2
95.8
94.6
87.9
3-bit
86.7
82.6
71.3
94.2
95.8
86.1
Fast
87.7
79.7
70.2
97.1
94.6
85.8
Qwen3.8-Flash-Next
Exact
90.7
87.7
85.3
100.0
100.0
87.9
Appendix
Table 6: Task accuracy (%) at a 248k-token budget. Fast is the full fast profile with relaxed verification. Avg. is over all five tasks, except for Flash-Next, where it leaves out the saturated AIME sets.
Requests
Two Sparks vs. one
1
1.56×
4
1.59×
16
1.79×
32
1.80×
Appendix
Table 7: Decode throughput of Qwen3.8-Flash-Next on two DGX Sparks relative to one. At 16 and 32 requests, Hibana uses the fast profile’s optimizations with a larger batch limit and shorter drafts.
Figure 9: Where each engine resumes on one turn of a SWE-bench trace, drawn to scale.
Figure 10: Share of each prompt served from cache on every turn of four AgentPerf-Local tasks, Qwen3.8-Flash-Next, one agent. Gray: the prompt’s length. Top: traces as published, where each SWE-bench request ends with a turn note that the next turn deletes. Bottom: the same traces with the note removed. The first turn of each task is omitted. Flare runs its recipe defaults; Table 8 omits llama.cpp.
Served from cache (%)
Re-read per turn (tokens)
Task
Kind
Turns
vLLM
Hibana
TensorFold
Flare
vLLM
Hibana
TensorFold
Flare
django-14500
SWE-bench
40
44
95
0
26
6,018
473
10,718
7,893
django-15851
SWE-bench
28
36
93
0
56
5,267
499
8,210
3,595
sphinx-10614
SWE-bench
40
49
96
0
26
6,482
507
12,708
9,345
council-votes
OpenClaw
21
93
92
93
93
2,483
2,520
2,490
2,483
council-budget
OpenClaw
20
91
91
91
91
1,991
2,020
1,998
1,991
Appendix
Table 8: Context reuse per task, traces as published, Qwen3.8-Flash-Next, one agent. Re-read is the prompt tokens an engine recomputes per turn, excluding each task’s first turn.
Language-model agents increasingly run on workstations with a few consumer GPUs, where large mixture-of-experts (MoE) models keep part of their experts in host memory. With existing CPU-GPU engines, these agents get fast prefill or fast decode, not both: llama.cpp's expert cache skips forward passes of more than 32 tokens to keep the experts of decode cached, so an agent waits 2.1-2.8 s after each tool output, whereas its default layer placement shortens this wait but is 1.5-1.6× slower in decode. Our key insight is that a short prefill touches too many distinct experts to upload, yet sends 84% of its expert activations (routes) to a stable set of experts that fits in GPU memory. We present Ostinato, a CPU-GPU MoE inference engine that computes these experts on the GPUs in every forward pass without changing what the model computes. Short prefills split their routes, leaving fewer than one of a token's six routed experts per layer to the CPU; long prefills upload each distinct cold expert once; and each decode step runs as one fixed-shape CUDA graph per GPU. On real coding and customer-service agent sessions with DeepSeek-V4-Flash on four RTX 4090D GPUs, Ostinato is the fastest engine in both phases and finishes every session sooner than all six baselines. Against llama.cpp and KTransformers, it cuts the median wait after a tool output by 22-55% and finishes sessions 1.4-2.5× sooner.
Rapidly evolving LLM architectures and heterogeneous mobile hardware make supporting new models in on-device inference engines a recurring and costly system challenge. Each model may require coordinated changes across conversion, graph construction, operators, and hardware backends. Existing coding agents often miss these cross-layer dependencies and struggle to diagnose failures across host--device execution paths. We present EngineAgent, an agentic system for automating on-device LLM model enablement. Its key insight is to separate reusable engine and hardware knowledge from model-specific implementation. Before enablement, EngineAgent constructs a hierarchy-aware engine index that captures engine abstraction layers and cross-layer dependencies, and pre-validates backend-specific build, execution, and debugging workflows using already supported models. For each new model, a builder uses this knowledge to coordinate cross-layer changes, while an auditor checks implementation completeness and drives iterative debugging through device-side validation. Across 12 tasks spanning three mobile LLMs and four engine--backend pairs on CPUs, GPUs, and NPUs, EngineAgent achieves 75% end-to-end success with either of two underlying LLMs, versus 33.3% for the strongest coding-agent baseline. Success requires compilation, on-device execution, and output-logit agreement with reference implementations. EngineAgent also reduces average per-task time and model-API cost relative to baselines using the same underlying LLMs.
Zhengxiang Huang, Yujie Sun, Shengheng Chen +6
Shanghai Jiao Tong University Shanghai, China · Alibaba Group Hangzhou, China · Alibaba Group Hangzhou, Zhejiang, China
Emerging multi-agent LLMs demand privacy-preserving edge deployment, yet current inference systems struggle with these collaborative workflows. Specifically, the memory-bound decode phase causes severe bus contention on unified memory architectures (UMA), paralyzing naive CPU-GPU co-execution. Furthermore, speculative decoding in multi-agent workloads faces extreme variance in drafting difficulty, alternating between complex reasoning and predictable structured generation. Compounded by frequent tool-induced stalls, this highly fragmented execution severely underutilizes hardware and defeats traditional static batching. We present EdgeAgent, a cross-layer inference system explicitly co-designed for edge UMA and multi-agent workloads. At the micro-architectural level, it bypasses rigid graph-compiler constraints to enable zero-copy UMA-aware tensor parallelism, utilizing asymmetric memory layouts to fully saturate both CPU and GPU compute units. At the scheduling level, it dynamically allocates draft budgets based on real-time sequence predictability to bound bandwidth waste. Concurrently, an asynchronous suspend-and-yield mechanism actively evicts stalled agents, ensuring continuous hardware saturation during unpredictable tool invocations. Extensive evaluations on an Apple M4 SoC demonstrate that the UMA-aware execution alone contributes a 1.29x speedup over batched speculative decoding. Adding the agent-aware scheduling lifts the full EdgeAgent system to a 1.77x speedup under extreme tool-use latencies.
Yuhai Long, Yuanxin Wei, Kai Wu +3
School of Computer Science and Engineering, Sun Yat-sen University · Sun Yat-sen University School of Computer Science and Engineering Guangzhou, China · China Mobile Internet Company Ltd +1