HYDRA: A Heterogeneous Chiplet DSE Framework for Serving Dynamic Hybrid LLM Workloads

Jiahao Lin; Alish Kanani; Sangwan Lee; Jaehyun Park; Umit Ogras

cs.ARAug 19, 2026

HYDRA: A Heterogeneous Chiplet DSE Framework for Serving Dynamic Hybrid LLM Workloads

Authors: Jiahao Lin, Alish Kanani, Sangwan Lee, Jaehyun Park, Umit Ogras

Organizations: University of Wisconsin–Madison · University of Ulsan

Abstract

Hybrid Transformer-Mamba large language models (LLMs) enhance long-context efficiency, but their heterogeneous computation and communication patterns complicate efficient hardware acceleration. Chiplet-based architectures offer a scalable solution by integrating specialized compute and memory units. However, the design space spanning static architectural configurations and dynamic runtime policies is prohibitively large to explore exhaustively. To address this challenge, we present HYDRA, a comprehensive design space exploration framework for hybrid LLM serving on heterogeneous chiplet systems. HYDRA jointly explores chiplet composition, placement, inter-chiplet bandwidth provisioning, dynamic batching, and runtime scheduling. It integrates communication-aware placement, dynamic batching, elastic task scheduling, and a fast Markov-based performance estimator that captures multi-tenant runtime dynamics for efficient and accurate exploration. Across all workloads, HYDRA delivers 1.55x the throughput and 43.7 percent lower time-to-first-token on average, with throughput gains reaching up to 2.3x compared to state-of-the-art baselines. These results highlight that co-designing architecture and runtime policies is critical for efficient large-scale LLM serving on heterogeneous chiplet systems.

Figures & tables

Fig. 1: (a) Design space of the chiplet-based system’s macroarchitecture and corresponding simulation cost (b) Runtime and design-time scheduling strategies for LLM services.

Framework	Workload	Hetero.	Scale	Target DS	Runtime Dyn.
Gemini	CNN/MLP	$\times$	8 chiplets	Microarch.	$\times$
Cascade	Edge App.	$\checkmark$	12 chiplets	Macroarch.	$\times$
WSC-LLM	Transformer LLM	$\times$	64 chiplets	Microarch.	$\times$
HYDRA	Hybrid LLM	$\boldsymbol{\checkmark}$	24 chiplets	Macroarch.	$\boldsymbol{\checkmark}$

TABLE I: Comparison of prior chiplet-based DSE frameworks across workload support, heterogeneity, scalability, target design space, and runtime-dynamics coverage.

Fig. 2: The overview of the proposed design space exploration framework targeting hybrid LLM services.

Fig. 3: The area breakdown of compute chiplets in different NoI bandwidth. Cores and SFUs are compute units for linear and non-linear operations. SRAM represents local buffers. DMA, PHY, and bumps represent communication components.

Notation	Definition
$k\in\mathcal{K}$	Schedulable task instance ( $k$ ) and set of tasks ( $\mathcal{K}$ ).
$A_{p},A_{d}$	Prefill-/decode-optimized ( $A_{p}$ / $A_{d}$ ) Attention accelerators
$M_{p},M_{d}$	Prefill-/decode-optimized ( $M_{p}$ / $M_{d}$ ) Mamba accelerators
$C_{A}$	The set of Attention accelerator chiplets
$C_{M}$	The set of Mamba accelerator chiplets
$Loc_{M},Loc_{A}$	The set of locations of HBM chiplets allocated to the Mamba blocks ( $Loc_{M}$ ) or Attention blocks ( $Loc_{A}$ )

TABLE II: Summary of notations used in this work

Fig. 4: The overview of the chiplet placement in HYDRA

Fig. 5: The overview of the dynamic batching in HYDRA

Fig. 6: The overview of the task scheduling in HYDRA

Fig. 7: Overview of the Markov-based performance estimation.

Fig. 8: Overview of the DSE on 12 workloads (model-dataset pairs). The red points are the Pareto configurations derived through the exhaustive search of macro-architectures. The blue points are the Pareto configurations estimated by the Markov-based method. $B^{\star}$ : Configurations achieving the highest throughput-TTFT ratio; $M_{T}$ : Configurations achieving the best throughput.

Chiplet Counts	LLAMA3	MAMBA2	Nemotron-H
Total	$24$
$N_{Ap}$ (Prefill)	$1$ – $21$	–	$1$ – $19$
$N_{Ad}$ (Decode)	$1$ – $21$	–	$1$ – $19$
$N_{Mp}$ (Prefill)	–	$1$ – $21$	$1$ – $19$
$N_{Md}$ (Decode)	–	$1$ – $21$	$1$ – $19$
$N_{HBM3}$	$2$ – $16$

TABLE III: Design space exploration parameters. Some configurations are shared across all models.

Fig. 9: Ablation study of communication-aware placement (CP, Section IV-A ), elastic task scheduling (ET, Section IV-C ), and dynamic batching (DB, Section IV-B ) strategies. It reports results for 12 model-dataset pairs on the x-axis for the

B^{\star}

and

M_{T}

	Most-specialized		Medium-specialized
	General	red	off-target evaluation

TABLE IV: Specialization–generality tradeoff across hybrid LLMs. Each row is the DSE target model set, and each column is the evaluation workload. Values are normalized to the workload-specific single-model optimum. (a) reports

B^{\star}

with normalized TP/TTFT; (b) reports

M_{T}

with normalized TP.

Fig. 10: Comparison of chiplet placement strategies across Nemotron-H workloads.

Fig. 11: NoI heatmaps of bandwidth utilization that compare the round-robin (a) and communication-aware placement (b).

Fig. 12: Comparison of task schedulers across Nemotron-H workloads.

Fig. 13: Comparison of compute-chiplet utilization across Nemotron-H workloads.

Model	Exhaustive	Roofline [ 33 ]	HYDRA
LLAMA3-7B	2 d. 8 h.	4 min	4 min
MAMBA2-2.8B	3 d. 1 h.	4 min	4 min
Nemotron-H-4B	8 d. 8 h.	15 min	15 min
Max Throughput	100%	74%	93%

TABLE V: Average exploration time for different search strategies and maximum-throughput recovery. Each exploration covers four datasets per model.

Hybrid Mamba-Transformer Architectures

See all 19 papers

Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical--Machine-Learning Predictors
Aug 7, 2026Saeid Shokoufa, Mohammad Erfan Sadeghi, Mehdi Kamal +1LLM Inference Energy-Aware LLM Inference
University of Southern California Los Angeles, California, USA
LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure
Feb 26, 2026Jaehong Cho, Hyunmin Choi, Guseul Heo +1LLM Serving Disaggregated LLM Serving
School of Computing KAIST Daejeon, South Korea
Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM
Oct 7, 2025Tianhao Zhu, Dahu Feng, Erhu Feng +1AI Accelerator Inference LLM Serving
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · Department of Precision Instrument, Tsinghua University

Aug 7, 2026J/K move · Enter open · S save

Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical--Machine-Learning Predictors

Saeid Shokoufa, Mohammad Erfan Sadeghi, Mehdi Kamal, Massoud Pedram

University of Southern California Los Angeles, California, USA

LLM Inference Energy-Aware LLM Inference

Open paper AskPDF arXiv

The rapid scaling of Large Language Models (LLMs) has significantly increased computational cost, energy consumption, and inference latency, making accurate estimation essential for sustainable artificial intelligence deployment and hardware-aware design. In this work, we introduce Hybrid Modeling for Energy and Latency of LLMs (HYMELL), a hybrid three-level framework for estimating LLM inference latency and energy by combining analytical modeling with machine learning (ML). HYMELL models LLM execution through a three-level hierarchy: analytical estimation of primitive operations, ML prediction of higher-level components, and an end-to-end model that captures system-level overheads across both prefill and decode phases. The framework supports diverse architectures, including dense and mixture-of-experts (MoE) feed-forward networks (FFNs), as well as multi-head attention (MHA) and grouped-query attention (GQA) mechanisms. Evaluated on an NVIDIA H100 graphics processing unit (GPU), HYMELL achieves high predictive accuracy; notably, for LLaMA 3 8B, it attains less than 5% error for both prefill and decode phases. By predicting execution costs directly from architectural parameters, it enables fast, hardware-free design space exploration and energy-efficient optimization.

Abstract

Hybrid Mamba-Transformer Architectures

Explore similar work

Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical--Machine-Learning Predictors

LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure

Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM