We address a fundamental question: How far should kernel fusion extend and which boundaries are worth preserving? Unlike the general convention of using one monolith megakernel at lower batch sizes or supporting only a selective model component as megakernel and rest of the model as conventional graph, we show that the best configuration is a composition of multiple megakernels with no overlapping conventional graph component, or a fully conventional graph. We present a framework that selects megakernel regions in a streaming inference model from conventional-graph profiles and standalone kernel benchmarks, without running any candidate design end to end. It simulates each candidate's tile and event graphs to predict time to first output and selects the measured best design in all 13 combinations of model and batch size we tested. Across NH-TTS, Qwen3-TTS and Llama-1B, the best composition ranges from two to four megakernels to fully conventional execution and none is a single megakernel. We introduce four model categories (megakernel-stable/ receding/ advancing/ averse) that capture how a model's best megakernel composition changes with batch size. Using NH-TTS model on H100 NVL GPU, we demonstrate that preserving three specialized megakernels can improve both latency and throughput despite introducing additional kernel boundaries. At batch size 32, this design delivers 21.4% lower mean batch-p95 time to first audio (TTFA) and 28% higher throughput than the evaluated monolith megakernel. Against conventional TensorRT graph execution, it reduces engine-level mean batch TTFA p95 by 4.05x and increases throughput by 19.7%.
Figures & tables
Fig. 1: Six representative NH-TTS layouts from the 81-candidate design space. The three-megakernel design with the lowest measured first-publication latency at batch 32 on H100 NVL is highlighted.
Fig. 3: Best megakernel composition for NH-TTS at batch size 32
Feature
Modeled in
Algorithmic change
Cross-kernel readiness
Tile graph
Edges: drop the barrier edges between consecutive kernels; keep the tile-to-tile tensor edges
Weight prefetch
Tile graph
Scheduling rule: one staging slot per worker. A staged tile’s weight load is queued while its predecessors still run; its compute still waits for them
On-chip forwarding
Tile graph
Node weights: remove the forwarded tensor’s store and reload from the memory queue
Producer–consumer overlap
Event graph
Edges, capacity: the first chunk depends on step r instead of the preceding task, and runs in its own CTA allocation
Fewer launches, internal synchronization
Event graph
Nodes: a dispatch node before each launch, weighted by the per-launch cost; synchronization adds to tile weights
Smaller CTA budget
Service table
Tile scheduling uses W workers. For an opaque operator with a grid of g CTAs, compute time is approximated by scaling with ⌈g/W⌉/⌈g/132⌉ , assuming equal-cost CTA waves.
TABLE I: Feature modeling and algorithmic changes.
Stage 1 a
Megakernel
TensorRT
Megakernel
0.991ms
1.290ms
Stage 0 a
TensorRT
1.216ms
9.760ms
a Refer Figure 3
TABLE II: Megakernel vs TensorRT Latencies
Model
Conventional baseline
First publication
Batch sizes
NH-TTS
TensorRT-LLM with CUDA graphs, plus TensorRT engines
Time to First Audio (TTFA) at engine publication
1, 32, 64
Qwen3-TTS
PyTorch with vLLM FlashAttention-3, torch.compile and CUDA graphs
Time to First Audio (TTFA) on the host
1, 16, 32, 48, 64
a Llama 3.2 1B
PyTorch SDPA (FlashAttention) in one CUDA graph
Time to First Token (TTFT)
1, 2, 16, 32, 48
a 496-token prompt
TABLE III: The three models in our evaluation.
Fig. 4: Predicted selections within the implemented validation pools. Predicted and measured margins compare the winner with the nearest alternative in the same pool.
Large
Batch
Pays
Doesn’t
Pays
Megakernel-stable (NH-TTS; Llama-1B for first-token latency)
Megakernel-receding (Qwen3-TTS; Llama-1B for inter-token latency)
Small
Doesn’t
Megakernel-advancing a
Megakernel-averse a
a Predicted, not observed.
TABLE IV: Four model categories by where megakernels pay