cs.PLMar 25, 2026

DVM: A Bytecode Virtual Machine Approach for Dynamic Tensor Computation

Authors: Jingzhi Fang, Xiong Gao, Renwei Zhang, Zichun Ye, Lei Chen, Jie Zhao, Chengnuo Huang, Hui Xu, 

Organizations: Huawei, China · Hunan University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Dynamism is common in AI computation, e.g., the dynamic tensor shapes and the dynamic control flows in models. Due to the long compilation time, existing runtime compilation damages the model efficiency, while the offline compilers either suffer from the long compilation time and device memory footprint to cover all the possible execution instances of a dynamic model, or sacrifice optimization opportunities for usability. In this paper, we rethink the feasibility of runtime compilation for dynamic models and identify that the key for it to work is to speed up the compilation or hide the compilation overhead. To do this, we propose a real-time compiler, DVM. In DVM, we design a runtime operator compiler based on a bytecode virtual machine to perform effective and efficient compilation for each dynamic operator instance given its input. Specifically, instead of compiling programs into machine code, we encode the operator program into bytecode on the CPU and decode the bytecode into virtual instructions for direct execution on the NPU. Based on the runtime operator compiler, we further propose an operator fuser, which performs symbol-deduction-based fusion on static graphs and runtime fusion on dynamic graphs. Both pattern- and stacking-based fusion are supported to increase fusion opportunities. Evaluation on operators, subgraphs, and models shows that, compared with TorchInductor, PyTorch-eager and MindSpore-graph-O0, we are up to 11.77×\times better in terms of the operator/model efficiency and up to 5 orders of magnitude faster in terms of the maximum compilation time.

Figures & tables

Figure 1: DVM Overview.
Figure 2: Hardware architecture abstraction (the example AI Core abstraction is about A2/A3 Ascend NPU) [ 5 ] .
Figure 3: The asynchronous computation flows in an AI Core.
Figure 4: Traditional compilation VS virtual machine.
Figure 5: The workflow of the operator compiler.
CategoryInstructionSemantics
MemoryLoad (dst, src, tile_stride, tile_size)Load a continuous tile from global memory to local memory
ViewLoad(dst, src, tile_stride[], tile_size[], tile_dims)Load a non-contiguous tile from global memory to local memory
Store(dst, src, tile_stride, tile_size)Store a tile from local memory to continuous global memory
ViewStore(dst, src, tile_stride[], tile_size[], tile_dims)Store a tile from local memory to non-continuous global memory
ComputationCopy(xd, xn, size)Copy size bytes data from xn to xd in local memory
Broadcast(xd, xn, M, size, N)Broadcast xn of shape [M,1,N] to xd of shape [M, size, N]
Table 1: Tile-level Virtual Instructions
Figure 6: An example process of bytecode generation.
Figure 7: Bytecode Program Format.
Figure 8: Fusing two vector operations via local memory.
Figure 9: Stacking kernels spatially and temporally.
Figure 10: An example of operator fusion on dynamic graphs.
NameInput ShapeShape Range
matmul([m,k],[k,n])([m,k],[k,n])m∈[1,8192]m\in[1,8192] , (n,k)∈D={(4096,4096),(11008,4096),(4096,11008),(n,k)\in\mathcal{D}=\{(4096,4096),(11008,4096),(4096,11008),
  (5120,5120),(13696,5120),(5120,13696),(8192,8192),(5120,5120),(13696,5120),(5120,13696),(8192,8192),
  (28672,8192),(8192,28672)}(28672,8192),(8192,28672)\}
LayerNorm([b,s,h])([b,s,h])b∈[1,60]b\in[1,60] , s=8192s=8192 , h∈{1024,2048,3072,4096}h\in\{1024,2048,3072,4096\}
addmm([m,k],[k,n],[m,n])([m,k],[k,n],[m,n])m∈[1,8192]m\in[1,8192] , (n,k)∈D(n,k)\in\mathcal{D}
if-else-add([b,s,f])([b,s,f])b∈[1,256]b\in[1,256] , s∈[1,512]s\in[1,512] , f∈[1,8192]f\in[1,8192]
Table 2: Dynamic shape ranges ( D\mathcal{D} are shapes extracted from common LLMs)
Figure 11: matmul
Figure 12: LayerNorm
Figure 13: if-else-add
Figure 14: addmm
Figure 15: ML models
Figure 16: LLM
NameMethodAvg. SpeedupSpeedup RangeRatio of
    speedup (>1)%
matmulInductor-r1.191.190.88 - 2.1862
Inductor-d1.311.310.93 - 2.6293
PT-eager1.091.090.91 - 1.2793
LayerNormInductor-r1.091.091.01 - 1.45100
Inductor-d1.631.631.01 - 11.77100
Table 3: PT-DVM Running Time Speedups against the baselines.
NameMethodMax CTTotal CTTotal RTTotal CT /
     Total RT
matmulPT-DVM0.110.110.480.48123.26123.263.89\times10−33.89\text{\times}{10}^{-3}
Inductor-r115.88115.884339.334339.33125.92125.923.45\times1013.45\text{\times}{10}^{1}
Inductor-d200.99200.99354.53354.53133.79133.792.65\times1002.65\text{\times}{10}^{0}
LayerNormPT-DVM0.100.101.511.51271.14271.145.57\times10−35.57\text{\times}{10}^{-3}
Inductor-r37 802.3637\,802.361 517 938.561\,517\,938.56282.74282.745.37\times1035.37\text{\times}{10}^{3}
Table 4: Op & subgraph compilation time comparison (time in ms). CT: compilation time; RT: running time.
GroupMethodMMoEBERT
Compilation Time (ms)Inductor-r4.84×1054.84\times 10^{5} - 3.76×1063.76\times 10^{6}1.50×1051.50\times 10^{5} - 3.43×1053.43\times 10^{5}
 PT-DVM2.71×1012.71\times 10^{1} - 2.04×1022.04\times 10^{2}4.09×1044.09\times 10^{4} - 4.23×1044.23\times 10^{4}
Compilation/ Running TimeInductor-r1.28×1041.28\times 10^{4} - 1.70×1041.70\times 10^{4}3.00×1033.00\times 10^{3} - 9.24×1039.24\times 10^{3}
 PT-DVM7.97×10−17.97\times 10^{-1} - 8.20×10−18.20\times 10^{-1}1.11×1031.11\times 10^{3} - 1.28×1031.28\times 10^{3}
Table 5: Compilation time comparison for MMoE and BERT.
GroupMethodQwenLlama
Compilation Time (ms)MS-O02.66×1022.66\times 10^{2} - 8.41×1028.41\times 10^{2}1.18×1021.18\times 10^{2} - 1.29×1021.29\times 10^{2}
 MS-DVM2.78×1022.78\times 10^{2} - 3.02×1023.02\times 10^{2}1.27×1021.27\times 10^{2} , 1.28×1021.28\times 10^{2}
Compilation/ Running TimeMS-O01.15×10−11.15\times 10^{-1} - 2.23×10−12.23\times 10^{-1}2.41×10−22.41\times 10^{-2} - 7.55×10−27.55\times 10^{-2}
 MS-DVM2.81×10−12.81\times 10^{-1} - 4.45×10−24.45\times 10^{-2}2.55×10−22.55\times 10^{-2} - 9.20×10−29.20\times 10^{-2}
Table 6: Compilation time comparison for Qwen and Llama.
More in this topic

Efficient Inference

Efficient inference reduces the compute, memory, latency, or energy needed to produce accurate predictions, posterior estimates, and decisions.

See all 109 papers

Explore similar work

CardsList

  1. Forge-UGC: FX optimization and register-graph engine for universal graph compiler

    Apr 14, 2026Satyam Kumar, Saurabh JhaCompiler OptimizationAI Accelerator Inference

  2. VTC: DNN Compilation with Virtual Tensors for Data Movement Elimination

    Feb 11, 2026Muyan Hu, Ahan Gupta, Jiachen Yuan +7Efficient Neural Network InferenceMemory-Efficient Inference

    University of Illinois Urbana-Champaign · Georgia Institute of Technology · Microsoft

  3. Toward Compiler World Models: Learning Latent Dynamics for Efficient Tensor Program Search

    Jun 8, 2026Haolin Pan, Lianghong Huang, Xvlin Zhou +2GPU Kernel OptimizationLatent World Models

    Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China · Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China