On-device inference imposes constraints on model storage and operand traffic and thus benefits from compressed operand representations, which mandate the use of encoding and decoding operators. However, naïve placement of codec operators can cancel the operand traffic gains sought from compression. In this work, I present an algorithm to place codec operators into a source computational graph to make it compatible with target hardware. The edges of the graph are colored so that each induced subgraph is weakly connected and has a maximal node that both postdominates all subgraph inputs and dominates all subgraph outputs; this unique node - dubbed the absolute dominator - is the natural location to observe the traffic flowing through the subgraph and learn parametric codec operators. The algorithm's time complexity is linear in the source graph's size. With respect to naïve local codec placement, the algorithm reduces modeled operand traffic by 24%-44% across four models, including 24% on SigLIP and 34% on SmolLM2.
Figures & tables
Figure 1: (a) Detail of a source computational graph: two upstream reshape operators (R) feed a concatenate operator (C), and the rightmost one feeds a downstream reshape operator (R). (b) The algorithm’s first step inserts input codec markers (light gray) and output codec markers (dark gray) around each operator. (c) The second step assigns each edge a semantics, partitioning the graph into weakly connected subgraphs (edges sharing a color and dash pattern); the third step then identifies each subgraph’s absolute dominator (colored node). (d) The fourth step merges the leftmost upstream reshape subgraph into the concatenate subgraph; the rightmost upstream reshape subgraph fans out to two subgraphs, and thus cannot be merged without breaking the dominance property.
Z
Y
X
W
ShuffleNetV2
8.8
2.3
2.3
2.3
U-Net
93.3
23.4
23.4
23.4
SigLIP
329.7
85.0
85.0
85.0
SmolLM2
405.1
101.4
101.4
101.4
Table 1: Model storage (in MiB) across the four compression policies.
Z
Y
X
W
ShuffleNetV2
54.8
57.9 ↑5.7%
58.0 ↑5.8%
39.4 ↓28.1%
U-Net
920.4
1039.9 ↑13.0%
1095.9 ↑19.1%
759.6 ↓17.5%
SigLIP
1802.7
2029.8 ↑12.6%
2047.8 ↑13.6%
1768.0 ↓1.9%
SmolLM2
741.1
546.3 ↓26.3%
541.8 ↓26.9%
427.2 ↓42.4%
Table 2: Model traffic (in MiB) across the four compression policies. Arrows report changes relative to Z, rounded to one decimal place. In the absence of hardware support for traffic optimization (e.g., fusing the accumulation and decompression of matrix-matrix multiplication outputs), less aggressive compression policies (Y, X) can increase traffic; only policy W realizes savings on every model.
Z
Y
X
W
ShuffleNetV2
0.0
4.2
6.3
4.3
U-Net
0.0
90.4
138.9
84.0
SigLIP
0.0
209.2
228.1
220.2
SmolLM2
0.0
47.9
50.0
47.1
Table 3: Number of codec items (in M) across the four compression policies. On graphs where chains of uncompressed operators (activations, normalizations) are interleaved with isolated data layout operators (reshape, view), policies prescribing that data layout operators use compressed operands to save traffic can increase the number of codec items with respect to policies prescribing that data layout operators use uncompressed operands.
Local
Non-local (w/o)
Non-local (w/)
ShuffleNetV2
70.6
39.4 ↓44.2%
39.4 ↓44.2%
U-Net
1251.1
767.1 ↓38.7%
759.6 ↓39.3%
SigLIP
2311.8
1768.0 ↓23.5%
1768.0 ↓23.5%
SmolLM2
643.1
427.2 ↓33.6%
427.2 ↓33.6%
Table 4: Model traffic (in MiB) across three codec placement algorithms with fixed weight compression. Arrows report changes relative to local placement, rounded to one decimal place. The local baseline places codecs at the boundaries of every operator that requires encoded operands, independently for each operator; this policy does not track cross-boundary relationships that reveal opportunities for codec parameter sharing, and thus introduces redundant decode-encode pairs. The non-local algorithms apply policy W without and with coalescence; coalescence delivers savings for U-Net, which contains several routing and data layout operators.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Operator
Port blocks
FP32
INT8
aten._native_batch_norm_legit_no_training.default
∅
Z
aten.add.Tensor
∅
Z
aten.cat.default
{({tensors[∗]},{out})}
Z
X
aten.clone.default
{({self},{out})}
Z
W
aten.constant_pad_nd.default
{({self},{out})}
Z
W
aten.convolution.default
∅
Z
Y
Appendix
Table 5: Commutation table C used for evaluation. Port blocks (i.e., pairs of input- and output-port sets that must share the same semantics) are shown for each operator: ∅ indicates that all port blocks are trivial (i.e., they contain only pairs of a singleton and an empty set); [*] denotes a variadic port. Letters Z , Y , X , and W indicate the first policy that selects each operator variant.
Model
Repository
License
Shape
ShuffleNetV2
pytorch/vision
BSD 3-Clause
(1,3,224,224)
U-Net
qubvel-org/segmentation_models.pytorch
MIT
(1,3,512,512)
SigLIP
HuggingFaceTB/SmolVLM2-256M-Video-Instruct
Apache 2.0
(1,3,512,512)
SmolLM2
HuggingFaceTB/SmolVLM2-256M-Video-Instruct
Apache 2.0
(1,128,576)
Appendix
Table 6: Model population used for evaluation. Models were traced to torch.fx.GraphModule objects using the torch.export.export function; rotary positional embedding (RoPE) and root mean square normalization (RMSNorm) subgraphs were pattern-matched and replaced with standalone operators before hardware adaptation was executed.
Institute of Advanced Technology University of Science and Technology of China Hefei, China · Supercomputing Center University of Science and Technology of China Hefei, China