Joint Quantization and Dynamic Partitioning for Context-Aware Edge Vision Transformers
Organizations: Walmart Sunnyvale, USA · Heidelberg University Heidelberg, Germany · Technische Universiteit Delft Delft, the Netherlands
Abstract
Deploying vision transformer models across connected and automated vehicles (CAVs) requires joint management of computation and Vehicle-to-Vehicle (V2V) communication. Static split-inference methods use fixed partition points and cannot adapt to time-varying link bandwidth, processing load, and concurrent multi-stream traffic. This paper proposes an edge orchestration framework that jointly selects model partition points, per-layer quantization bitwidths, and execution placement across heterogeneous vehicular compute nodes. The joint partitioning and mixed-precision allocation problem is formulated as a Mixed-Integer Linear Program (MILP) that minimizes compute and communication energy subject to end-to-end latency and task-quality constraints. The framework is evaluated on Jetson Orin Nano (8GB), Jetson Orin NX (16GB), and Jetson AGX Orin (32~GB) nodes using OPV2V-driven multi-stream workloads with . Across the evaluated configurations, the proposed formulation reduces V2V communication energy by -- relative to a Uniform INT8 split. At , end-to-end task completion latency is reduced by on the Orin Nano and on the Orin NX relative to the same baseline, while limiting the reduction in 3D detection accuracy to -- percentage points in relative to FP16 execution.
Figures & tables
| Work | Partition | Precision/Comm. | Concurrent | Domain |
|---|---|---|---|---|
| Adaptation | Workload | |||
| Matsubara et al. [ 4 ] | Static | No | No | Edge vision |
| Zhang et al. [ 5 ] | Dynamic | No | Yes | Mobile edge |
| Moosmann et al. [ 10 ] | None | Quantization | No | Embedded vision |
| Devoto et al. [ 6 ] | Fixed | Token adaptation | No | Transformer edge |
| Katare et al. [ 12 ] | Adaptive | Approximation | Yes | CAV |
| Configuration / Method | Stream | Compute Energy (J) | Comm. Energy | Total Energy | Avg. Latency | 3D Detection | |
| Count ( ) | Local ( ) | Aggregator ( ) | (J) | (J) | (ms) | (%) | |
| Local Platform: Jetson Orin Nano (8 GB) Aggregator: Jetson AGX Orin (32 GB) | |||||||
| Full Local FP16 | 2 | 0.88 | 0.00 | 0.00 | 0.88 | 58.4 | 79.2 |
| Static Early Split ( , FP16) | 2 | 0.21 | 0.49 | 0.38 | 1.08 | 44.1 | 78.8 |
| Static Late Split ( , FP16) | 2 | 0.58 | 0.22 | 0.46 | 1.26 | 51.2 | 78.5 |
| Uniform INT8 Split ( ) | 2 | 0.16 | 0.34 | 0.21 | 0.71 | 36.5 | 74.8 |
| Parameter | Orin Nano | Orin NX | AGX Orin |
|---|---|---|---|
| Memory | 8 GB | 16 GB | 32 GB |
| GPU | 1024-core | 1024-core | 1792-core |
| Tensor Cores | 32 | 32 | 56 |
| Execution Role | Local | Local | Aggregator |
| Precision | FP16, INT8 | FP16, INT8 | FP16, INT8 |
| Power Monitoring | INA3221 | INA3221 | INA3221 |
Edge Inference
Edge inference runs trained AI models on nearby devices such as phones, cameras, robots, and embedded computers under tight resource and latency limits.
See all 246 papers