Fixed-function CNN accelerators make edge inference fast and energy-efficient, but they execute only networks that obey a precise numerical contract. On the MAX7800x family, this contract requires power-of-two, per-tensor, symmetric scales, 8-bit activations and biases, integer-only arithmetic, and one statically configured shift per layer. We present ShiftQAT, the quantization-aware training (QAT) flow we developed for this family, which produces networks satisfying the contract by construction. It is the default training path of ai8x-training, the family's open-source training toolchain. ShiftQAT resolves residual, concatenation, and weight-sharing dependencies so that activation scales propagate statically and each layer's requantization reduces to one offline-configured shift. It also trains weights and biases on a joint power-of-two grid that fits the 8-bit bias budget. Among the compared methods, only ShiftQAT satisfies all of the contract's scaling constraints, including signed 8-bit activation storage. Even so, on ImageNet it matches the W8A8 accuracy drop of less constrained QAT methods and stays within 2.9 percentage points of FP32 at W4A8. On MAX78002 silicon, ShiftQAT-trained MobileNetV2 and ResNet-18 classify ImageNet images in 14.6 and 8.4 ms at 5.1 and 2.8 mJ per inference.
Figures & tables
Constraint
Required mechanism
C1
Signed fixed-point (Q7) number format; uniform, symmetric, no zero-point
Symmetric quantizers everywhere (§ 4.2 , § 4.5 )
C2
Rescaling only by 2k , k∈[−15,15] , applied to the 32-bit accumulator
PoT scales for Sx , Sw (§ 4.2 , § 4.5 )
C3
One output shift per layer (per-tensor)
Per-tensor Sx , Sw
C4
Biases stored in 8 bits; weights in 8/4/2/1 bits
Joint weight/bias scale (§ 4.5 )
C5
Activations stored as signed 8-bit; ReLU outputs clipped to [0,127]
Signed symmetric activation quantization
C6
No floating point in the inference path
All scales folded offline (§ 4.4 )
Table 1: The numerical contract of the MAX7800x CNN accelerator, as specified in the public MAX7800x documentation, and the ShiftQAT mechanism each constraint requires. Constraints C1–C7 are properties of the datapath, not design preferences.
Figure 1: Per-layer inference datapaths. (a) Per-layer scale recovery as in integer-only QAT ( Jacob et al., 2018 ; Esser et al., 2020 ) . The product W′X′ is rescaled by Sw and Sx before a full-precision bias is added, and the result is then requantized. (b) The structure the MAX7800x contract requires, as produced by ShiftQAT. Weights and biases share one power-of-two scale Sw , and activation scales are resolved and propagated offline. The only runtime rescaling is a single power-of-two shift per layer (the PoT scaler, Eq. 11 ).
Figure 2: The ShiftQAT flow. FP32 training is followed by offline initialization, which consists of BN folding, activation-scale proposal, and activation-scale refinement (dependency resolution and propagation). QAT then runs with per-iteration weight/bias scale determination.
Figure 3: Dependency resolution. (a) For a residual connection, both input layers adopt the scale of the element-wise addition. (b) For a concatenation, the inputs adopt the largest scale among them. (c) For weight sharing, all layers feeding the shared layer adopt the largest scale among them, so that one set of shared weights and one shift serve every invocation.
Figure 4: Static scale propagation on a two-layer network. On the left, each layer’s output carries its real-valued scale. On the right, each layer emits scale-normalized outputs, downstream biases and scales are adjusted offline (Eq. 5 ), and a single recovery remains at the output. Activations are omitted from the diagram for clarity, but the derivation requires them to be positively homogeneous.
Model
Method
Regime
FP (%)
Q (%)
Δ (pp)
Unif.
Sym.
PoT
Per-ten.
Static prop.
MobileNetV2
ShiftQAT
QAT
71.878
70.852
1.026
+
+
+
+
+
TQT ( Jain et al., 2020 )
QAT
71.7
71.8
−0.1
+
+
+
+
−
QAT ( Jacob et al., 2018 )
QAT
71.9
70.9
1.0
+
−
−
+
−
HPTQ ( Habi et al., 2021 )
PTQ
71.812
71.46
0.352
+
+
+
−
−
ResNet-18
ShiftQAT
QAT
69.758
69.2
0.558
+
+
+
+
+
LSQ ( Esser et al., 2020 )
QAT
70.5
71.1
−0.6
+
+
−
+
−
Table 2: W8A8 ImageNet top-1 accuracy. Baseline MobileNetV2 results are as reported by HPTQ ( Habi et al., 2021 ) , and baseline ResNet-18 results are as reported by LSQ ( Esser et al., 2020 ) . A “ + ” marks a satisfied property and a “ − ” an unsatisfied one. The Static prop. column marks the static scale propagation of Section 4.4 , which is this paper’s mechanism and so is satisfied only by ShiftQAT. The Regime column gives post-training (PTQ) or quantization-aware training (QAT). ShiftQAT is the only row satisfying every property column. All methods use 32-bit biases.
Model
Method
Regime
FP (%)
Q (%)
Δ (pp)
Unif.
Sym.
PoT
Per-ten.
Static prop.
MobileNetV2
ShiftQAT
QAT
71.878
69.014
2.864
+
+
+
+
+
QAT ( Nagel et al., 2021 )
QAT
71.72
70.17
1.55
+
−
−
+
−
AdaRound ( Nagel et al., 2020 )
PTQ
72.62
69.25
3.37
+
−
−
+
−
BRECQ ( Li et al., 2021 )
PTQ
72.62
71.42
1.2
+
−
−
−
−
EPTQ ( Gordon et al., 2024 )
PTQ
72.62
72.06
0.56
+
+
−
−
−
ResNet-18
ShiftQAT
QAT
69.758
67.854
1.904
+
+
+
+
+
Table 3: W4A8 ImageNet top-1 accuracy. PTQ baseline results are as reported by EPTQ ( Gordon et al., 2024 ) . QAT baseline results are the per-tensor results of Nagel et al. (2021) (average of three runs). Columns as in Table 2 . All methods use 32-bit biases.
Model
Input
Acc. (%)
Weights
Biases
Inference
Num. (M)
Size (MB)
Num. (k)
Size (kB)
Time (ms)
Energy ( μ J)
MobileNetV2 ( m=0.75 )
128 × 128
62.812
2.29
2.03
6.6
6.6
14.6
5137
ResNet-18 ( m=0.5 )
128 × 128
53.230
3.06
2.05
3.4
3.4
8.4
2768
Table 4: MobileNetV2 and ResNet-18 deployed end-to-end on MAX78002 with the ShiftQAT flow (CNN clock 200 MHz). Accuracy is ImageNet top-1. Time and energy are per inference, excluding weight and input loading, measured on the VCOREA rail with idle power subtracted.
Quantization-aware training (QAT) is typically performed for a single target numeric format, while practical deployments often need to choose numerical precision at inference time based on hardware support or runtime constraints. We study multi-format QAT, where a single model is trained to be robust across multiple quantization formats. We find that multi-format QAT can match single-format QAT at each target precision, yielding one model that performs well overall across different formats, even formats that were not seen during training. To enable practical deployment, we propose the Slice-and-Scale conversion procedure for both MXINT and MXFP that converts a high-precision representation into lower-precision formats without re-training. Building on this, we introduce a pipeline that (i) trains a model with multi-format QAT, (ii) stores a single anchor format checkpoint (MXINT8/MXFP8), and (iii) allows on-the-fly conversion to lower MXINT or MXFP formats at runtime with negligible-or no-additional accuracy degradation. Together, these components provide a practical path to elastic precision scaling and allow selecting the runtime format at inference time across diverse deployment targets.
Quantization-aware training (QAT) is widely adopted to quantize language models by training full-precision weights using gradients from the quantized model. The main bottleneck is its slow convergence and early performance plateau, particularly below 4-bit-widths. While this problem has been observed in prior work, its precise cause remains unclear. In this paper, we analyze the convergence of QAT by estimating the spectrum of the loss-surface Hessians. We find that the weights converge to flat regions around saddle points, where a large fraction of the Hessian eigenvalues are both positive and negative. During training, an increasing fraction of Hessian eigenvalues concentrates around zero, whose magnitude decreases. At lower bit-widths, the magnitude of eigenvalues in the Hessian spectrum is significantly smaller. To mitigate these issues, we propose an algorithm called WinQ to accelerate QAT, which involves: (1) periodically resetting weights to the linear interpolation of full-precision and quantized weights, reducing the distance to the quantization grid and increasing eigenvalue magnitude, and (2) computing gradients of noise-injected weights to regularize the Hessian. Extensive experiments show that WinQ accelerates QAT by up to 4 times across various quantization methods and models. Under the same training cost, WinQ improves state-of-the-art sub-4-bit quantization by up to 8.8%. These results are consistent across 16 settings with different language models, quantization methods, and bit widths.
Quantization-aware training (QAT) with low-bit floating-point formats enables efficient LLM deployment, yet introduces subtle failure modes invisible to standard training metrics. We present a systematic study of HiF8 W8A8 QAT for OpenPangu-Embedded-1B through the lens of Delayed Tensor Scaling (DTS). Across eight controlled experiments, we identify and disentangle two orthogonal failure modes: (i)amax saturation, where delayed scale estimates silently corrupt knowledge-sensitive representations via forward-pass clipping, and (ii)catastrophic forgetting, where an aggressive learning rate overwrites pretrained commonsense knowledge independently of quantization. Neither is detectable from training loss alone. We address amax saturation with a conservative max-algorithm DTS strategy over a 64-step history window, and mitigate forgetting via a 500-step BF16 warmup followed by QAT at lr=10^{-5}. Both fixes are necessary and sufficient: our final configuration achieves 0.43% MMLU drop, 0.58% HellaSwag drop, and 0.22% ARC-Challenge drop versus a matched BF16 baseline, with a training loss APE of only 0.11% over 10,000 steps.