Sensitivity Inflation from Cross-Record Precision Routing in Private Quantized Fine-Tuning

Annan Zhang; Shengjie Song; Wenshuai Yao; Lihao Zhang; Tianmiao

cs.LGNov 1, 2026

Sensitivity Inflation from Cross-Record Precision Routing in Private Quantized Fine-Tuning

Authors: Annan Zhang, Shengjie Song, Wenshuai Yao, Lihao Zhang, Tianmiao

Organizations: Peking University · Beihang University

Abstract

Private fine-tuning of large language models increasingly pairs DP-SGD with low-rank adapters on frozen, quantized backbones, and every such stack inherits one unstated premise: frozen layers keep each record's clipped gradient a function of that record alone. We show that LLM.int8 breaks this premise. It routes outlier activation columns to a higher-precision path using a union over the whole batch, so one record decides how every other record's activations are computed, even though the weights are frozen and the quantization scales are per row. We bound the resulting sensitivity inflation, state a record-isolation contract that restores the standard bound, and define record-owned precision dispatch that meets it with the same GEMMs as batched LLM.int8, so isolation needs no extra arithmetic. We then audit the native CUDA stack (bitsandbytes, PEFT, Opacus) against four matched controls. A constructed LoRA-Opacus witness reaches $15.0\times$ the add/remove bound. On pretrained OPT-125M with natural text, other records alter 20% of record-layer precision splits at batch size 16. An exploratory re-clipping analysis finds 10 of 64 candidate neighbors above the bound once every gradient is clipped; the top pair, revalidated natively, reaches $1.065C$ while every control stays within $C$ , and forcing one layer's outlier mask removes the excess. A two-million-draw audit of the released adapters yields $\varepsilon$ lower bounds ( $3.25$ at $δ=10^{-5}$ ) below the advertised guarantee ( $4.38$ ). Resolving an excess this small at the release level would take about 200 times more draws, whereas checking the sensitivity contract directly exposes it in a few native runs, which makes contract checking the practical audit for private quantized training.

Figures & tables

Panel of Figure 1 — Figure 1: Frozen weights do not make records independent. LLM.int8 sends an activation column to an FP16 path if any row of the flattened batch reaches the threshold $\tau$ , and applies that batch-wide set to every row. (a) Adding or changing one record ( $x$ ) therefore changes the arithmetic of unchanged records ( $r_{1},r_{2}$ ), although all weights are frozen and quantization scales are per row. (b) Record-owned routing derives each record’s outlier set $O_{r}$ from its own rows, which restores the separable clipping bound.

Rung	Setting	Native result	Controls	Establishes (and does not)
Operator (§ 6.1 )	constructed, one INT8 layer	$7.0\times$ (replacement)	$0$	the CUDA kernels execute the coupling (not: a natural task)
LoRA–Opacus (§ 6.1 )	constructed, pinned stack	$15.0\times$ (add/remove)	$\leq 1$	Proposition 1 ’s worst case through real Opacus (not: a natural task)
Coupling (§ 6.2 )	OPT-125M, TinyStories, $B{=}2$ – $16$	5–20% of pairs coupled	–	routing coupling is routine on natural text (not: sensitivity)
Sensitivity (§ 6.3 )	OPT-125M, 64 candidates, 40 cases	$1.0653\times$ (add/remove)	$\leq 1$	the separable premise fails natively (not: prevalence)
Mechanism (§ 6.4 )	forward factorial, forced mask	one column, one layer	–	the union change causes the excess (not: exact independence)
Release (§ 6.5 )	$2\times 10^{6}$ actual releases	$\varepsilon_{\rm lb}{=}3.25$ vs. $4.38$	–	no detected guarantee violation (not: a privacy proof)

Table 1: Evidence at a glance. Ratios are clipped-sum distances over the separable bound (

C

for add/remove,

2C

for replacement). Sensitivity measurements pair native batching with record-isolation, threshold-zero, BF16 and (where applicable) NF4 controls on identical inputs.

	Operator	LoRA–Opacus
Execution	$\Delta/2C$	$\Delta/C$	$\Delta/2C$
	replace	add/remove	replace
Native batched INT8	$\mathbf{7.000}$	$\mathbf{14.998}$	$\mathbf{6.999}$
Record-isolated INT8	0	1.000	0
Threshold zero	0	1.000	0
BF16	0	1.000	0

Table 2: Constructed witnesses on the native CUDA stack. Ratios are clipped-sum distances over the separable bound; the last row is the worst case of Proposition 1 for

m{=}7

altered records.

Panel of Figure 2 — Figure 2: Coupling is routine on natural text. Native LLM.int8 forward passes of pretrained OPT-125M on the first 16 TinyStories training windows. (a) Of the 72 INT8 layers, 42 have a nonempty outlier union at every batch size; the bars count (record, layer) pairs in those 42 layers. At $B{=}16$ , one pair in five computes at least one column in FP16 only because another record crossed the threshold. (b) Number of records, out of 16, whose own outlier set differs from the batch union in each layer; blank cells have no outlier columns. The query, key and value projections read the same input and share one row. Coupling concentrates in a few layers: every record is affected in layer 7’s attention input, but most layers with outliers show none, because their outlier columns are shared by all records.

Panel of Figure 3 — Figure 3: Sensitivity on pretrained OPT-125M appears once clipping is active. (a) Exploratory add/remove ratio $\kappa=\Delta/C$ for all 64 candidate additions to one seven-record TinyStories batch, from re-clipping the saved native per-record gradients at nine radii (all 576 scores shown). Orange: the 10 candidates that exceed $C$ by more than $0.1\%$ . At the original screening radius $C{=}1$ most gradients are not clipped, the added record contributes less than $C$ , and no candidate exceeds the bound. (b) Forty fresh native cases for the top candidate at $C{=}0.25$ and $C{=}0.001$ : native batching exceeds the bound by $6.53\%$ at both radii, while record isolation, threshold zero, BF16 and NF4 stay within it. Identical-batch repeats give exactly zero distance, so the measurement floor is zero.

Panel of Figure 4 — Figure 4: One layer’s outlier mask causes the excess. Selected OPT-125M pair at $C{=}0.25$ , with $D$ (7 records) and $D^{\prime}$ (8 records). (a) Of the 72 INT8 unions, 42 are nonempty and exactly one differs between $D$ and $D^{\prime}$ : decoder layer 6’s first feed-forward projection gains one column. (b) Forward factorial at that layer, input batch fixed to $D$ : the common records’ activations are bitwise identical under $D$ and $D^{\prime}$ , and swapping only the mask reproduces the entire logit change. (c) Forcing that layer to use the frozen union $O(D^{\prime})$ through native forward and backward shrinks every common record’s gradient change by four orders of magnitude and brings the clipped sum below $C$ . The residual is nonzero, so exact independence is not established, and the post hoc mask is a diagnostic rather than a private policy.

Panel of Figure 5 — Figure 5: No release-level violation is detected, and none could be at this audit size. (a) Two million actual FP32 adapter releases of the selected pair ( $10^{6}$ per neighbor). Triangles: held-out Monte Carlo lower bounds on $\varepsilon$ (family-wise 95% Clopper–Pearson). Lines: exact real-valued Gaussian privacy curves (Eq. 5 ) at the advertised sensitivity (black) and at the measured pair (orange). (b) Expected bound at $\delta{=}10^{-5}$ when the same procedure audits an ideal Gaussian release, with an oracle threshold (analytic). Our observed bound lies between the two curves; a bound above the advertised $\varepsilon$ needs roughly 200 times more draws.

Privacy-Preserving ML

Privacy-preserving machine learning develops methods to learn, share, and run models while limiting exposure of sensitive data.

See all 438 papers

Oct 6, 2026

Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

Quantization-aware training (QAT) leverages lower-precision arithmetic to reduce the cost of LLM deployment, but aggressive quantization degrades final model performance. A common remedy is mixed-precision training, in which high precision is assigned to some of the layers to maintain performance while keeping the cost constrained. This approach then requires precision assignments for model layers during training. We provide a new approach, called Q-PACE, consisting of a second-order sensitivity model that predicts the loss increase as a sum of quantization noise MSE weighted by per-layer curvature coefficients. During training, we periodically re-compute these coefficients using perturbations across layers, and re-assign precision. Pretraining and supervised fine-tuning experiments on LLMs of up to 4B parameters show that Q-PACE consistently improves over existing mixed-precision training recipes, and achieves comparable loss at substantially lower total memory budgets. We further find that quantization sensitivity is highly predictable by depth and layer type, and its stability during training allows for infrequent, cheap recalibration.

Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2

ISTA · EPFL, ISTA

LLM Quantization Quantization-Aware Training Quantization Mixed-Precision Quantization Adaptive Quantization Low-Precision Training+2

Ask PDF

Jul 20, 2026

MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

4-bit quantization enables efficient LLM inference, but suffers from significant accuracy degradation due to outliers. Prior work addresses this problem via data rotation or mixed-precision integer quantization, but often relies on software-managed scaling and frequent dequantization, incurring substantial overhead. Microscaling formats, such as MXINT, eliminate these inefficiencies by encoding scales in hardware, yet remain incompatible with rotation-based methods. Our analysis reveals that outliers vary in severity, from rare extremes to frequent mild deviations, and that quantization sensitivity is unevenly distributed across layers and columns. These insights motivate a fine-grained, sensitivity-guided approach. We introduce MXSens, a training-free method that assigns mixed mantissa bitwidths (4/6/8) based on column- and layer-wise sensitivity, naturally leveraging the block-wise structure of MXINT. MXSens outperforms state-of-the-art quantization methods across a range of models and tasks. Under the W4A4KV4 setting, MXSens achieves perplexities of 3.77 and 7.63 on LLaMA-2-70B and LLaMA-3-8B, respectively, substantially improving over existing baselines on WikiText-2. Our work establishes a new balance between accuracy and resource efficiency for LLM quantization.

Simla Burcu Harma, Danila Mishin, Zhengyuan Su +7

EcoCloud, EPFL · Tsinghua University · MangoBoost Inc. +2

Mixed-Precision Quantization LLM Quantization LLM Inference Acceleration Post-Training Quantization Neural Network Quantization Quantization Sensitivity Analysis

Ask PDF

Nov 4, 2026

TileQuant: Tile as the Precision-Allocation Unit for LLM Mixed-Precision Post-Training Quantization

Post-training quantization (PTQ) reduces large language model (LLM) deployment costs, but preserving quality requires accounting for uneven weight sensitivity. Uniform PTQ uses one weight bit-width, importance-aware PTQ adjusts scales or reconstructs weights within that width, and column-wise mixed precision retains selected columns at higher precision. However, sensitivity varies within columns, so whole-column retention also protects less sensitive weights, while runtime outlier detection adds decoding overhead. We therefore introduce TileQuant, a training-free framework that assigns precision to hardware-aligned 2D weight tiles. Offline calibration provides input-activation-aware scoring from forward statistics for lower calibration cost and gradient-informed scoring from loss sensitivity for stronger quality preservation. Within each projection's retention budget, the highest-scoring tiles remain at 16-bit and the rest use 8-bit. The resulting map guides reconstruction into compact weight buffers, which the kernel reuses for native integer and 16-bit computation with local accumulation to reduce memory traffic. Experiments show that TileQuant reaches 2.66x the original model's generation throughput. Across language modeling, reasoning, long-context retrieval, and question answering, it preserves quality and surpasses quantized and original models on multiple tasks. These results support selecting precision units that capture local sensitivity while enabling efficient hardware execution. The implementation is: https://github.com/HanzhiZhang-Ulrica/TileQuant

Hanzhi Zhang, Qiao Zhang, Qinglei Cao +4

LLaVi Lab, Computer Science & Engineering, University of North Texas · Computer Science, Saint Louis University · Data Science, University of North Texas

LLM Quantization Post-Training Quantization Neural Network Quantization Mixed-Precision Quantization Hardware-Aware Quantization

Ask PDF

Abstract

Privacy-Preserving ML

Explore similar work

Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

TileQuant: Tile as the Precision-Allocation Unit for LLM Mixed-Precision Post-Training Quantization