Deploying foundation models is increasingly constrained by memory footprint, latency, and hardware costs. Post-training compression can mitigate these bottlenecks by reducing the precision of model parameters without significantly degrading performance; however, its practical implementation remains challenging as practitioners navigate a fragmented landscape of quantization algorithms, precision budgets, data-driven calibration strategies, and hardware-dependent execution regimes. We present OneComp, an open-source compression framework that transforms this expert workflow into a reproducible, resource-adaptive pipeline. Given a model identifier and available hardware, OneComp automatically inspects the model, plans mixed-precision assignments, and executes progressive quantization stages, ranging from layer-wise compression to block-wise refinement and global refinement. A key architectural choice is treating the first quantized checkpoint as a deployable pivot, ensuring that each subsequent stage improves the same model and that quality increases as more compute is invested. By converting state-of-the-art compression research into an extensible, open-source, hardware-aware pipeline, OneComp bridges the gap between algorithmic innovation and production-grade model deployment.
Figures & tables
Figure 1. Overview of the OneComp pipeline. Top : end-to-end workflow from a pretrained foundation model to a deployment-ready quantized model. Bottom : the three quantization granularity levels on a Transformer with mixed-precision bit allocation. Cell colors indicate per-layer bit-widths assigned by AutoBit: Darker colors indicate lower precision. Layer-wise PTQ operates on one linear layer; block-wise PTQ on one Transformer block; global PTQ on the full model.
Figure 2. Comparison of 4-bit and 2-bit uniform quantization on the same weight distribution. Each panel overlays the quantization grid on a histogram of weights. 4-bit uniform quantization provides 16 fine-grained levels; 2-bit uniform quantization provides only 4, yielding coarser approximation and larger errors.
Figure 3. Three levels of quantization granularity for a weight matrix. Each color represents a group of elements sharing the same scale and zero-point. Per-tensor uses one set of parameters for the entire matrix; per-channel uses one per row; per-group divides each row into smaller groups.
List of Codes 1 Composable refinement pipeline. Each refiner accepts a quantized model and returns an improved one.
List of Codes 2 Minimal usage of OneComp . A single high-level invocation quantizes the target model, evaluates the result, and saves the resulting checkpoint.
Figure 4. Two quantization formats supported by OneComp . Left: GPTQ stores per-group integer weights Q , scales s , and zero-points z , and reconstructs weights via W^ij=sg(qij−zg) . Right: MDBF represents the weight matrix using shared binary sign bases Sa,Sb together with low-rank real-valued envelopes AP⊤ and BG⊤ , yielding W^=(Sa⊙AP⊤)(Sb⊙BG⊤)⊤ .
Figure 5. Example module-wise bit allocation on Llama 3 under a 4.16 average bpw budget, which is equivalent to 4-bit quantization with a group size of 128. Top: naive mixed-precision allocation. Bottom: activation-aware allocation using diagonal activation statistics only. The activation-aware variant tends to reserve higher precision for modules with stronger activation outliers or higher activation sensitivity.
Model
FP
Method
Average bit per weight (bpw)
PPL
ACC
4.16 (4b/128)
4.00 (4b/ch)
PPL
ACC
PPL
ACC
Llama-3-8B
6.14
0.715
Uniform
9.53
0.697
383.26
0.595
Naive
10.28
0.691
10.95
0.682
Act.-aware
6.85
0.702
7.04
0.701
Qwen3-8B
9.72
0.707
Uniform
10.92
0.706
10.65
0.679
Table 1. AutoBit mixed-precision allocation under fixed average bit-per-weight (bpw) budgets. For each model, the full-precision (FP) reference is shown once and shared across all quantized variants. We compare three allocation methods: Uniform, Naive, and Act.-aware (diagonal activation only). For each bpw budget, perplexity (PPL; lower is better) and average 0-shot accuracy (ACC; higher is better) are reported separately.
Model
FP
Method
Quantization setting
PPL
ACC
4b/128
4b/ch
3b/128
3b/ch
PPL
ACC
PPL
ACC
PPL
ACC
PPL
ACC
Llama-3-8B
6.14
0.715
Base
12.66
0.697
665.94
0.533
45.22
0.521
1721.06
0.386
QEP
6.66
0.710
7.67
0.688
8.95
0.634
17.93
0.496
Submodule
6.59
0.709
7.33
0.703
8.54
0.640
22.40
0.482
Table 2. Layer-wise PTQ on Llama-3-8B under a single-GPU budget. The full-precision (FP) reference is shown once. Base: GPTQ without correction. QEP: layer-level error-propagation correction. Submodule: QEP extended with residual-stream correction. PPL: WikiText-2 perplexity ( ↓ ). ACC: average 0-shot accuracy ( ↑ ).
Model
FP
Method
Quantization setting
PPL
ACC
4b/128
4b/ch
3b/128
3b/ch
PPL
ACC
PPL
ACC
PPL
ACC
PPL
ACC
Llama-3-8B
6.14
0.715
GPTQ
12.66
0.697
665.94
0.533
45.22
0.521
1721.06
0.386
JointQ
6.67
0.705
8.46
0.676
9.26
0.650
21.15
0.583
Qwen3-8B
9.72
0.707
GPTQ
10.29
0.697
10.97
0.672
11.71
0.654
20.21
0.496
JointQ
10.29
0.695
11.33
0.660
12.38
0.668
42.10
0.506
Table 3. Comparison between JointQ and standard GPTQ under the same packed quantized-weight format, with activation ordering and MSE-based tuning disabled. For each model, the full-precision (FP) reference is shown once. Here, 128 denotes group-wise quantization with group size 128 , and ch denotes per-channel quantization. We report perplexity (PPL ↓ ) and average zero-shot accuracy (ACC ↑ ). Boldface indicates the better result between GPTQ and JointQ for each setting.
Model
1.00 BPW
1.50 BPW
DBF
MDBF
DBF
MDBF
PPL
ACC
PPL
ACC
PPL
ACC
PPL
ACC
Qwen3-0.6B
43.07
0.412
34.45
0.426
30.80
0.435
26.56
0.446
Llama-3.2-1B
21.48
0.465
18.29
0.473
16.58
0.491
15.00
0.492
TinyLlama-1.1B
13.57
0.449
12.29
0.449
10.38
0.491
9.84
0.490
Llama-2-7B
9.95
0.480
9.36
0.489
7.34
0.548
7.17
0.564
Table 4. Extreme low-bit quantization results after the full three-stage pipeline. For each bit budget we report WikiText-2 perplexity (PPL ↓ ) and average zero-shot accuracy (ACC ↑ ). Boldface indicates the better result between DBF and MDBF.
Figure 6. Progressive quality improvement on Llama-3.2-1B under DBF quantization at 2.0, 1.5, and 1.0 bits per weight. Each pipeline stage, layer-wise, block-wise, and global PTQ, provides monotonic accuracy improvement regardless of bit-width.
PQuantML is a new open-source, hardware-aware neural network model compression library tailored to end-to-end workflows. Motivated by the need to deploy performant models to environments with strict latency constraints, PQuantML simplifies training of compressed models by providing a unified interface to apply pruning and quantization, either jointly or individually. The library implements multiple pruning methods with different granularities, as well as fixed-point quantization with support for High-Granularity Quantization. We evaluate PQuantML on representative tasks such as the jet substructure classification, so-called jet tagging, an on-edge problem related to real-time LHC data processing. Using various pruning methods with fixed-point quantization, PQuantML achieves substantial parameter and bit-width reductions while maintaining accuracy. The resulting compression is further compared against existing tools, such as QKeras and HGQ.
Roope Niemi, Anastasiia Petrovych, Arghya Ranjan Das +9
European Center for Nuclear Research (CERN), CH-1211 Geneva, Switzerland · Purdue University, West Lafayette, IN 47907, USA · California Institute of Technology, Pasadena, CA 91125, United States +4
Fairness-aware model compression requires selecting methods and configurations that balance accuracy, fairness, and deployment cost. These decisions become more difficult when compression methods are composed or the user's requirements change. In this paper, we propose FairCompressAgent (FCA), an agentic framework that integrates fairness-aware pruning, incremental quantization, and sparse low-rank factorization through a common operator interface. A language-model planner uses model profiles and measured outcomes to select compression configurations, while an execution layer performs compression, fine-tuning, evaluation, and constraint-based selection. FCA also supports requirement updates and reports the remaining violation when a request cannot be satisfied. Experiments on Fitzpatrick-17k with VGG-11 compare four search methods over 40 measured configurations. Under the accuracy-constrained request, FCA selects a compressed model with 59.54% less inference tensor storage, while validation average precision increases from 0.5141 to 0.5233 and equalized opportunity (EOpp) decreases from 0.2251 to 0.2168. It reaches the same final selection as one-shot planning with 7.33 versus 12 candidate evaluations on average, under their respective stopping policies. Repeated fine-tuning, held-out testing, and online requirement updates characterize the stability and interactive use of this compression workflow. The results demonstrate how measured feedback and explicit constraints support the selection and interactive refinement of fairness-aware compression configurations.
Yuanbo Guo, Yiyu Shi
Department of Computer Science and Engineering University of Notre Dame Notre Dame, USA
To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these methods highly rely on calibration data, domain shift issues may arise for unseen downstream tasks. We propose a test-time quantization (TTQ) framework which compresses large models on the fly at inference time to resolve this issue. With an efficient online calibration, instant activation-aware quantization can adapt every prompt regardless of the downstream tasks, yet achieving inference speedup. Several experiments demonstrate that TTQ can improve the quantization performance over state-of-the-art baselines.
Toshiaki Koike-Akino, Jing Liu, Ye Wang
Mitsubishi Electric Research Laboratories (MERL), 201 Broadway, Cambridge, MA 02139, USA