OneComp: One-Line Revolution for Generative AI Model Compression

Yuma Ichikawa; Keiji Kimura; Akihiro Yoshida; Yudai Fujimoto; Hiroki Tokura; Yamato Arai; Yoshiyuki Ishii; Yusei Kawakami

cs.LGMar 30, 2026

OneComp: One-Line Revolution for Generative AI Model Compression

Authors: Yuma Ichikawa, Keiji Kimura, Akihiro Yoshida, Yudai Fujimoto, Hiroki Tokura, Yamato Arai, Yoshiyuki Ishii, Yusei Kawakami, +6 more

Organizations: Fujitsu Limited · RIKEN Center for AIP · Institute of Science Tokyo · The University of Tokyo · Tokai University

Abstract

Deploying foundation models is increasingly constrained by memory footprint, latency, and hardware costs. Post-training compression can mitigate these bottlenecks by reducing the precision of model parameters without significantly degrading performance; however, its practical implementation remains challenging as practitioners navigate a fragmented landscape of quantization algorithms, precision budgets, data-driven calibration strategies, and hardware-dependent execution regimes. We present OneComp, an open-source compression framework that transforms this expert workflow into a reproducible, resource-adaptive pipeline. Given a model identifier and available hardware, OneComp automatically inspects the model, plans mixed-precision assignments, and executes progressive quantization stages, ranging from layer-wise compression to block-wise refinement and global refinement. A key architectural choice is treating the first quantized checkpoint as a deployable pivot, ensuring that each subsequent stage improves the same model and that quality increases as more compute is invested. By converting state-of-the-art compression research into an extensible, open-source, hardware-aware pipeline, OneComp bridges the gap between algorithmic innovation and production-grade model deployment.

Figures & tables

Figure 1. Overview of the OneComp pipeline. Top : end-to-end workflow from a pretrained foundation model to a deployment-ready quantized model. Bottom : the three quantization granularity levels on a Transformer with mixed-precision bit allocation. Cell colors indicate per-layer bit-widths assigned by AutoBit: Darker colors indicate lower precision. Layer-wise PTQ operates on one linear layer; block-wise PTQ on one Transformer block; global PTQ on the full model.

Figure 2. Comparison of 4-bit and 2-bit uniform quantization on the same weight distribution. Each panel overlays the quantization grid on a histogram of weights. 4-bit uniform quantization provides 16 fine-grained levels; 2-bit uniform quantization provides only 4, yielding coarser approximation and larger errors.

Figure 3. Three levels of quantization granularity for a weight matrix. Each color represents a group of elements sharing the same scale and zero-point. Per-tensor uses one set of parameters for the entire matrix; per-channel uses one per row; per-group divides each row into smaller groups.

Panel of Figure 4 — List of Codes 1 Composable refinement pipeline. Each refiner accepts a quantized model and returns an improved one.

Panel of Figure 5 — List of Codes 2 Minimal usage of OneComp . A single high-level invocation quantizes the target model, evaluates the result, and saves the resulting checkpoint.

Figure 4. Two quantization formats supported by OneComp . Left: GPTQ stores per-group integer weights

\bm{Q}

, scales

\bm{s}

, and zero-points

\bm{z}

, and reconstructs weights via

\hat{W}_{ij}=s_{g}(q_{ij}-z_{g})

. Right: MDBF represents the weight matrix using shared binary sign bases

S_{a},S_{b}

together with low-rank real-valued envelopes

AP^{\top}

and

BG^{\top}

, yielding

\hat{W}=(S_{a}\odot AP^{\top})(S_{b}\odot BG^{\top})^{\top}

Model	FP		Method	Average bit per weight (bpw)
	PPL	ACC		4.16 (4b/128)		4.00 (4b/ch)
	PPL	ACC		PPL	ACC	PPL	ACC
Llama-3-8B	6.14	0.715	Uniform	9.53	0.697	383.26	0.595
			Naive	10.28	0.691	10.95	0.682
			Act.-aware	6.85	0.702	7.04	0.701
Qwen3-8B	9.72	0.707	Uniform	10.92	0.706	10.65	0.679

Table 1. AutoBit mixed-precision allocation under fixed average bit-per-weight (bpw) budgets. For each model, the full-precision (FP) reference is shown once and shared across all quantized variants. We compare three allocation methods: Uniform, Naive, and Act.-aware (diagonal activation only). For each bpw budget, perplexity (PPL; lower is better) and average 0-shot accuracy (ACC; higher is better) are reported separately.

Model	FP		Method	Quantization setting
	PPL	ACC		4b/128		4b/ch		3b/128		3b/ch
	PPL	ACC		PPL	ACC	PPL	ACC	PPL	ACC	PPL	ACC
Llama-3-8B	6.14	0.715	Base	12.66	0.697	665.94	0.533	45.22	0.521	1721.06	0.386
			QEP	6.66	0.710	7.67	0.688	8.95	0.634	17.93	0.496
			Submodule	6.59	0.709	7.33	0.703	8.54	0.640	22.40	0.482

Table 2. Layer-wise PTQ on Llama-3-8B under a single-GPU budget. The full-precision (FP) reference is shown once. Base: GPTQ without correction. QEP: layer-level error-propagation correction. Submodule: QEP extended with residual-stream correction. PPL: WikiText-2 perplexity (

\downarrow

). ACC: average 0-shot accuracy (

\uparrow

Model	FP		Method	Quantization setting
	PPL	ACC		4b/128		4b/ch		3b/128		3b/ch
	PPL	ACC		PPL	ACC	PPL	ACC	PPL	ACC	PPL	ACC
Llama-3-8B	6.14	0.715	GPTQ	12.66	0.697	665.94	0.533	45.22	0.521	1721.06	0.386
Llama-3-8B	6.14	0.715	JointQ	6.67	0.705	8.46	0.676	9.26	0.650	21.15	0.583
Qwen3-8B	9.72	0.707	GPTQ	10.29	0.697	10.97	0.672	11.71	0.654	20.21	0.496
Qwen3-8B	9.72	0.707	JointQ	10.29	0.695	11.33	0.660	12.38	0.668	42.10	0.506

Table 3. Comparison between JointQ and standard GPTQ under the same packed quantized-weight format, with activation ordering and MSE-based tuning disabled. For each model, the full-precision (FP) reference is shown once. Here,

128

denotes group-wise quantization with group size

128

, and ch denotes per-channel quantization. We report perplexity (PPL

\downarrow

) and average zero-shot accuracy (ACC

\uparrow

). Boldface indicates the better result between GPTQ and JointQ for each setting.

Model	1.00 BPW				1.50 BPW
	DBF		MDBF		DBF		MDBF
	PPL	ACC	PPL	ACC	PPL	ACC	PPL	ACC
Qwen3-0.6B	43.07	0.412	34.45	0.426	30.80	0.435	26.56	0.446
Llama-3.2-1B	21.48	0.465	18.29	0.473	16.58	0.491	15.00	0.492
TinyLlama-1.1B	13.57	0.449	12.29	0.449	10.38	0.491	9.84	0.490
Llama-2-7B	9.95	0.480	9.36	0.489	7.34	0.548	7.17	0.564

Table 4. Extreme low-bit quantization results after the full three-stage pipeline. For each bit budget we report WikiText-2 perplexity (PPL

\downarrow

) and average zero-shot accuracy (ACC

\uparrow

). Boldface indicates the better result between DBF and MDBF.

Figure 6. Progressive quality improvement on Llama-3.2-1B under DBF quantization at 2.0, 1.5, and 1.0 bits per weight. Each pipeline stage, layer-wise, block-wise, and global PTQ, provides monotonic accuracy improvement regardless of bit-width.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model and data
Symbol	Description
$L$	Number of transformer blocks
$d,~d_{k},~d_{\mathrm{ff}}$	Hidden, per-head, and feed-forward dimensions
$H,~H_{\mathrm{kv}}$	Number of attention / key-value heads
$N,~M,~T$	Weight input/output dimensions, sequence length
$P,~n$	Total parameters, number of calibration samples

Appendix

Table 5. Summary of principal notation.

Explore similar work

PQuantML: A Tool for End-to-End Hardware-aware Model Compression
Mar 27, 2026Roope Niemi, Anastasiia Petrovych, Arghya Ranjan Das +9Model Compression Neural Network Compression
European Center for Nuclear Research (CERN), CH-1211 Geneva, Switzerland · Purdue University, West Lafayette, IN 47907, USA · California Institute of Technology, Pasadena, CA 91125, United States +4
FairCompressAgent: An Agentic Framework for Fairness-Aware Model Compression for FPGA Deployment
Sep 15, 2026Yuanbo Guo, Yiyu ShiModel Compression Algorithmic Fairness
Department of Computer Science and Engineering University of Notre Dame Notre Dame, USA
TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly
Mar 11, 2026Toshiaki Koike-Akino, Jing Liu, Ye WangLLM Quantization Neural Network Quantization
Mitsubishi Electric Research Laboratories (MERL), 201 Broadway, Cambridge, MA 02139, USA

Mar 27, 2026J/K move · Enter open · S save

PQuantML: A Tool for End-to-End Hardware-aware Model Compression

Roope Niemi, Anastasiia Petrovych, Arghya Ranjan Das, Enrico Lupi+8

European Center for Nuclear Research (CERN), CH-1211 Geneva, Switzerland · Purdue University, West Lafayette, IN 47907, USA · California Institute of Technology, Pasadena, CA 91125, United States+4

Model Compression Neural Network Compression Neural Network Quantization Neural Network Pruning

Open paper AskPDF arXiv

PQuantML is a new open-source, hardware-aware neural network model compression library tailored to end-to-end workflows. Motivated by the need to deploy performant models to environments with strict latency constraints, PQuantML simplifies training of compressed models by providing a unified interface to apply pruning and quantization, either jointly or individually. The library implements multiple pruning methods with different granularities, as well as fixed-point quantization with support for High-Granularity Quantization. We evaluate PQuantML on representative tasks such as the jet substructure classification, so-called jet tagging, an on-edge problem related to real-time LHC data processing. Using various pruning methods with fixed-point quantization, PQuantML achieves substantial parameter and bit-width reductions while maintaining accuracy. The resulting compression is further compared against existing tools, such as QKeras and HGQ.

Abstract

Explore similar work

PQuantML: A Tool for End-to-End Hardware-aware Model Compression

FairCompressAgent: An Agentic Framework for Fairness-Aware Model Compression for FPGA Deployment

TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly