Compressing Onboard Language Models for Space Robots under a Behavioral-Fidelity Requirement

Jia Deng; Zizhe Wang; Xin Lu; Yi Huang; Zehao Jia

cs.ROOct 21, 2026

Compressing Onboard Language Models for Space Robots under a Behavioral-Fidelity Requirement

Authors: Jia Deng, Zizhe Wang, Xin Lu, Yi Huang, Zehao Jia

Organizations: Astroup Lab, Beijing, China

Abstract

Language models are being explored as the interface through which space robots interpret operator commands and select among verified skills. Running them onboard calls for low-bit compression, and flight use adds requirements that generic post-training quantization leaves implicit: a fixed memory budget and storage format, a static artifact under configuration control, and behavior close to the full-precision model validated on the ground. We treat compression as a ground-segment step and study QSelect, which composes one static low-bit model from compatible AWQ, GPTQ, and AutoRound checkpoints by assigning each linear module to one quantizer family and scoring every substitution at the final next-token distribution. The default fidelity mode needs no labels or tuned thresholds and accepts a substitution only when the forward KL divergence from the BF16 reference decreases; an optional target-adapted mode adds answer likelihood on a small labeled set. On DeepSeek-R1-Distill-Qwen-7B, Llama-3.1-8B, and Llama-3.2-3B at 3 and 4 bits, composed models reach lower mean held-out perplexity than every single-family model in all six settings, and the fidelity mode lowers perplexity and KL further. In a matched scaffold, final-output scoring gives lower held-out KL than representation-similarity scoring in all 12 paired runs. With labels from training or development splits of the evaluated task families, the target-adapted mode raises 3-bit pooled accuracy on the two Llama models by 4.3 and 4.5 points over the best single-family model. At 3 bits, unquantized embedding and output layers make up 43-46% of the packed artifact. All evaluations use general-domain benchmarks on a ground GPU; space-domain tasks and flight hardware are left to future work.

Figures & tables

Fig. 1: Overview of QSelect, with Llama-3.2-3B at W4 as the running example. The three W4 candidates share one packed format. Step 1 gives module

\ell

of the current model

F(a)

to family

r

and scores the result by the forward KL from the cached reference distribution; the lowest-KL family is kept only if it lowers the current KL (right). Step 2 commits the choice and moves on, in one shallow-to-deep pass over all 196 modules. The static artifact is uplinked after acceptance tests; onboard use is not evaluated here. Bar heights and grid colors are schematic; counts, token budgets, and times are measured values (Sections V and VI ).

Requirement	Design choice in QSelect	Evidence in this paper
R1 Fixed memory budget and a storage format the onboard runtime supports	All candidates share bit width, group size, and packed integer format; selection never changes the bit width	Table VII , Fig. 3
R2 Static artifact under configuration control	One candidate per module, fixed after selection; no runtime selector, routing, or ensemble; the assignment vector and checkpoint hashes identify the artifact	By construction (Section IV-F )
R3 Behavior close to the ground-validated reference	Candidates are scored by the forward KL from the reference next-token distribution on calibration text; acceptance can check the same quantity on held-out text	Tables II – III
R4 Few or no mission labels	Fidelity mode needs only unlabeled text; the target-adapted mode uses a small labeled set	Tables II and III
R5 Bounded, one-time ground compute	Decision granularity and pass count trade ground time for fidelity	Fig. 2 , Table VI

TABLE I: Deployment requirements for an onboard language model and how they are addressed

Setting	Configuration	$\Delta$ PPL	$\Delta$ KL	$\Delta$ Micro	$\Delta$ Macro
DS W3	QSelect-KL	-4.8%	-7.5%	-0.15	+0.75
	Local-MSE	+8.0%	+1.7%	-2.16	-1.20
DS W4	QSelect-KL	-1.7%	-13.8%	+0.02	-0.45
	Local-MSE	+2.9%	-13.5%	+0.04	+0.32
Llama-3.1 W3	QSelect-KL	-2.1%	-4.3%	-0.90	-1.20
	Local-MSE	-0.7%	+4.3%	-3.60	-3.30

TABLE III: Fidelity mode and a local reconstruction criterion, both relative to the module-wise QSelect-TA mean. PPL and KL are relative changes of three-replicate means; accuracy values are percentage points. Negative PPL/KL and positive accuracy favor the listed configuration. Local-MSE was run on the 7B/8B settings only.

Setting	$\Delta$ PPL	$\Delta$ KL	$\Delta$ Micro	$\Delta$ Macro
DS W3	-1.42%	-9.96%	+0.74	+1.17
DS W4	+2.03%	-5.07%	-0.03	-0.46
Llama-3.1 W3	-6.26%	-19.14%	+5.45	+4.29
Llama-3.1 W4	-0.81%	-17.14%	+0.68	+0.88

TABLE IV: CKA versus guarded final-distribution KL (0.1% margin) inside the same sequential layer-wise scaffold. Deltas are final-KL minus CALM-reimpl.; PPL/KL are relative percentages and accuracy values are percentage points.

Setting	GPTQ	AutoRound
DS W3	3	0
DS W4	3	0
Llama-3.1 W3	3	0
Llama-3.1 W4	3	0
Llama-3.2 W3	0	3
Llama-3.2 W4	2	1

TABLE V: Whole-model family choices made by the same KL-shortlist and answer-NLL rule. Counts are over calibration-bank replicates 17, 23, and 41.

Fig. 2: Ground selection time versus fidelity and task accuracy on DS-R1-Qwen-7B W3; labels in (c) apply to all panels. Filled circles use the final output (layer-, component-, and module-wise QSelect-TA, the guarded layer-wise KL selector, and module-wise QSelect-TA with three passes); hollow squares are proxy criteria (Local-MSE and CALM-reimpl. with CKA). All values come from Tables II , III , VI , XI , and XVI . Local-MSE and three-pass metrics are derived from reported relative changes. Times for Local-MSE and the layer, component, and module scans are the approximate values reported for the DS settings (about 0.8, 18, 35–40, and 210 minutes); the bar on the component point spans 35–40 minutes. The triangle is module-wise QSelect-KL, derived from Table III ; its time is not reported, so it is drawn at the module-wise QSelect-TA time, which bounds it from above.

Setting	Measured	Est. †	BF16 share †	CUDA	$\Delta$ PPL	$\Delta$ KL
	Artifact reduction			Peak	Drift
DS W3	68.74%	68.79%	45.9%	26.3%	-0.029%	-0.089%
DS W4	63.37%	63.40%	39.1%	24.2%	-0.028%	+0.024%
Llama-3.1 W3	69.70%	69.78%	43.3%	27.9%	+0.033%	-1.314%
Llama-3.1 W4	64.26%	64.30%	36.7%	25.9%	-0.277%	-0.232%

TABLE VII: Packed export of the calibration-replicate-17 assignments (QSelect-TA). Artifact and peak-CUDA entries are reductions relative to the BF16 reference;

\Delta

PPL and

\Delta

KL are packed-versus-fake-quantized drift. † From the analytic storage model (Section VII -A), not measured.

Fig. 3: Analytic composition of the serialized artifact. Green segments are the BF16 embedding, output head, and normalization layers; the percentages give their share of the quantized artifact. Each label compares the estimated reduction with the measured one in Table VII .

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 280 papers

May 2, 2026

Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels

Large Language Models are routinely compressed via post-training quantization to reduce inference costs and memory footprint for cloud and edge deployment, yet the impact of this compression on model quality remains poorly understood. Existing studies typically compare only two conditions (full-precision vs. a single quantized variant), rely on aggregate bias metrics, and evaluate a single model family, making it impossible to distinguish gradual degradation from threshold-dependent safety failures. We conduct a controlled empirical study of three instruction-tuned models (Qwen2.5-7B, Mistral-7B, Phi-3.5-mini) at five precision levels (BF16 through 3-bit) on 12,148 BBQ bias benchmark items across 5 random seeds, totaling 911,100 inference records. Our results reveal that 3-bit quantization causes 6-21% of previously unbiased items to develop new stereotypical behaviors, following a clear dose-response pattern confirmed via logistic regression, while models' willingness to select "unknown" answers declines by 17.4%. Crucially, these item-level changes are invisible to standard quality metrics: perplexity increases by less than 0.5% at 8-bit and under 3% at 4-bit across all three models, yet 2.5-5.6% of items already develop new biases at 4-bit. These findings demonstrate that aggregate evaluation metrics systematically miss fairness-critical degradation, underscoring the need for quality-aware compression protocols that explicitly test for bias emergence before deployment.

Plawan Kumar Rath, Rahul Maliakkal

Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment

Ultra-low-bit language models can reduce storage and memory bandwidth, but a nominal "1.58-bit" label does not fully describe the stored representation, retained capability, or runtime behavior. We study an end-to-end post-training conversion of Qwen, an instruction-tuned 4B-parameter model, using KOTMS rotation, E2M-ATQ ternarization, and GPTQ-style error compensation from TWLA. The experiment is weight-only: activations remain at 16-bit precision, so ILA-AMP is omitted. We evaluate effective bit accounting, task capability retention, perplexity, calibration sensitivity, checkpoint composition, and deployment behavior. The final conversion uses 1.641 effective bits per weight for quantized linear weights, with 81.62% of model parameters targeted. Across ten scored capability comparisons, accuracy falls from 64.5% to 54.7%. Degradation is uneven: BoolQ retains 84.6% chance-corrected teacher performance, while ARC-Challenge retains 43.8%. Perplexity rises from 13.639 to 18.748 on WikiText-2, 24.700 to 31.992 on PTB, and 19.831 to 28.966 on C4. A subsequent packing run preserves the ternary planes and scales, reducing reported model size from 8.29 GiB to 3.96 GiB with essentially unchanged perplexity. A separate third-party packing attempt was lossy and is excluded from the primary artifact claim. The packed artifact has not been benchmarked end-to-end for task accuracy or generation throughput. A preliminary Triton GEMV microbenchmark is 4.6x slower than FP16 cuBLAS on one tested shape. We therefore do not claim that compression alone yields faster inference.

Anirudh Malik, M Sparsh Mehra, Poojith Devan

OneBit AI

LLM Quantization Ternary Quantization LLM Compression Weight-Only Quantization Post-Training Quantization

Ask PDF

Apr 22, 2026

FASQ: Flexible Accelerated Subspace Quantization for Calibration-Free LLM Compression

Compressing large language models (LLMs) for deployment on commodity GPUs remains challenging: conventional scalar quantization is limited to fixed bit-widths (e.g., 8/4/3-bit), offers only a few discrete compression points, and typically requires calibration data. We present FASQ (Flexible Accelerated Subspace Quantization), a calibration-free framework that applies product quantization to LLM weight matrices. By tuning two parameters, sub-vector size and codebook cardinality, FASQ exposes a continuous design space spanning 27-49% of the original FP16 model size, filling compression gaps that fixed-bit schemes cannot reach. On Meta-Llama-3-8B, FASQ surpasses 4-bit GPTQ and AWQ in accuracy (67.1-67.7 avg.) at 37-42% model size, with consistent results on Qwen3-8B and Qwen3.5-9B-Base. To make product quantization practical at inference time, we design custom CUDA kernels: a LUT-free direct-compute GEMV for decode and an output-stationary double-buffered LUT GEMM for prefill, both with split-K parallelism. On an RTX~3090, FASQ achieves 45.2 tok/s decode at effective 4-bit (2.56x memory reduction) and 51.8 tok/s at effective 3-bit (2.80x), both surpassing FP16 tensor-core performance (43.9 tok/s) and delivering 1.6 to 1.8x the throughput of AWQ, 2.5 to 2.5x of GPTQ, and 4.3 to 5x of RTN. FASQ is the only compressed method that accelerates decode beyond FP16, offering calibration-free compression, continuous size-quality trade-offs, and real-time inference on a single consumer GPU.

Ye Qiao, Yian Wang, Zhiheng Chen +2

University of California, Irvine Irvine, California, USA

LLM Compression GPU Acceleration Vector Quantization LLM Inference Acceleration Product Quantization Post-Training Quantization+1