LatentQuant: Preserving the Policy-Facing Latent Contract under NVFP4 VAE Quantization

Ziye Deng; Lufang Chen; Shuyu Feng; Zhenwei Duan; Zicong Ye; Yu Sun; Xiaofan Li; Ruyi Gan

cs.AIOct 2, 2026

LatentQuant: Preserving the Policy-Facing Latent Contract under NVFP4 VAE Quantization

Authors: Ziye Deng, Lufang Chen, Shuyu Feng, Zhenwei Duan, Zicong Ye, Yu Sun, Xiaofan Li, Ruyi Gan,

Organizations: UC San Diego · XSquare Robot · Sun Yat-Sen University · University of Chinese Academy of Sciences ·

Abstract

Recent world action models (WAMs) reuse pretrained video VAEs whose encoder latents directly condition downstream action policies. Quantization must therefore preserve not only reconstruction fidelity but also the policy-facing latent contract expected by the frozen policy. Direct NVFP4 leaves W4A4 quantization error uncompensated, whereas joint quantization-aware training (QAT) can recover reconstruction by moving this representation. On Wan2.1, joint QAT nearly matches FP32 VBench-7 (0.7403 versus 0.7409), yet LIBERO success collapses from 95.5% to 10.5%. Controlled decoder-only experiments show that activation quantize-dequantize operations alter the reconstruction signal and decoder Jacobian, redirecting the gradient returned to the encoder and inducing persistent latent drift. Based on this mechanism, we introduce LatentQuant, a two-stage NVFP4 QAT framework that first aligns the quantized encoder with its high-precision counterpart, then freezes it while adapting the decoder. Across Wan2.1 and Wan2.2, LatentQuant preserves near-baseline control and high reconstruction quality, achieving 95.75% success on LIBERO and 68.8% on RoboTwin. On NVIDIA B300 GPUs, NVFP4 execution achieves 1.17x-1.26x end-to-end VAE speedups over BF16 cuDNN.

Figures & tables

Figure 1: LatentQuant separates encoder alignment from decoder adaptation. Stage I aligns the NVFP4 encoder with a frozen high-precision reference; Stage II freezes this interface and adapts only the NVFP4 decoder, preventing reconstruction gradients from rewriting the policy-facing latent.

Figure 2: Mechanism of latent-contract violation under joint QAT. Low precision progressively shifts the learned latent (a); decoder activation QDQ redirects its reconstruction gradient (b); and the drift persists after removing QDQ, showing consolidation in encoder parameters (c).

VAE	Method	Robot VBench-7 $\uparrow$	LIBERO VBench-7 $\uparrow$	$d_{z}\downarrow$	$d_{a}\downarrow$	Success $\uparrow$
Wan2.1	FP32	0.7409	0.7747	0.000	0.000	95.50
	Direct NVFP4	0.7266	0.7513	0.288	0.033	94.38
	Joint QAT	0.7403	0.7565	4.796	0.270	10.50
	Joint+KL	0.7341	0.7571	1.190	0.118	86.75
	LatentQuant	0.7340	0.7513	0.202	0.018	95.75
Wan2.2	BF16	0.7436	0.7750	0.000	0.000	69.80

Table 1: Main comparison across Wan2.1 and Wan2.2. VBench-7 measures reconstruction quality, while

d_{z}

and

d_{a}

measure latent and action deviations. Lower

d_{z}

and

d_{a}

are better.

Metric	Direct	Freeze-Only	LatentQuant
Robot VBench-7 $\uparrow$	.7266	.7326	.7340
LIBERO VBench-7 $\uparrow$	.7513	.7512	.7513
$d_{z}\downarrow$	.288	.288	.202
$d_{a}\downarrow$	.033	.033	.018

Table 2: Stage-I and boundary ablation on Wan2.1. Freeze-Only omits Stage-I alignment and therefore retains the Direct NVFP4 interface metrics.

Figure 3: Physical-domain validation. (a) Interface deviations on fixed observations. (b) LatentQuant–FP32 differences in three-run mean milestone scores over 15 real-robot tasks: 7 lower, 2 tied, and 6 higher, with median 0. The positive tail uses a nonuniform scale.

Figure 4: Native NVFP4 VAE deployment efficiency on NVIDIA B300.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

VAE	Trainable block	Epochs	Batch/GPU	LR schedule	Objective
Wan2.1	Encoder	1	10	$5\!\times\!10^{-6}\!\rightarrow\!1\!\times\!10^{-6}$	$\mathcal{L}_{\mathrm{align}}$
Wan2.1	Decoder	5	16	$5\!\times\!10^{-6}\!\rightarrow\!1\!\times\!10^{-6}$	$\mathcal{L}_{\mathrm{rec}}$
Wan2.2	Encoder	1	10	$5\!\times\!10^{-6}\!\rightarrow\!1\!\times\!10^{-6}$	$\mathcal{L}_{\mathrm{align}}$
Wan2.2	Decoder	5	12	$5\!\times\!10^{-6}\!\rightarrow\!1\!\times\!10^{-6}$	$\mathcal{L}_{\mathrm{rec}}$

Appendix

Table 3: Two-stage optimization configuration. Stage I updates only the encoder using sample-wise latent alignment. Stage II freezes the aligned encoder and updates only the decoder using the reconstruction objective.

$\lambda$	Val. PSNR (dB)	SD ratio	Max SD	Moment KL	Mean gap
0	29.906	1.922	2.133	0.9296	0.3973
0.03	30.103	1.088	1.121	0.0088	0.0238
0.1	29.969	1.031	1.052	0.0019	0.0235
0.3	29.768	1.009	1.022	0.0009	0.0227
1	29.500	0.999	1.011	0.0008	0.0219
3	29.309	0.994	1.006	0.0009	0.0222

Appendix

Table 4: Wan2.1 soft alignment sweep. Values are measured after 1,000 matched joint-QAT steps on 64 held-out clips. Moment KL and mean gap are lower-is-better; an SD ratio of one matches the pristine FP32 latent spread.

Figure 5: Soft alignment sweep on matched Wan2.1 clips. Increasing the alignment penalty reduces the latent standard-deviation ratio toward the FP32 reference, while held-out reconstruction PSNR decreases once the interface is tightly constrained.

VAE	Video source	Method	SC	BC	TF	MS	DD	AQ	IQ	Mean
Wan2.1	Robot	HP	.9589	.9732	.9802	.9906	.3281	.3514	.6040	.7409
Wan2.1	Robot	Direct	.9408	.9623	.9750	.9864	.3203	.3255	.5760	.7266
Wan2.1	Robot	Ours	.9517	.9673	.9795	.9901	.3359	.3508	.5625	.7340
Wan2.1	LIBERO	HP	.9433	.9638	.9796	.9931	.5000	.4246	.6183	.7747
Wan2.1	LIBERO	Direct	.9319	.9476	.9755	.9889	.4531	.3791	.5830	.7513
Wan2.1	LIBERO	Ours	.9367	.9548	.9792	.9918	.4609	.3723	.5634	.7513

Appendix

Table 5: Complete seven-dimension VBench reconstruction results. The mean is unweighted, and each setting contains 128 reconstructed clips.

VAE	Method	$d_{z}$	Latent rel. $\ell_{2}$	$d_{a}$	Action rel. $\ell_{2}$
Wan2.1	HP	0.000	0.000	0.000	0.000
Wan2.1	Direct	0.288	0.189	0.033	0.068
Wan2.1	Joint	4.796	3.223	0.270	0.548
Wan2.1	J+KL	1.190	0.803	0.118	0.241
Wan2.1	Ours	0.202	0.135	0.018	0.036
Wan2.2	HP	0.000	0.000	0.000	0.000

Appendix

Table 6: Matched interface replay. Relative

\ell_{2}

differences are reported in addition to the normalized deviations used in the main paper. Lower is better.

Method	Spatial	Object	Goal	LIBERO-10	Total
HP	188/200	195/200	195/200	186/200	764/800
Direct	187/200	195/200	190/200	183/200	755/800
Joint	3/200	65/200	0/200	16/200	84/800
J+KL	177/200	180/200	160/200	177/200	694/800
Ours	190/200	194/200	191/200	191/200	766/800

Appendix

Table 7: Wan2.1 LIBERO success by suite.

Task	HP	D	O	Task	HP	D	O
adjust_bottle	9/10	10/10	9/10	place_can_basket	4/10	4/10	3/10
beat_block_hammer	9/10	9/10	10/10	place_cans_plasticbox	10/10	10/10	10/10
blocks_ranking_rgb	5/10	2/10	2/10	place_container_plate	9/10	9/10	9/10
blocks_ranking_size	1/10	1/10	1/10	place_dual_shoes	7/10	7/10	3/10
click_alarmclock	8/10	8/10	8/10	place_empty_cup	9/10	9/10	8/10
click_bell	10/10	10/10	10/10	place_fan	9/10	9/10	10/10

Appendix

Table 8: Wan2.2 RoboTwin success by task. Each task uses 10 rollouts. The two halves report high precision ( HP ), Direct NVFP4 ( D ), and two-stage QAT ( O ).

Figure 6: Representative Wan2.2 open-loop action traces. Blue and red report mean absolute action error against the same reference trajectory for the BF16 and two-stage NVFP4 VAEs; purple reports their direct prediction difference.

	FP32				LatentQuant
Task	R1	R2	R3	Mean	R1	R2	R3	Mean	$\Delta$
put_spoon_to_bowl	2.00	2.00	0.00	1.33	0.00	0.00	2.00	0.67	-0.67
put_glasses_on_woodshelf	0.00	6.00	4.00	3.33	0.00	0.00	0.00	0.00	-3.33
put_items_into_drawer	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
pick_items_into_basket	19.25	12.75	21.75	17.92	22.00	8.50	13.50	14.67	-3.25
pick_fruits_into_basket	13.00	15.00	11.00	13.00	11.00	32.00	31.00	24.67	+11.67

Appendix

Table 9: Complete real-robot evaluation breakdown. Each task is evaluated over three independent rounds under the same physical-robot protocol. Each entry is the accumulated task score (maximum 100). “Mean” averages the three rounds, and

\Delta

denotes LatentQuant minus FP32.

Panel of Figure 7 — Figure 7: Layerwise forward and backward transport. Top: local QDQ injection, accumulated reconstruction difference, isolated and cumulative loss changes, and signed telescoping increments. Bottom: isolated and cumulative gradient changes and their transport to the latent through the decoder VJP. Shaded regions show variation across the paired clips.

Quantity	Measurement
Local activation perturbation	$9.708\%$
First-convolution output difference	$8.315\%$
Final reconstruction difference	$11.857\%$
Loss increase	$0.006120$
Output-gradient cosine / norm ratio	$0.3155$ / $1.640\times$
Latent-gradient angle / norm ratio	$67.45^{\circ}$ / $1.126\times$

Appendix

Table 10: Measured one-site causal chain. Gradient angles and norm ratios are relative to the matched FP32 branch.

Figure 8: Matched FP32 and strict-NVFP4 training trajectories. The top row reports gradient direction, norm ratio, and the instantaneous radial-force proxy. The bottom row reports latent standard deviation, sample-wise latent drift, and their cross-condition ratio.

Figure 9: Directional visualization of matched gradients. Connected markers follow training order for the FP32 and strict-NVFP4 gradient-direction trajectories in the three-dimensional contrast basis.

Figure 10: Channel-wise strict-NVFP4 minus FP32 training trajectories. The panels report latent standard-deviation ratio, latent-gradient norm, and RMS-directed-force differential for all 16 latent channels, measured every 20 training samples.

VAE	Frames	BF16 enc.	BF16 dec.	NVFP4 enc.	NVFP4 dec.	Total reduction	Speedup
Wan2.1	17	22.970	34.079	19.954	27.978	$16.08\%$	$1.192\times$
Wan2.1	33	40.854	60.905	36.104	50.242	$15.16\%$	$1.179\times$
Wan2.1	65	76.608	113.663	68.378	94.420	$14.45\%$	$1.169\times$
Wan2.2	17	26.316	41.702	23.486	31.524	$19.13\%$	$1.237\times$
Wan2.2	33	43.522	75.752	38.778	56.129	$20.43\%$	$1.257\times$
Wan2.2	65	91.246	146.416	81.915	108.366	$19.97\%$	$1.249\times$

Appendix

Table 11: End-to-end VAE encode–decode latency on NVIDIA B300. Resolution is

208\times 224

, batch size is one, and values are medians in milliseconds.

Figure 11: Stage- and operator-level acceleration over BF16 cuDNN. Panels (a) and (b) report encoder and decoder speedups from the complete temporal sweep. Panel (c) reports a

2.044\times

Wan2.1 Conv2D-region speedup, a

1.305\times

Wan2.1 Conv3D-region speedup, and a

1.458\times

call-count-weighted Wan2.2 convolution speedup; 169 of 188 measured Wan2.2 workloads are faster under NVFP4.

VAE	Metric	BF16	NVFP4	Reduction	Unit
Wan2.1	Resident VAE model state	242.03	95.93	$60.36\%$	MiB
Wan2.1	Full-VAE DRAM traffic	25.176	19.258	$23.51\%$	GiB
Wan2.2	Resident VAE model state	1344.09	410.68	$69.45\%$	MiB
Wan2.2	Full-VAE DRAM traffic	37.127	15.588	$58.01\%$	GiB

Appendix

Table 12: Measured deployment resources. Batch size is one and temporal length is 17 frames. Each row compares BF16 with the packed NVFP4 execution path.

Neural Network Quantization

Neural network quantization represents weights, activations, and states with fewer bits to reduce memory use, computation, and deployment costs.

See all 403 papers

CHASE-VLA: Post-Training Quantization Framework for Vision-Language-Action Models with Chunk-Aware Scale Estimation
Oct 2, 2026Jin Hyun, Jung Gyu Min, Gyuhyun Jung +1Post-Training Quantization VLM Quantization
KAIST, Daejeon, South Korea · POSTECH, Pohang, South Korea
Recovering Low-Bit Failures in Vision-Language-Action Policies via Quantization-Conditioned Reinforcement Learning
Oct 11, 2026Seongmin Park, Jaewon Lee, Yujun Heo +4VLM Quantization Efficient VLA Model Inference
Hanyang University, Republic of Korea · LG Electronics, Republic of Korea
FoldQuantVLA: Native Low-Bit Quantization of Vision-Language-Action Models via Consistent Folding
Sep 21, 2026Hung T. Ho, Khanh D. Nguyen, Quang D. Nguyen +5VLM Quantization Efficient VLA Model Inference
VinRobotics, Vietnam · AICV Lab, EECS Department, University of Arkansas · School of Advanced Manufacturing and Robotics, Peking University, China +2

Oct 2, 2026J/K move · Enter open · S save

CHASE-VLA: Post-Training Quantization Framework for Vision-Language-Action Models with Chunk-Aware Scale Estimation

Jin Hyun, Jung Gyu Min, Gyuhyun Jung, Youngjoo Lee

KAIST, Daejeon, South Korea · POSTECH, Pohang, South Korea

Post-Training Quantization VLM Quantization Efficient VLA Model Inference Activation Quantization

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models map visual observations and language instructions to continuous robot actions, but a diffusion-based action expert (AE) poses a key challenge for low-bit post-training quantization (PTQ). The AE is repeatedly invoked across denoising steps and policy queries, where fixed calibration scales can be mismatched with activation ranges that vary with denoising progress and intended motion. We propose CHASE-VLA, a chunk-aware PTQ method that exploits a VLA-specific signal readily available from the policy: the generated action chunk, including its unexecuted future suffix. Rather than relying only on static scale matching for AE layers, CHASE-VLA combines the previously generated chunk as causal action context with denoising step group information to adapt AE activation scales. This enables W4A4 quantization of both MLP and attention projections in the repeated AE without modifying the pretrained policy. On LIBERO, CHASE-VLA achieves 97.3% average success rate on

π_{0.5}

when both MLP and attention projections in the AE are quantized to W4A4, restoring FP16-level performance. CHASE-VLA also reduces the weight storage of the quantized AE linear layers by 73.4% and their single-chunk memory traffic by 70.9% and 71.2% on

π_{0.5}

and GR00T N1.6, respectively, with a predictor overhead of at most 1.26% of the saved storage.

Abstract

Neural Network Quantization

Explore similar work

CHASE-VLA: Post-Training Quantization Framework for Vision-Language-Action Models with Chunk-Aware Scale Estimation

Recovering Low-Bit Failures in Vision-Language-Action Policies via Quantization-Conditioned Reinforcement Learning

FoldQuantVLA: Native Low-Bit Quantization of Vision-Language-Action Models via Consistent Folding