RAC: Rectified Flow Auto Coder

Sen Fang; Yalin Feng; Yanxin Zhang; Yihao Quan; Juyi Lin; Yifan Shen; Ziwei Dong; Sisong Bei

cs.CVMar 6, 2026

RAC: Rectified Flow Auto Coder

Authors: Sen Fang, Yalin Feng, Yanxin Zhang, Yihao Quan, Juyi Lin, Yifan Shen, Ziwei Dong, Sisong Bei,

Abstract

In this paper, we propose a Rectified Flow Auto Coder (RAC) inspired by Rectified Flow to replace the traditional VAE: 1. It achieves multi-step decoding by applying the decoder to flow timesteps. Its decoding path is straight and correctable, enabling step-by-step refinement. 2. The model inherently supports bidirectional inference, where the decoder serves as the encoder through time reversal (hence Coder rather than encoder or decoder), reducing parameter count by nearly 41%. 3. This generative decoding method improves generation quality since the model can correct latent variables along the path, partially addressing the reconstruction--generation gap. Experiments show that RAC achieves a Pareto improvement over SOTA VAEs, where even a 10 $\times$ parameter-reduced decoder exceeds full-scale VAE performance in both reconstruction and generation quality, validating the effectiveness of our approach.

Figures & tables

Figure 1: The trajectory demonstration of RAC: Make the reconstruction task a condition generation task; Make the decoder the encoder; Make the single-step decoding and encoding a multi-step decoding and encoding.

Figure 2: Method Overview. (i) Training. To prevent latent space collapse, we freeze the VAE encoder and train only the RAC decoder; reverse-time inference then serves as encoding. (ii) State Construction. Extra channels beyond RGB are padded with 0.5, keeping the velocity field shape constant and ensuring bidirectional consistency. (iii) RAC Input. RAC takes time $t$ and the current state as input, driving the transition from latent initialization to the target image.

Figure 3: Reconstruction is Condition Generation: The previous reconstructions were more accurate because they could relatively approach the manifold. The previous generations predicted variables that were often some distance away from the manifold. This is part of the reason for the past differences in the performance of generation and reconstruction. However, our method theoretically aims for perfect reconstruction, and the multi-step decoding can correct the potential variables provided by Unet/DiT. Therefore, both reconstruction and generation are significantly superior to traditional VAEs.

Figure 4: Conceptual and empirical views of RAC generation trajectories. Left: a conceptual illustration of trajectory-based generation from latent space to image space. Right: sampled state trajectories projected into a 2D PCA space.

Panel of Figure 5 — Figure 5: The State Space of RAC is a More Organized Representation Space. (a) We used PCA to conduct a visual analysis of the state space that we encoded. This state space represents the state before the final output of the encoder. We can clearly observe that the training of RAC has made the original Latent representation more orderly and clean. (b) Comparison of gFID-50K on ImageNet 256×256 Deng et al. (2009) across different VAE configurations. We report results for the baseline methods (REPA Yu et al. (2024) , REPA-E Leng et al. (2025) ; Rombach et al. (2022) and EQ-VAE Kouzelis et al. (2025) ), comparing each against our approach trained under identical settings. Lower is better. RAC consistently outperforms the corresponding baselines across all VAE settings.

Autoencoder gFID

\downarrow

sFID

\downarrow

\uparrow

Prec.

\uparrow

Rec.

\uparrow

\Delta

Params SD-VAE Rombach et al. (2022) 24.1 6.25 55.7 0.62 0.60 0.0% +REPA-E Leng et al. (2025) 16.3 5.69 75.0 0.68 0.60 0.0% +RAC (Ours) 14.7 5.51 78.4 0.70 0.61 -40.8% IN-VAE Rombach et al. (2022) ; Leng et al. (2025) 22.7 5.47 56.0 0.62 0.62 0.0% +REPA-E Leng et al. (2025) 12.7 5.57 84.0 0.69 0.62 0.0% +RAC (Ours) 11.2 5.21 87.5 0.71 0.63 -40.7% VA-VAE Yao et al. (2025) 12.8 6.47 83.8 0.71 0.58 0.0% +REPA-E Leng et al. (2025) 11.1 5.31 88.8 0.72 0.61 0.0% +RAC (Ours) 9.8 5.08 91.4 0.73 0.62 -40.7%

Decoder rFID GFLOPs SD-VAE-0.1x 0.44 94.3 SD-VAE-0.2x 0.39 152.4 SD-VAE-0.3x 0.35 254.5 SD-VAE 0.62 310.4 Scalable Decoders: improve rFID while remaining much more efficient than VAEs. 0.1x indicates that the parameter being used is one-tenth of the SD.

Table 1: VAE architecture and decoder scale on ImageNet-1K. Left: generation quality across VAE backbones and training configurations. Right: reconstruction quality (rFID) and computation (GFLOPs) across RAC decoder scales. RAC consistently improves quality while reducing parameters or computation. Default settings are highlighted in gray . See App. C.3 for generation.

Figure 6: Qualitative Results on Imagenet 256 $\times$ 256 using RAC. It can achieve significant results in the state reconstruction mode with just 30k steps.

Figure 7: rFID vs. Training Steps under Different Inference Steps. Results on a lightweight SD-VAE structure for 1, 4, 8, and 16 decoding steps. Lower is better.

Figure 8: Generative Decoding under Ultra-short Training: To visually demonstrate the benefits of our generative decoding on the results, we only trained for 1k steps (this is merely 1/10k of the full training duration in other works), and then we systematically observed the number of decoding inferences impacts performance. It’s clearly see that during the initial training phase, multi-step decoding significantly contributed to the improvement in quality compared to single-step baselines.

Panel of Figure 9 — Figure 9: Parameter Efficiency of Bidirectional Design. A conventional VAE-style bidirectional model uses separate encoder and decoder backbones to map between compact latent and image spaces. In contrast, RAC uses a shared bidirectional model in the full-resolution state space, achieving encoding and decoding by reversing the flow direction rather than duplicating the backbone.

Diff. Model	gFID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$
SiT-B (130M)	49.5	7.00	27.5	0.46	0.59
+REPA-E	34.8	6.31	39.1	0.57	0.59
+RAC (Ours)	32.1	6.18	41.3	0.59	0.60
SiT-L (458M)	24.1	6.25	55.7	0.62	0.60
+REPA-E	16.3	5.69	75.0	0.68	0.60
+RAC (Ours)	14.7	5.51	78.4	0.70	0.61

Table 2: Model scale comparison on ImageNet 256

\times

256. Our RAC improves generation quality across all scales, achieving consistent gFID reductions over both the vanilla baseline and REPA-E.

Figure 10: Encoding and Decoding Trajectory Symmetry. Left: RAC decodes the full-resolution state to the image through a sequence of intermediate states. Right: the trajectories of sampled spatial positions are projected to a 2D PCA space, revealing that the decoding process follows structured and coordinated paths in the learned state space rather than independent pixel-wise changes.

Panel of Figure 11 — Figure 11: Mechanism-level ability of RAC. Left: reconstruction errors during training. Decoding teacher latents with RAC simulates frozen-backbone generation, while state reconstruction measures the upper bound of reconstruction within RAC’s shared full-resolution state space. Right: qualitative state reconstruction examples.

Method	gFID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$
100K Iterations (20 Epochs)
REPA Yu et al. (2024)	19.40	6.06	67.4	0.64	0.61
REPA-E (scratch)	14.12	7.87	83.5	0.70	0.59
RAC (scratch, 1-step)	14.05	7.83	83.9	0.70	0.61
RAC (scratch, 4-step)	13.47	7.31	86.2	0.71	0.61

Table 3: Training and decoding-step ablation: We compare training configurations and decoding steps under 100K iterations of end-to-end training from scratch, evaluating their contributions to generation quality.

Panel of Figure 12 — Figure 12: Baseline latent manifolds show architecture-dependent anisotropy, with principal components dominated by high-frequency texture or coarse low-frequency blocks. RAC reallocates variance to spatially coherent components, yielding cleaner, more content-aligned latent geometry while preserving scene identity.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Generative decoder	Independent encoder	Full sharing	Full reversal	Explicit round-trip loss	Representation / compatibility
Epsilon-VAE Zhao et al. (2025)	Conditional diffusion	Yes	No	No	No	Encoder-conditioned continuous latent
DiTo Chen et al. (2025)	Diffusion decoder	Yes	No	No	No	Jointly learned continuous tokenizer
SSDD Vallaeys et al. (2025)	Diffusion + distillation	Yes	No	No	No	KL-VAE replacement
Diffusion Autoencoders Preechakul et al. (2022)	DPM/DDIM	Semantic encoder	Partial	Stochastic code only	No	Two-part semantic/ stochastic code
FlexTok Bachmann et al. (2025)	Rectified flow	Yes	No	No	No	Variable-length 1D tokens
SEMANTICIST Wen et al. (2025)	Conditional diffusion	Yes	No	No	No	Structured 1D causal tokens

Appendix

Table 4: Mechanism-level comparison with generative-decoder and diffusion-tokenizer methods. Full sharing means that the complete encoding and decoding transformations use one parameter set; full reversal means that the complete encoded state is obtained by reversing the same field used for decoding.

Method	Decoder params.	Total params.	Decode NFE	PSNR $\uparrow$	SSIM $\uparrow$
FlexTok-d12-d12 Bachmann et al. (2025)	172.45M	341.65M	20	17.152	0.4598
SEMANTICIST-L Wen et al. (2025)	457.05M	609.76M	250	22.199	0.6521
SSDD-S-F8C4 Vallaeys et al. (2025)	14.06M	48.22M	8	19.863	0.5126
RAC-Tiny (ours)	1.46M	1.46M	4	23.917	0.7063

Appendix

Table 5: Pixel-fidelity comparison using released checkpoints. All methods use the same held-out inputs and evaluation code. Decoder and total counts distinguish decoder parameters from the complete system; RAC shares one network across both directions.

Relative $\sigma$	TAESD PSNR $\uparrow$	RAC-4 PSNR $\uparrow$	RAC paired gain $\uparrow$	TAESD $\Delta$ PSNR	RAC-4 $\Delta$ PSNR	RAC robustness gain $\uparrow$	Output-drift reduction $\uparrow$
0.0	24.157	24.694	$+0.537\pm 0.060$	0.000	0.000	0.000	–
0.1	23.675	24.355	$+0.680\pm 0.054$	$-0.482$	$-0.339$	$+0.143\pm 0.011$	80.2%
0.2	22.469	23.441	$+0.972\pm 0.047$	$-1.688$	$-1.253$	$+0.435\pm 0.026$	70.1%
0.3	20.927	22.154	$+1.226\pm 0.045$	$-3.229$	$-2.540$	$+0.690\pm 0.037$	60.8%
0.5	17.812	19.141	$+1.329\pm 0.050$	$-6.344$	$-5.552$	$+0.792\pm 0.048$	44.6%

Appendix

Table 6: Robustness to relative Gaussian perturbations of each method’s native latent. PSNR gains and degradation differences are paired across the same images. The final column reports the reduction in output-drift MSE achieved by RAC relative to TAESD.

Decoder	gFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	GFLOPs $\downarrow$
SD-VAE-0.1 $\times$	6.98	126.4	0.70	0.65	94.3
SD-VAE-0.2 $\times$	6.86	127.9	0.71	0.65	152.4
SD-VAE-0.3 $\times$	6.74	129.2	0.71	0.66	254.5
SD-VAE Zheng et al. (2026)	7.13	124.9	0.70	0.65	310.4

Appendix

Table 7: Scaled RAC generation with frozen DiT-XL. All settings use 80 epochs without CFG; GFLOPs measure decoder cost.

Method	NFE	Total params.	B1 latency	B8 latency/image	B1 peak memory
Standard SD-VAE	1	83.65M	$29.059\pm 0.117$ ms	31.144 ms	319.2 MiB
RAC (default)	4	49.51M	$106.421\pm 0.284$ ms	125.659 ms	319.4 MiB

Appendix

Table 8: Decoder-local efficiency under matched hardware and precision. Batch-8 latency is reported per image.

Autoencoders

Autoencoders learn compressed representations by encoding data into latent codes and decoding them for reconstruction, generation, denoising, or anomaly detection.

See all 152 papers

RefDecoder: Enhancing Visual Generation with Conditional Video Decoding
May 14, 2026Xiang Fan, Yuheng Wang, Bohan Fang +2Image-to-Video Generation Conditional Variational Autoencoder
University of Washington · University of North Carolina at Chapel Hill
Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generation
Feb 22, 2026Lunjie Zhu, Yushi Huang, Xingtong Ge +5Channel Pruning Variational Autoencoders
iComAI Lab, Hong Kong University of Science and Technology, Hong Kong SAR, China · School of Data Science, Lingnan University, Hong Kong SAR, China
DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders
May 21, 2026Tianhang Wang, Yitong Chen, Wei Song +3Latent Diffusion Models Autoencoders
Zhejiang University · Shanghai Innovation Institute · Fudan University +1

May 14, 2026J/K move · Enter open · S save

RefDecoder: Enhancing Visual Generation with Conditional Video Decoding

Xiang Fan, Yuheng Wang, Bohan Fang, Zhongzheng Ren+1

University of Washington · University of North Carolina at Chapel Hill

Image-to-Video Generation Conditional Variational Autoencoder Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Video generation powers a vast array of downstream applications. However, while the de facto standard, i.e., latent diffusion models, typically employ heavily conditioned denoising networks, their decoders often remain unconditional. We observe that this architectural asymmetry leads to significant loss of detail and inconsistency relative to the input image. To address this, we argue that the decoder requires equal conditioning to preserve structural integrity. We introduce RefDecoder, a reference-conditioned video VAE decoder by injecting high-fidelity reference image signal directly into the decoding process via reference attention. Specifically, a lightweight image encoder maps the reference frame into the detail-rich high-dimensional tokens, which are co-processed with the denoised video latent tokens at each decoder up-sampling stage. We demonstrate consistent improvements across several distinct decoder backbones (e.g., Wan 2.1 and VideoVAE+), achieving up to +2.1dB PSNR over the unconditional baselines on the Inter4K, WebVid, and Large Motion reconstruction benchmarks. Notably, RefDecoder can be directly swapped into existing video generation systems without additional fine-tuning, and we report across-the-board improvements in subject consistency, background consistency, and overall quality scores on the VBench I2V benchmark. Beyond I2V, RefDecoder generalizes well to a wide range of visual generation tasks such as style transfer and video editing refinement.

Abstract

Autoencoders

Explore similar work

RefDecoder: Enhancing Visual Generation with Conditional Video Decoding

Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generation

DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders