cs.CV · 2605.00503 Copy arXiv ID · May 1, 2026 Save End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer Authors: Wenda Chu , Bingliang Zhang , Jiaqi Han , Yizhuo Li , Linjie Yang , Yisong Yue , Qiushan Guo
Organizations: ByteDance Seed · California Institute of Technology · Stanford University
Abstract Autoregressive image modeling relies on visual tokenizers to compress images into compact latent representations. We design an end-to-end training pipeline that jointly optimizes reconstruction and generation, enabling direct supervision from generation results to the tokenizer. This contrasts with prior two-stage approaches that train tokenizers and generative models separately. We further investigate leveraging vision foundation models to improve 1D tokenizers for autoregressive modeling. Our autoregressive generative model achieves strong empirical results, including a state-of-the-art FID score of 1.48 without guidance on ImageNet 256x256 generation.
Explore similar work Jun 30, 2026 · Bin Lin, Zheyuan Liu, Chenguo Lin +8 Autoregressive Image Generation Generative Models
Apr 27, 2026 · Maitreya Patel, Jingtao Li, Weiming Zhuang +2 Autoregressive Image Generation Autoregressive Generation
May 7, 2026 · Bowen Zheng, Weijian Luo, Guang Yang +2 Autoregressive Image Generation Autoregressive Model
Jun 30, 2026 · cs.CV J/K move · Enter open · S save
Bin Lin, Zheyuan Liu, Chenguo Lin, Sixiang Chen +7
Visual generative models are typically trained in two stages. A tokenizer is first trained for reconstruction and then frozen, after which a generator is trained on its discrete indices or continuous latents. This decoupling leaves the tokenizer unaware of what the generator finds easy to model. We present GEAR (Guided End-to-end AutoRegression), which trains a vector-quantized (VQ) tokenizer and an autoregressive (AR) generator jointly and end-to-end, guided by representation alignment. The key obstacle is that the VQ index fed to the AR model is non-differentiable, so gradients cannot reach the tokenizer, and a straight-through estimator collapses. GEAR resolves this with a dual read-out of the codebook assignment. A hard, one-hot branch trains the AR with next-token prediction, while a differentiable soft branch carries a representation-alignment loss that flows back to guide only the tokenizer. The AR model thereby steers its tokenizer toward an index distribution it can predict more easily. This shifts the alignment burden from the tokenizer to the AR: the tokenizer's own features become less DINOv2-like while the AR's become more so, the opposite of diffusion-side recipes that make the latent itself semantic. GEAR speeds up ImageNet gFID convergence by up to 10x relative to the strong LlamaGen-REPA baseline, learns markedly better patch-level and spatially-coherent features, and generalizes across quantizers (VQVAE, LFQ, IBQ) and to text-to-image generation.