Existing generative models, such as diffusion and auto-regressive networks, are inherently static, relying on a fixed set of pretrained parameters to handle all inputs. In contrast, humans flexibly adapt their internal generative representations to each perceptual or imaginative context. Inspired by this capability, we introduce Composer, a new paradigm for adaptive generative modeling based on test-time instance-specific parameter composition. Composer generates input-conditioned parameter adaptations at inference time, which are injected into the pretrained model's weights, enabling per-input specialization without fine-tuning or retraining. Adaptation occurs once prior to multi-step generation, yielding higher-quality, context-aware outputs with minimal computational and memory overhead. Experiments show that Composer substantially improves performance across diverse generative models and use cases, including lightweight/quantized models and test-time scaling. By leveraging input-aware parameter composition, Composer establishes a new paradigm for designing generative models that dynamically adapt to each input, moving beyond static parameterization.
Figures & tables
Figure 1: Comparison of static versus adaptive parameterization. Composer dynamically composes instance-specific parameter updates, allowing per-input adaptation without fine-tuning.
Figure 2: Overview of Composer. Given any weight matrix W from the backbone, Composer generates a low-rank update W′=W+AB conditioned on the input. Specifically, the query and value matrices WQ and WV from the pretrained model are linearly projected from Rd×d to R2r×dmodel . The projected representations are then separated to initialize tokens Ai0 and Bi0∈R1×dmodel . During training, these tokens are combined with prompt tokens Pi∈R1×dmodel and processed by a transformer to produce W∗=AB . The adapted parameters W′=W+W∗ are used for generation. At inference, the first projection layers are removed, while Ai0 and Bi0 are stored for fast instance-specific adaptation.
Figure 3: Illustration of the attention scheme. Component tokens attend to prompt tokens for context, maintain local block-wise attention, and the first token of each block captures inter-block correlations.
Figure 4: Impact of low-rank dimension r and context-aware sampling parameter α on ImageNet 256×256 class-conditional image generation. (a) Inception Score (IS) vs. r , (b) FID vs. r , (c) IS vs. α , (d) FID vs. α . Higher IS and lower FID indicate better generation quality.
While scaling test-time compute can substantially improve model performance, existing approaches either rely on static compute allocation or sample from fixed generation distributions. In this work, we introduce a test-time compute allocation framework that jointly adapts where computation is spent and how generation is performed. Our method begins with a warm-up phase that identifies easy queries and assembles an initial pool of question-response pairs from the test set itself. An adaptive phase then concentrates further computation on unresolved queries while reshaping their generation distributions through evolving in-context demonstrations -- conditioning each generation on successful responses from semantically related queries rather than resampling from a fixed distribution. Experiments across math, coding, and reasoning benchmarks demonstrate that our approach consistently outperforms existing baselines while consuming substantially less inference-time compute.
Bowen Zuo, Dongruo Zhou, Yinglun Zhu
University of California, Riverside · 2Indiana University Bloomington
Compositional generalization requires models to produce novel configurations from familiar parts. In diffusion models, prior compositional generation methods typically assume that the relevant concepts or conditioning signals are already available. We instead ask whether a pretrained diffusion model can discover query-specific concepts from the time-indexed scores it learns for the noisy marginals pt(xt) and compose them at test time. Given a single out-of-distribution query, our method performs gradient ascent on sθ(xt,t)≈∇xtlogpt(xt) at multiple noising timesteps to recover local density modes, maps these modes into clean-space Gaussians, greedily selects relevant prototypes with a submodular likelihood objective, and combines them into a product-of-experts (PoE) teacher model with an analytic score. This teacher model can be sampled directly through classifier-free guidance or used to generate a sample pool for training a new class embedding and low-rank adapter. On held-out composition benchmarks built from ColorMNIST and CelebA, both the analytic PoE sampler and the low-rank adapted model outperform query-only and nearest trained-class baselines. These results suggest that the time-indexed score geometry of the diffusion model contains reusable density-mode concepts that support test-time compositional generation without a predefined concept library.
Zekun Wang, Anant Gupta, Tianyi Zhu +1
Georgia Institute of Technology · University of Virginia
Large-scale autoregressive models have demonstrated remarkable capabilities in image generation. However, their sequential raster-scan decoding relies on strictly next-token prediction, making inference prohibitively expensive. Existing acceleration methods typically either introduce entirely new generation paradigms that necessitate costly pre-training from scratch, or enable parallel generation at the expense of a training-inference gap or altered prediction objectives. In this paper, we introduce FlashAR, a lightweight post-training adaptation framework that efficiently adapts a pre-trained raster-scan autoregressive model into a highly parallel generator based on two-way next-token prediction. Our key insight is that effective adaptation should minimize modifications to the pre-trained model's original training objective to preserve its learned prior. Accordingly, we retain the original AR head as a horizontal head for row-wise prediction and introduce a complementary, lightweight vertical head for column-wise prediction. To facilitate efficient adaptation, we branch the vertical head from an intermediate layer rather than the final layer, bypassing the inherent horizontal head bias. Moreover, since horizontal and vertical predictions capture complementary dependencies whose relative importance varies across target positions, we employ a learnable fusion gate to dynamically combine the two predictions at each position. To further reduce adaptation cost, we propose a two-stage adaptation pipeline: the vertical head is first initialized through adaptation from the pre-trained autoregressive model before jointly fine-tuned with backbone to adapt to the new decoding paradigm. Extensive experiments on LlamaGen and Emu3.5 show that FlashAR achieves up to a 22.9x speedup for 512x512 image generation through a lightweight post-training with merely 0.05% of the original training data.
Junkang Zhou, Yefei He, Feng Chen +2
Zhejiang University, China · University of Adelaide, Australia