cs.AINov 16, 2026

DNeo: Your Parallel Draft Model Needs Better Target Feature Encoding

Authors: Jameson Sandler, Shane Bergsma, Mostafa Elhoushi, Bin Claire Zhang, Gavia Gray, Joel Hestness, Nolan Dey, Ferdinando Fioretto

Organizations: University of Virginia · Cerebras Systems

Abstract

Speculative-diffusion decoding uses a small parallel drafter model to generate b draft tokens in one forward pass, which then undergo parallel verification via a larger target model. Faster inference can be achieved when multiple draft tokens pass verification simultaneously. Recent parallel draft architectures (DFlash, DSpark) improve draft quality by conditioning on rich features computed by the target, through a mechanism known as feature encoding. However, standard feature encoding mechanisms have three limitations: (1) they are lossy, applying encoding projections that are destructive to the underlying target feature-space via a discovered effect known as residual mixing, (2) they are memory-less, with no record of previously processed proposals, and (3) they utilize features sampled uniformly from across the target layers, omitting important hidden states. To address these issues, we propose Mixture-of-Features Attention (MoFA), a lightweight feature-encoding approach equipped with a working memory that encodes previously rejected proposals. Leveraging MoFA, rejection memory, and a set of further enhancements, including 2-tap causal convolutions, prefix-only cross-attention, and non-uniform target layer selection, the final architecture, DNeo, achieves 1.3x faster generation speed than SoTA (DFlash-2) without impacting drafter footprint or generation quality. DNeo marks a new state of the art in high quality, fast LLM inference.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. D3D: Decoupled Diffusion Drafting for Asynchronous Speculative Decoding

    Nov 4, 2026Baoyou Chen, Ruiqiao Mei, Guancong Lin +4Speculative DecodingSpeculative Decoding for Diffusion Language Models

  2. Give the Drafter a Second Chance to Guess: Parallel Reconsideration for Speculative Decoding

    Nov 3, 2026Hoang Anh Duy Le, Huibing Dong, Alireza Khadem +6Speculative DecodingLLM Inference Acceleration

  3. DAWN: Dependency-Aware Fast Inference for Diffusion LLMs

    Feb 6, 2026Lizhuo Luo, Zhuoran Shi, Jiajun Luo +4Diffusion Language Model InferenceDiffusion Language Model Decoding