cs.LGJan 30, 2026

TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification

Authors: Haoyun Jiang, Junqi He, Feng Hong, Xinlong Yang, Jianwei Zhang, Zhengyang Zhuge, Zheng Li, Xiaofeng Cao, 

Organizations: CMIC, Shanghai Jiao Tong University · Qwen Team, Alibaba Group · Hong Kong Baptist University

Abstract

Inference efficiency in Large Language Models (LLMs) is fundamentally limited by their serial, autoregressive generation, especially as reasoning becomes a key capability and response sequences grow longer. Speculative decoding (SD) offers a powerful solution, providing significant speed-ups through its lightweight drafting and parallel verification mechanism. While existing work has nearly saturated improvements in draft effectiveness and efficiency, this paper advances SD from a new yet critical perspective: the verification cost. We propose TriSpec, a novel ternary SD framework that, at its core, introduces a lightweight proxy to significantly reduce computational cost by approving easily verifiable draft sequences and engaging the full target model only when encountering uncertain tokens. TriSpec can be integrated with state-of-the-art SD methods like EAGLE-3 to further reduce verification costs, achieving greater acceleration. Extensive experiments on the Qwen3 and DeepSeek-R1-Distill-Qwen/LLaMA families show that TriSpec achieves up to 35% speedup over standard SD, with up to 50% fewer target model invocations while maintaining comparable accuracy.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

    Apr 8, 2026Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6Speculative DecodingLLM Inference Acceleration

  2. Entropy-Aware Token Rejection for Improving Speculative Decoding

    Dec 29, 2025Tiancheng Su, Meicong Zhang, Guoxiu HeSpeculative DecodingLLM Inference Acceleration

  3. SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

    Mar 19, 2026Shenggui Li, Chao Wang, Yikai Zhu +14Speculative DecodingLLM Inference Acceleration