cs.CLJan 30, 2026

Inference-time Alignment via Sparse Junction Steering

Authors: Runyi Hu, Jie Zhang, Shiqian Zhao, Jiale Meng, Jiwei Li, Jason Zeng, Ming Wu, Michael Heinrich, 

Organizations: Nanyang Technological University · CFAR and IHPC, A⋆STAR, Singapore · Zhejiang University · 0G Labs

Abstract

Token-level steering has emerged as a pivotal approach for inference-time alignment, enabling fine-grained control over large language models (LLMs) by modulating their output distributions without parameter updates. While effective, existing methods rely on dense intervention at every decoding step. This persistent manipulation not only incurs substantial computational overhead but also risks compromising generation quality by excessively drifting from the model's intrinsic distribution. In this work, we show that dense intervention is unnecessary and propose Sparse Inference-time Alignment (SIA), which performs sparse junction steering, intervening only at critical decision points along the generation trajectory. Our key insight is that high-entropy junctions are empirically consistent with pivotal decision points in the generation trajectory and are particularly susceptible to misalignment, suggesting that these points benefit from alignment-related reward signal. Extensive experiments across different model families and alignment objectives show that steering only 20%-80% of tokens achieves the superior alignment-efficiency trade-offs. For strong base models such as Qwen3, intervening on as few as 20% of tokens matches or even surpasses heavily post-trained instruct models. This sparsity enables stronger guidance while better preserving the model's native distribution, integrates seamlessly with search-based methods (e.g., Best-of-N), and improves efficiency by reducing the required sampling budget while yielding practical wall-clock speedups under matched-quality comparisons.

Explore similar work

CardsList
  1. Gradient-Guided Reward Optimization for Inference-time Alignment

    Jun 8, 2026Hankun Lin, Ruqi ZhangReward HackingLLM Alignment

  2. To Intervene or Not: Guiding Inference-time Alignment with Probabilistic Model Blending

    Apr 22, 2026Jin Gan, Xin Li, Jun LuoLLM AlignmentInference-Time Language Model Alignment

  3. Trust Your Guide Only When Certain: Uncertainty-Aware Sparse Alignment at Inference Time

    Sep 1, 2026Zeen Zhu, Zhuo Li, Weiyang Guo +4LLM Safety AlignmentInference-Time Language Model Alignment