cs.CVNov 16, 2026

Beyond Token Stacking: Multi-scale Visual Modulation and Adaptive Fused-feature Calibration for MLLMs

Authors: Ye Xiao, Shesheng Song, Yi Ren, Shasha Li, Bin Ji, Xiaodong Liu, Jun Ma, Jie Yu

Organizations: National University of Defense Technology, Changsha, China · Hefei University of Technology, Hefei, China

Abstract

Current Multimodal Large Language Models (MLLMs) have advanced rapidly and follow two main paradigms: one involves concatenating visual and textual tokens for input into Large Language Model (LLM), while the other injects visual knowledge into Feed-Forward Networks (FFNs) of the LLM. However, the former introduces a large number of redundant visual tokens, substantially increasing computational overhead. The latter reduces sequence length but removes images from the input stage entirely, which may hinder the activation of image-related knowledge. Moreover, naively injecting visual knowledge into the FFNs could induce activation distribution shifts due to cross-modal semantic discrepancies. To address these challenges, we develop Beyond Token Stacking (BTS), a novel visual enhancement framework for MLLMs. Specifically, we propose a Multi-scale Visual Modulation (MVM) mechanism that injects multi-scale visual information at the input stage via modulation without increasing the token sequence length. Furthermore, we develop an Adaptive Fused-feature Calibration (AFC) module that performs per-channel affine calibration to decouple visual injection from distribution alignment in the FFN layers, improving multimodal fusion stability while preserving language priors. Experiments show that our approach achieves state-of-the-art performance.

Figures & tables

Explore similar work

CardsList
  1. Late-Layer Fusion is Enough: Dual-Path Vision Token Routing for Multimodal Large Language Models under Visual Saturation

    Jun 8, 2026Siyuan Liu, Jinyang WuEfficient Multimodal InferenceMultimodal Large Language Models

  2. CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

    Apr 14, 2026Yunkai Dang, Yizhu Jiang, Yifan Jiang +4Visual Token PruningMultimodal Large Language Models

  3. Mema: Memory-Augmented Adapter for Enhanced Vision-Language Understanding

    Feb 28, 2026Ying Liu, Yudong Han, Kean Shi +1Memory-Augmented VLMsMultimodal Large Language Models