cs.CVNov 12, 2026

UniSpark: Exploring Hybrid Attention for Multimodal Understanding and Generation

Authors: Jiancheng Pan, Yuzheng Liu, Hengyuan Cao, Qiyuan Zhang, Yangyan Li, Biao Gong, Zheng Zhang, Ka Long Cheng, 

Abstract

We present UniSpark-9B, a unified multimodal model that investigates hybrid attention for visual understanding and generation. Built on a Mixture-of-Transformer-Experts backbone with approximately 9B parameters in each branch, UniSpark combines full attention for explicit token interactions with GatedDeltaNet linear attention for compact context propagation. Autoregressive text prediction and visual flow matching jointly support visual question answering, text-to-image generation, instruction-guided editing, and interleaved text-image prediction. In interleaved generation, noisy visual targets are isolated from subsequent predictions, while clean visual representations provide context for later outputs. A blockwise protocol enforces this distinction through attention masks in full-attention layers and recurrent-state retention rules in linear-attention layers. The selected training sources contain 159.7M generation samples, 213.4M understanding samples, and 624.8K interleaved sequences. A four-stage curriculum jointly trains multimodal understanding and visual generation, refining answer accuracy and instruction following while extending image synthesis to 2K. UniSpark scores 2470.3 on MME-S and 0.83 on GenEval. Qualitative examples further illustrate its capabilities in interleaved image generation, video synthesis, and video editing. Together, these results demonstrate that hybrid attention provides an effective backbone for unified multimodal understanding and generation.

Figures & tables

Explore similar work

CardsList
  1. Steering Visual Generation in Unified Multimodal Models with Understanding Supervision

    May 7, 2026Zeyu Liu, Zanlin Ni, Yang Yue +5Unified Multimodal ModelsImage Generation

  2. Lance: Unified Multimodal Modeling by Multi-Task Synergy

    May 18, 2026Fengyi Fu, Mengqi Huang, Shaojin Wu +10Unified Multimodal ModelsMulti-Task Learning

  3. Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification

    Jun 16, 2026Wujian Peng, Lingchen Meng, Yuxuan Cai +7Unified Multimodal ModelsAutoregressive Image Generation