ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs

Yang Yang; Qinyu Zhao; Mouxiang Chen; Xiaohui Li; Lixin Gu; Wenhai Wang; Hongjie Zhang; Wenwei Zhang

cs.CVAug 4, 2026

ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs

Authors: Yang Yang, Qinyu Zhao, Mouxiang Chen, Xiaohui Li, Lixin Gu, Wenhai Wang, Hongjie Zhang, Wenwei Zhang

Organizations: 1The Australian National University · 2Shanghai AI Laboratory · 3Zhejiang University · 4Shanghai Jiao Tong University · 5Nanjing University

Abstract

Existing scaling strategies for Multimodal Large Language Models (MLLMs) typically expand either model parameters or sequential inference computation, incurring substantial memory or latency overhead. More importantly, most existing methods fail to alter the rigid, fixed computation allocation between the Vision Transformer and the Large Language Model components, limiting task-specific optimization. To address this, we introduce the Parallel Vision-Language (ParVL) scaling framework for MLLMs, which scales parallel computation by reusing the existing ViT and LLM backbone parameters across multiple vision and language branches. This framework raises a central question: given a fixed backbone parameter budget, how should additional shared-backbone computation be allocated between the vision and language modalities? We instantiate each parallel computational stream with branch-specific prefix parameters over a shared backbone, and train the entire model end-to-end via full-parameter supervised fine-tuning on roughly 13B tokens. We systematically study the computation-allocation trade-off between the ViT encoder and LLM decoder. ParVL improves overall multimodal performance over same-recipe single-branch baselines, and the best evaluated vision--language allocation varies across tasks. Code is available at https://github.com/YangYangGirl/ParVL.

Explore similar work

Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs
Jul 22, 2026Pengcheng Wang, Zhiquan Wang, Jayoung Lee +5High Inference Cost
AVIS: Adaptive Test-Time Scaling for Vision-Language Models
Jun 10, 2026Ahmadreza Jeddi, Minh Ngoc Le, Amirhossein Kazerouni +8Large Vision Language Models Visual Reasoning
FlowTrain: Flow-Based Decoupled Training for Industrial-Grade Vision-Language Models
Jun 22, 2026Zhida Jiang, Zhaolong Xing, Yang Pei +14Large Vision Language Models Large Models

Jul 22, 2026 · cs.CVJ/K move · Enter open · S save

Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

Pengcheng Wang, Zhiquan Wang, Jayoung Lee, Zhuoyan Xu+4

Purdue University, West Lafayette, IN, USA · University of Wisconsin–Madison, Madison, WI, USA · NVIDIA, Santa Clara, CA, USA

Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance across vision-language tasks. However, their high inference cost, arising from both the large number of input visual tokens and the heavy computation of the large language model (LLM), remains a key barrier to practical deployment. Recent work attempts to reduce the cost by adaptively optimizing individual dimensions, e.g., pruning redundant visual tokens or skipping LLM layers and heads. Nonetheless, prior approaches typically treat these dimensions independently and overlook a fundamental coupling: the available compute resources must be dynamically allocated across all dimensions based on the input content. To bridge the gap, we propose SmartVL, a unified adaptive inference framework that jointly controls vision token number and model compute capability in response to varying input contents and compute budgets. SmartVL introduces a vision-side token controller that dynamically selects informative visual tokens and an LLM-side compute controller that adaptively adjusts LLM computation. Importantly, these controllers are trained to coordinate with each other so that the overall inference cost satisfies a target budget. To allow this joint scheduling, we connect the controllers using a shared budget encoding and leverage a differentiable latency estimator for end-to-end training. This design enables SmartVL to learn cross-stage allocation strategies that adapt to both input complexity and runtime compute constraints. Experiments across multiple MLLM benchmarks demonstrate that, with joint scheduling, SmartVL consistently outperforms prior adaptive methods and achieves superior accuracy-efficiency Pareto frontiers. Project page: https://www.schaterji.io/publications/2026/jointtokencompute.

High Inference Cost

Open paper Ask about it PDF

Abstract

Explore similar work

Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

AVIS: Adaptive Test-Time Scaling for Vision-Language Models

FlowTrain: Flow-Based Decoupled Training for Industrial-Grade Vision-Language Models