cs.CVNov 8, 2026

Virtual Task Prompting for Multi-Task Scene Understanding

Authors: Zedong Wang, Yucheng Wang, Dan Xu

Organizations: The Hong Kong University of Science and Technology

Abstract

Multi-task scene understanding models typically rely on modified vision encoders or complex decoders for cross-task interaction, which limits their flexibility and compatibility with off-the-shelf vision foundation models (VFMs). We introduce Virtual Task Prompting (VTP), which frames multi-task prompting as retrieving task-adaptive compositions from a shared virtual state, where both the state and routing process are optimized in an end-to-end manner. Concretely, VTP maintains an expanded set of virtual tasks alongside the backbone. At each layer, learnable routing matrices read task-specific compositions as input to interact with image tokens and write the processed results back, forming explicit pathways that capture the intricacies among tasks outside the encoder. Since VTP keeps the backbone clean, it integrates seamlessly with pretrained VFMs, advancing the state of the art on Pascal-Context and NYUD-v2. Stress test on Taskonomy with 13 heterogeneous tasks further shows VTP's robustness in extreme settings. Ablations and analysis indicate that VTP's gains stem from adaptive routing rather than increased capacity. The project page is available at https://jacky1128.github.io/VTP/.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Task-Instructed Causal Routing of Vision Foundation Models for Multi-Task Learning

    Jun 14, 2026Donghyun Han, Yuseok Bae, Jung Uk Kim +1Vision Foundation Model AdaptationMulti-Task Learning

  2. T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs

    Nov 20, 2025Shao-Jun Xia, Huixin Zhang, Zhengzhong TuVision-Language ModelsIn-Context Learning

  3. Towards Accurate UAV Image Perception: Guiding Vision-Language Models with Stronger Task Prompts

    Dec 8, 2025Mingning Guo, Mengwei Wu, Shaoxian Li +2VLM AdaptationRemote Sensing VLMs