Test-Time Computing for Referring Multimodal Large Language Models

Mingrui Wu; Hao Chen; Jiayi Ji; Xiaoshuai Sun; Zhiyuan Liu; Liujuan Cao; Ming-Ming Cheng; Rongrong Ji

cs.CVFeb 23, 2026

Test-Time Computing for Referring Multimodal Large Language Models

Authors: Mingrui Wu, Hao Chen, Jiayi Ji, Xiaoshuai Sun, Zhiyuan Liu, Liujuan Cao, Ming-Ming Cheng, Rongrong Ji

Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Zhongguancun Academy, Beijing, China.100094. · Tsinghua University. · VCIP, CS, Nankai University.

Abstract

We propose ControlMLLM++, a novel test-time adaptation framework that injects learnable visual prompts into frozen multimodal large language models (MLLMs) to enable fine-grained region-based visual reasoning without any model retraining or fine-tuning. Leveraging the insight that cross-modal attention maps intrinsically encode semantic correspondences between textual tokens and visual regions, ControlMLLM++ optimizes a latent visual token modifier during inference via a task-specific energy function to steer model attention towards user-specified areas. To enhance optimization stability and mitigate language prompt biases, ControlMLLM++ incorporates an improved optimization strategy (Optim++) and a prompt debiasing mechanism (PromptDebias). Supporting diverse visual prompt types including bounding boxes, masks, scribbles, and points, our method demonstrates strong out-of-domain generalization and interpretability. The code is available at https://github.com/mrwu-mac/ControlMLLM.

Figures & tables

Figure 1: Comparison between the training method and our test-time computing method. The training method typically requires a large amount of in-domain data for training and cannot generalize to out-of-domain prompts. In contrast, our method can easily adapt to prompts from a new domain in a test-time computing manner.

Figure 2: The attention maps in various layers of the MLLMs, with the numbers indicating the respective layer indices. The top line visualizes the attention between the prompt token “hat” and the visual tokens, while the bottom line visualizes the attention between the context token (mentioned in Sec. 4.2 ) and the visual tokens.

Figure 3: Manipulating attention with various methods: (a), (b), and (c) demonstrate the manipulation of the attention map by adding an adjustment coefficient $\eta$ on the attention map in the first step during the model inference. (d) illustrates the step-by-step editing approach. (e) showcases that optimizing a learnable context tokens (mentioned in Sec. 4.2 ) instead of visual tokens, while (f) presents the results of our method optimizing the learnable latent variable.

Figure 4: The overview of ControlMLLM. With the provided visual prompt, we convert it into a mask, and compute the mask-based energy function between the mask and the pooled attention map. During the inference process, we conduct backpropagation to optimize a learnable latent variable. This process is executed at the 0-th step of model inference and iterated $T$ times.

Figure 5: The text tokens’ contribution to attention maps, where the attention is focused on the answer-start token (

|

START

|

Figure 6: The loss distribution across decoder layers, showing that the text-visual attention is primarily concentrated in the middle layers.

Figure 7: a) An example where ControlMLLM focus on the correct region, yet produce the same unexpected output as LLaVA due. b) Different prompts lead to different optimization performance. Such behavior is likely caused by the model’s overreliance on linguistic priors.

Figure 8: The examples of referring MLLM with four types of visual prompt, including box (a), mask (b), scribble (c) and point (d). The correct referring expressions are marked in green, incorrect referring expressions are marked in red, and hallucinated expressions are marked in orange. Compared to the baseline model, our method enhances interpretability and controllability with visual prompts, while also helping the model mitigate hallucination issues.

Figure 9: Comparison of OCR performance across models. Ferret, though trained with grounding supervision, still struggles with localized understanding on OCR task. In contrast, our method (LLaVA + Ours) accurately identifies the referred textual content.

Models	ROC (In-Domain)				RTC (Out-of-Domain)
Models	Box	Mask	Scribble	Point	Box	Mask
Training Methods:
Kosmos-2 ( Peng et al., 2023 )	55.17	-	-	-	23.12	-
GPT4-ROI ( Zhang et al., 2023b )	58.59	-	-	-	57.17	-
Shikra-7B ( Chen et al., 2023b )	64.60	-	-	56.27	60.41	-
Ferret-7B ( You et al., 2023 )	71.71	72.39	71.58	68.54	58.70	59.81

Table 1: Performance on Referring Object Classification (ROC) and Referring Text Classification (RTC) tasks. ROC prompts follow the template: “ Is the object { location } a { class A } or a { class B } ? ”; RTC prompts: “ Is the text { location } ‘ { text A } ’ or ‘ { text B } ’? Please select only one. ” “–” indicates unsupported input types. Gray numbers are for reference only.

Models	ROC	RTC
LLaVA-1.5 ( Liu et al., 2024 )	54.72	57.42
LLaVA-1.5 + ControlMLLM	60.59	63.06
LLaVA-1.5 + ControlMLLM++	71.19	74.66
LLaVA-HR ( Luo et al., 2024b )	53.81	57.00
LLaVA-HR + ControlMLLM	58.92	66.89
LLaVA-HR + ControlMLLM++	69.06	82.68

Table 2: The results of combining with different MLLMs on ROC and RTC tasks (box, test set).

Figure 10: Ablation study of the ControlMLLM and ControlMLLM++. (a) Effect of different energy weights $\alpha$ of ControlMLLM on optimization stability and accuracy, larger $\alpha$ leading to overfitting of the model. (b) Influence of EMA decay rate $\beta$ of ControlMLLM (with fixed $\alpha=400$ ), smaller value of $\beta$ results in slower convergence of the learnable latent variable. (c) Impact of language bias mitigation weight $\gamma$ in PromptDebias, $\gamma$ =0.7 with the best performance.

Model	RefCOCOg (In-Domain)				Screenshot (Out-of-Domain)
Model	B@4	M	C	S	B@4	M	C	S
LLaVA-1.5	5.02	13.15	55.61	17.61	0.32	3.96	9.80	3.58
LLaVA-1.5 + ControlMLLM	5.53	14.00	59.75	19.08	0.45	5.08	19.74	5.81
LLaVA-1.5 + ControlMLLM++	6.24	15.05	67.37	21.46	0.57	6.53	40.01	9.14
LLaVA-HR	5.28	13.45	56.29	18.55	0.29	4.27	10.88	4.59
LLaVA-HR + ControlMLLM	6.32	15.00	68.82	21.55	0.64	6.79	37.10	8.54

Table 3: Referring Description performance on RefCOCOg and Screenshot datasets. Metrics include BLEU-4 (B@4), METEOR (M), CIDEr (C), and SPICE (S). Our method not only equips non-referring models with effective grounding ability but also complements modern referring-capable MLLMs by enhancing their generalization and precision.

$T\rightarrow$	0	4	5
Acc.	57.00	60.50	62.50

Table 4: The ablation of ES on ControlMLLM (

\alpha=400,\beta=0.5

) on validation set of ROC task.

Optim++
Adam	$A^{(st)}$	LayerSelection	PromptDebias	Acc.
				60.59
✓				66.93
✓	✓			65.70
✓		✓		65.70
✓	✓	✓		68.02

Table 5: Ablation study of ControlMLLM++ on the ROC task. Optim++ includes three components: Adam optimizer,

A^{(st)}

(answer-start token attention), and LayerSelection. PromptDebias denotes the language bias mitigation module. Accuracy ( Acc. ) is reported for each variant.

Models	Vanilla	ControlMLLM	ControlMLLM++
LLaVA-1.5-7B	54.72	60.59	71.19
LLaVA-1.5-13B	55.69	58.40	68.22
Qwen2.5-VL-3B	75.84	76.23	77.84
Qwen2.5-VL-7B	78.81	79.20	79.20

Table 6: Performance comparison of Vanilla, ControlMLLM, and ControlMLLM++ across different MLLM architectures with different decoder sizes on the ROC task (box, test set).

Models	Latency(s)	GPU Mem.
LLaVA-1.5 (6 tokens)	0.94	7.6G
+ControlMLLM	1.88	13.6G
+ControlMLLM++	1.79	13.0G
(w/o PromptDebias)
+ControlMLLM++	2.54	21.3G
LLaVA-1.5 (436 tokens)	21.28	8.9G

Table 7: Inference latency and GPU memory usage for ControlMLLM and ControlMLLM++ compared to the baseline LLaVA-1.5 model. Results are measured for short (6 tokens) and long (436 tokens) output sequences on a single NVIDIA GTX 4090 GPU. Early stopping is disabled in this experiment.

Test-Time Adaptation of VLMs

Test-time adaptation of vision-language models uses unlabeled deployment data to improve predictions under domain shifts without retraining.

See all 79 papers

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models
Feb 26, 2026Yuxiang Shen, Hailong Huang, Zhenkun Gao +6VLM Reasoning Large Vision-Language Models
the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China +3
Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification
Mar 27, 2026Shuai Lv, Chang Liu, Feng Tang +5Multimodal Large Language Models Multimodal Reasoning
University of Science and Technology of China, Hefei, China · Huawei Foundation Model Department, Shanghai, China · The Chinese University of Hong Kong, Hong Kong, China +1
Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning
May 25, 2026Longteng Guo, Yifan Wang, Pengkang Huo +4Visually Grounded Reasoning Visual Reasoning
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Feb 26, 2026J/K move · Enter open · S save

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

Yuxiang Shen, Hailong Huang, Zhenkun Gao, Xueheng Li+5

the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China+3

VLM Reasoning Large Vision-Language Models Fine-Grained Visual Perception Efficient VLM Inference

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details. While effective, large-scale training is computationally expensive, which has spurred growing interest in lightweight, training-free solutions. However, existing training-free methods suffer from two flaws: perceptual redundancy from indiscriminate cropping, which increases computational cost and introduces noise; and a drift between semantic intent and spatial attention, which prevents accurate localization of user-focused regions. To address these challenges, we propose LookWise, a framework for adaptive visual reasoning. LookWise follows a two-stage pipeline: a confidence-based module decides when to look more carefully, and a semantic-guided localization module determines where to look. This design enables MLLMs to adaptively acquire fine-grained visual evidence without additional training. Experiments on fine-grained and high-resolution visual reasoning benchmarks show that LookWise consistently improves accuracy over strong baselines while achieving an approximately

4.0\times

inference speedup over the search-based method ZoomEye, demonstrating robust cross-model generalization.

Abstract

Test-Time Adaptation of VLMs

Explore similar work

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning