We propose ControlMLLM++, a novel test-time adaptation framework that injects learnable visual prompts into frozen multimodal large language models (MLLMs) to enable fine-grained region-based visual reasoning without any model retraining or fine-tuning. Leveraging the insight that cross-modal attention maps intrinsically encode semantic correspondences between textual tokens and visual regions, ControlMLLM++ optimizes a latent visual token modifier during inference via a task-specific energy function to steer model attention towards user-specified areas. To enhance optimization stability and mitigate language prompt biases, ControlMLLM++ incorporates an improved optimization strategy (Optim++) and a prompt debiasing mechanism (PromptDebias). Supporting diverse visual prompt types including bounding boxes, masks, scribbles, and points, our method demonstrates strong out-of-domain generalization and interpretability. The code is available at https://github.com/mrwu-mac/ControlMLLM.
Figures & tables
Figure 1: Comparison between the training method and our test-time computing method. The training method typically requires a large amount of in-domain data for training and cannot generalize to out-of-domain prompts. In contrast, our method can easily adapt to prompts from a new domain in a test-time computing manner.
Figure 2: The attention maps in various layers of the MLLMs, with the numbers indicating the respective layer indices. The top line visualizes the attention between the prompt token “hat” and the visual tokens, while the bottom line visualizes the attention between the context token (mentioned in Sec. 4.2 ) and the visual tokens.
Figure 3: Manipulating attention with various methods: (a), (b), and (c) demonstrate the manipulation of the attention map by adding an adjustment coefficient η on the attention map in the first step during the model inference. (d) illustrates the step-by-step editing approach. (e) showcases that optimizing a learnable context tokens (mentioned in Sec. 4.2 ) instead of visual tokens, while (f) presents the results of our method optimizing the learnable latent variable.
Figure 4: The overview of ControlMLLM. With the provided visual prompt, we convert it into a mask, and compute the mask-based energy function between the mask and the pooled attention map. During the inference process, we conduct backpropagation to optimize a learnable latent variable. This process is executed at the 0-th step of model inference and iterated T times.
Figure 5: The text tokens’ contribution to attention maps, where the attention is focused on the answer-start token ( ∣ START ∣ ).
Figure 6: The loss distribution across decoder layers, showing that the text-visual attention is primarily concentrated in the middle layers.
Figure 7: a) An example where ControlMLLM focus on the correct region, yet produce the same unexpected output as LLaVA due. b) Different prompts lead to different optimization performance. Such behavior is likely caused by the model’s overreliance on linguistic priors.
Figure 8: The examples of referring MLLM with four types of visual prompt, including box (a), mask (b), scribble (c) and point (d). The correct referring expressions are marked in green, incorrect referring expressions are marked in red, and hallucinated expressions are marked in orange. Compared to the baseline model, our method enhances interpretability and controllability with visual prompts, while also helping the model mitigate hallucination issues.
Figure 9: Comparison of OCR performance across models. Ferret, though trained with grounding supervision, still struggles with localized understanding on OCR task. In contrast, our method (LLaVA + Ours) accurately identifies the referred textual content.
Models
ROC (In-Domain)
RTC (Out-of-Domain)
Box
Mask
Scribble
Point
Box
Mask
Training Methods:
Kosmos-2 ( Peng et al., 2023 )
55.17
-
-
-
23.12
-
GPT4-ROI ( Zhang et al., 2023b )
58.59
-
-
-
57.17
-
Shikra-7B ( Chen et al., 2023b )
64.60
-
-
56.27
60.41
-
Ferret-7B ( You et al., 2023 )
71.71
72.39
71.58
68.54
58.70
59.81
Table 1: Performance on Referring Object Classification (ROC) and Referring Text Classification (RTC) tasks. ROC prompts follow the template: “ Is the object { location } a { class A } or a { class B } ? ”; RTC prompts: “ Is the text { location } ‘ { text A } ’ or ‘ { text B } ’? Please select only one. ” “–” indicates unsupported input types. Gray numbers are for reference only.
Models
ROC
RTC
LLaVA-1.5 ( Liu et al., 2024 )
54.72
57.42
LLaVA-1.5 + ControlMLLM
60.59
63.06
LLaVA-1.5 + ControlMLLM++
71.19
74.66
LLaVA-HR ( Luo et al., 2024b )
53.81
57.00
LLaVA-HR + ControlMLLM
58.92
66.89
LLaVA-HR + ControlMLLM++
69.06
82.68
Table 2: The results of combining with different MLLMs on ROC and RTC tasks (box, test set).
Figure 10: Ablation study of the ControlMLLM and ControlMLLM++. (a) Effect of different energy weights α of ControlMLLM on optimization stability and accuracy, larger α leading to overfitting of the model. (b) Influence of EMA decay rate β of ControlMLLM (with fixed α=400 ), smaller value of β results in slower convergence of the learnable latent variable. (c) Impact of language bias mitigation weight γ in PromptDebias, γ =0.7 with the best performance.
Model
RefCOCOg (In-Domain)
Screenshot (Out-of-Domain)
B@4
M
C
S
B@4
M
C
S
LLaVA-1.5
5.02
13.15
55.61
17.61
0.32
3.96
9.80
3.58
LLaVA-1.5 + ControlMLLM
5.53
14.00
59.75
19.08
0.45
5.08
19.74
5.81
LLaVA-1.5 + ControlMLLM++
6.24
15.05
67.37
21.46
0.57
6.53
40.01
9.14
LLaVA-HR
5.28
13.45
56.29
18.55
0.29
4.27
10.88
4.59
LLaVA-HR + ControlMLLM
6.32
15.00
68.82
21.55
0.64
6.79
37.10
8.54
Table 3: Referring Description performance on RefCOCOg and Screenshot datasets. Metrics include BLEU-4 (B@4), METEOR (M), CIDEr (C), and SPICE (S). Our method not only equips non-referring models with effective grounding ability but also complements modern referring-capable MLLMs by enhancing their generalization and precision.
T→
0
4
5
Acc.
57.00
60.50
62.50
Table 4: The ablation of ES on ControlMLLM ( α=400,β=0.5 ) on validation set of ROC task.
Optim++
Adam
A(st)
LayerSelection
PromptDebias
Acc.
60.59
✓
66.93
✓
✓
65.70
✓
✓
65.70
✓
✓
✓
68.02
Table 5: Ablation study of ControlMLLM++ on the ROC task. Optim++ includes three components: Adam optimizer, A(st) (answer-start token attention), and LayerSelection. PromptDebias denotes the language bias mitigation module. Accuracy ( Acc. ) is reported for each variant.
Models
Vanilla
ControlMLLM
ControlMLLM++
LLaVA-1.5-7B
54.72
60.59
71.19
LLaVA-1.5-13B
55.69
58.40
68.22
Qwen2.5-VL-3B
75.84
76.23
77.84
Qwen2.5-VL-7B
78.81
79.20
79.20
Table 6: Performance comparison of Vanilla, ControlMLLM, and ControlMLLM++ across different MLLM architectures with different decoder sizes on the ROC task (box, test set).
Models
Latency(s)
GPU Mem.
LLaVA-1.5 (6 tokens)
0.94
7.6G
+ControlMLLM
1.88
13.6G
+ControlMLLM++
1.79
13.0G
(w/o PromptDebias)
+ControlMLLM++
2.54
21.3G
LLaVA-1.5 (436 tokens)
21.28
8.9G
Table 7: Inference latency and GPU memory usage for ControlMLLM and ControlMLLM++ compared to the baseline LLaVA-1.5 model. Results are measured for short (6 tokens) and long (436 tokens) output sequences on a single NVIDIA GTX 4090 GPU. Early stopping is disabled in this experiment.
Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details. While effective, large-scale training is computationally expensive, which has spurred growing interest in lightweight, training-free solutions. However, existing training-free methods suffer from two flaws: perceptual redundancy from indiscriminate cropping, which increases computational cost and introduces noise; and a drift between semantic intent and spatial attention, which prevents accurate localization of user-focused regions. To address these challenges, we propose LookWise, a framework for adaptive visual reasoning. LookWise follows a two-stage pipeline: a confidence-based module decides when to look more carefully, and a semantic-guided localization module determines where to look. This design enables MLLMs to adaptively acquire fine-grained visual evidence without additional training. Experiments on fine-grained and high-resolution visual reasoning benchmarks show that LookWise consistently improves accuracy over strong baselines while achieving an approximately 4.0× inference speedup over the search-based method ZoomEye, demonstrating robust cross-model generalization.
Yuxiang Shen, Hailong Huang, Zhenkun Gao +6
the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China +3
Multimodal Large Language Models (MLLMs) achieve strong multimodal reasoning performance, yet we identify a recurring failure mode in long-form generation: as outputs grow longer, models progressively drift away from image evidence and fall back on textual priors, resulting in ungrounded reasoning and hallucinations. Interestingly, Based on attention analysis, we find that MLLMs have a latent capability for late-stage visual verification that is present but not consistently activated. Motivated by this observation, we propose Visual Re-Examination (VRE), a self-evolving training framework that enables MLLMs to autonomously perform visual introspection during reasoning without additional visual inputs. Rather than distilling visual capabilities from a stronger teacher, VRE promotes iterative self-improvement by leveraging the model itself to generate reflection traces, making visual information actionable through information gain. Extensive experiments across diverse multimodal benchmarks demonstrate that VRE consistently improves reasoning accuracy and perceptual reliability, while substantially reducing hallucinations, especially in long-chain settings. Code is available at https://github.com/Xiaobu-USTC/VRE.
Shuai Lv, Chang Liu, Feng Tang +5
University of Science and Technology of China, Hefei, China · Huawei Foundation Model Department, Shanghai, China · The Chinese University of Hong Kong, Hong Kong, China +1
Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.
Longteng Guo, Yifan Wang, Pengkang Huo +4
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences