Organizations: Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen · School of Computer Science and Engineering, Central South University · Text Computing and Cognitive Intelligence Ministry of Education Engineering Research Center, Guizhou University · Shanghai Aviation Electric Co., Ltd, Aviation Industry Corporation of China, Shanghai
Recently, Interleaved-modal Chain-of-Thought (ICoT) reasoning has achieved remarkable success by leveraging both multimodal inputs and outputs, attracting increasing attention. While achieving promising performance, current ICoT methods still suffer from two major limitations: (1) Static Visual Thought Positioning, which statically inserts visual information at fixed steps, resulting in inefficient and inflexible reasoning; and (2) Broken Visual Thought Representation, which involves discontinuous and semantically incoherent visual tokens. To address these limitations, we introduce Interleaved-modal Chain-of-Thought reasoning with Dynamic and Precise Visual Thoughts (DaP-ICoT), which incorporates two key components: (1) Dynamic Visual Thought Integration adaptively introduces visual inputs based on reasoning needs, reducing redundancy and improving efficiency. (2) Precise Visual Thought Guidance ensures visual semantically coherent and contextually aligned representations. Experiments across multiple benchmarks and models demonstrate that DaP-ICoT achieves state-of-the-art performance. In addition, DaP-ICoT significantly reduces the number of inserted images, leading to a 72.6% decrease in token consumption, enabling more efficient ICoT reasoning.
Figures & tables
Figure 1: (a) Current ICoT: While supporting multimodal inputs and outputs, it suffers from Static Visual Thought Integration , which requires the insertion of visual information after each step, and Broken Visual Thought Representation , in which the inserted visual tokens are lack coherence, resulting in inefficient reasoning. (b) Our DaP-ICoT : It provides Dynamic Visual Thought Integration and Precise Visual Thought Guidance , enabling efficient reasoning.
Figure 2: An overview of Interleaved-modal Chain-of-Thought reasoning with D ynamic a nd P recise Visual Thoughts ( DaP-ICoT ), including Dynamic Visual Thought Integration ( \lx@sectionsign\refDVTI ), and Precise Visual Thought Guidance ( \lx@sectionsign\refPVTG ).
Figure 3: Ablation Study on Qwen2-VL-7B : “w/o PVTG” indicates removal of Precise Visual Thought Guidance for Visual Cues, and “w/o DVTI” indicates removal of Dynamic Visual Thought Integration for Adaptive Reasoning
Figure 4: A comparison of the total token consumption between DaP-ICoT and baseline methods on the M 3 CoT benchmark using the Qwen2-VL-7B . DaP-ICoT achieves a 72.6% reduction in token consumption compared to ICoT.
Figure 5: A comparison of image insertion frequency and the number of inserted image tokens between DaP-ICoT and ICoT on the Qwen2-VL-7B model.
Figure 6: The proportion of samples where the confidence level increases after image insertion, as well as the average confidence value improvement, for DaP-ICoT and ICoT. The Blue Color denotes the baseline method ICoT, while the Green Color represents our DaP-ICoT .
Figure 7: Performance comparison across the Confidence Threshold τ range of (0,1] : Optimal model performance achieved at τ=0.2 , with 57.2% Accuracy in the 0-shot setting and 58.7% in the 1-shot setting, effectively balancing visual integration and textual reasoning.
Figure 8: The case study. Figure (a) illustrates the reasoning process of the current ICoT, where after each reasoning step, it is necessary to insert image tokens that are most similar to the previous text. However, these tokens are broken image tokens, which ultimately leads to the incorrect answer (D) . Figure (b) illustrates our DaP-ICoT . In contrast, DaP-ICoT only inserts images when the text confidence is low. Furthermore, the inserted images are precise and complete , having been segmented using the SAM2 model. After efficient interleaved visual-textual reasoning, the final correct answer (A) is obtained.
Multimodal chain-of-thought reasoning studies how AI models combine visual, audio, and textual evidence through intermediate reasoning to solve complex tasks.
Recent advances in multi-modal large language models have shifted toward the "thinking with images" paradigm to improve reasoning faithfulness, yet existing methods rely heavily on explicit coordinate- based grounding. We identify two fundamental limitations in this approach, i.e., a semantic gap where sparse regressions miss high-dimensional latent structures, and a resolution gap due to the granularity mismatch between patch-level tokens and pixel-level coordinates. To address these, we propose TiCO (Thinking with images without COordinates). It reformulates visual reasoning as a semantic evidence discovery process within the latent space, and evidence is injected as token-level regions without pixel coordinates. Specifically, TiCO utilizes a dedicated probe token to estimate similarity distributions and performs importance sampling over spatially connected components to generate diverse, semantically dominant evidence rollouts. Extensive experiments across multiple high-resolution and general visual reasoning benchmarks demonstrate that our TiCO consistently outperforms state-of-the-art coordinate-based methods with significantly improved inference efficiency.
Meng Cao, Haoze Zhao, Yi Wang +2
Mohamed bin Zayed University of Artificial Intelligence · Shanghai AI Lab · Sun Yat-sen University
Chain-of-Thought (CoT) improves the performance of Large Language Models (LLMs) and has been extended to Multimodal Large Language Models (MLLMs). More recent work further moves from text-based multimodal reasoning toward interleaved-modal reasoning, where intermediate steps can incorporate both textual rationales and visual evidence. In this work, we propose a bolder and more ambitious idea: could images alone serve as the reasoning medium for both language and multimodal tasks? To explore this, we propose optical reasoning, which treats images as a standalone reasoning medium. We instantiate this concept with two variants: typographic-based optical reasoning, which optimizes visual layouts for compact rationale rendering, and graphical-based optical reasoning, which composes text and graphical elements into structured visual rationales. Across mathematical, scientific, and interleaved-modal reasoning benchmarks, optical reasoning can match or even exceed traditional text reasoning while reducing reasoning tokens by an average of 28.57% on language tasks and 16% on multimodal tasks, achieving 1.96 times the token efficiency of text reasoning. These results show that images can effectively and efficiently encode rationales while providing a unified visual canvas for reasoning.
Chain-of-thought (CoT) reasoning has proven effective for enhancing problem-solving in large language models. However, when applied to multimodal LLMs (MLLMs), existing CoT approaches suffer from a fundamental limitation: they perform reasoning entirely in text without accessing visual features during the reasoning process. After initial visual encoding, image information becomes inaccessible, forcing models to reason based solely on whatever was captured in the initial description, which forms a `vision-blind reasoning' paradigm that limits fine-grained visual extraction, error verification, and adaptive attention. We propose Text-Visual Interleaved Chain-of-Thought (TVI-CoT), a framework that enables explicit interleaving of textual reasoning and visual feature access through learnable control tokens <THINK>, <LOOK> and <ANSWER>. These tokens allow dynamic switching between reasoning and visual grounding, attending to relevant image regions conditioned on the evolving reasoning state. Experiments on eight benchmarks demonstrate state-of-the-art results among MLLM-based CoT methods and notable performance boost compared to the baseline: +6.1% on MMMU, +3.8% on MathVerse, +3.4% on MathVista, and +3.4% on ScienceQA. Code is available at https://github.com/hulianyuyy/TVI-CoT.
Lianyu Hu, Xiaoyu Ma, Zeqin Liao +1
College of Computing and Data Science of Nanyang Technological University, Singapore.