Fuel Gauge: Estimating Chain-of-Thought Length Ahead of Time in Large Multimodal Models

Yuedong Yang; Xiwen Wei; Mustafa Munir; Radu Marculescu

cs.CVMar 11, 2026

Fuel Gauge: Estimating Chain-of-Thought Length Ahead of Time in Large Multimodal Models

Authors: Yuedong Yang, Xiwen Wei, Mustafa Munir, Radu Marculescu

Organizations: Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin

Abstract

Reasoning Large Multi-modality Models (LMMs) have become the de facto choice for many applications. However, these models rely on a Chain-of-Thought (CoT) process that is lengthy and unpredictable at runtime, often resulting in inefficient use of computational resources (due to memory fragmentation) and sub-optimal accuracy (due to under- and over-thinking). We observe empirically that the CoT process follows a very simple form, whose behavior is independent of the specific generated samples. This suggests that the CoT length can be estimated ahead of time based on a hidden parameter representing the amount of "fuel" available to support the reasoning process. Based on this insight, we propose Fuel Gauge, the first method which extracts this hidden signal and predicts CoT length ahead of time. We demonstrate the utility on the Fuel Gauge on two downstream tasks: predictive KV cache allocation, which addresses memory fragmentation in LMM serving systems, and CoT length modulation, which mitigates under-thinking and over-thinking. Extensive experiments on LMMs across text-only, image-text, and video-text question answering benchmarks demonstrate the effectiveness, generalizability, and practical value of our Fuel Gauge. For example, on the GPQA-Diamond benchmark, our Fuel Gauge achieves less than half the CoT length prediction error compared to the baseline; this translates into a 13.37x reduction in the memory allocation frequency.

Figures & tables

Figure 1: Example of the output of reasoning LMM, which consists of a long CoT section wrapped with special symbols “<think>” and “</think>”, and a short Conclusion section.

Figure 2: Correlation between Chain-of-Thoughts (CoT) and LMM accuracy collected from Qwen3 [ 1 ] , Qwen3VL [ 6 ] , Intern-S1 [ 3 ] and GLM [ 31 ] across multiple text-only, image-text and video-text benchmarks. Using accuracy as a proxy for task difficulty, we observe a clear negative correlation between CoT length and task difficulty. This trend motivates our hypothesis that CoT length is predictable based solely on the question itself.

Figure 3: CoT length estimation using the Fuel Gauge. Numbers in the figure are randomly chosen for illustration purpose. See Section 4.4 for implementation details. In Stage 1, the hidden signal

S_{i}

is extracted using

f_{\text{sig}}

and the corresponding fuel level is estimated with

f_{\text{fuel}}

. In Stage 2, a linear model is fitted to all predicted fuel-level points, and the zero-crossing point of this line is taken as the final CoT length prediction. Based on this CoT length estimation, we further develop two novel downstream applications (see Section 5 ).

Method	Same Modality, Task-to-Task Generalization, rMAE $\downarrow$				Image-to-Video & Task-to-Task Generalization, rMAE $\downarrow$
	GPQA-Diamond		MathVision-m		LongVideoBench-15		LongVideoBench-60
	Qwen3-4B	Qwen3-8B	Qwen3VL-2B	Qwen3VL-4B	Qwen3VL-2B	Qwen3VL-4B	Qwen3VL-2B	Qwen3VL-4B
Mean	0.5721	0.5003	0.6798	0.7982	3.159	3.706	7.384	4.887
Median	0.5967	0.5212	0.7621	0.8931	3.557	4.122	7.967	5.410
Direct	0.4932	0.5795	0.4748	0.4965	9.833	4.782	10.397	5.877
Fuel Gauge	0.3185	0.2732	0.2934	0.3139	0.4834	0.4527	0.5049	0.4645

Table 1: Results for CoT length prediction. Values are in rMAE, hence the lower the better. The best results are highlighted. Baseline methods are introduced in Section 6.1 . GPQA-Diamond and MathVison show the generalizability of Fuel Gauge from general tasks to specialized tasks, and LongVideoBench shows the both the cross-task and cross-modality ( i.e. , text-image to text-video) generalizability.

Method	GPQA-Diamond		MathVision-m
Method	Qwen3-4B	Qwen3-8B	Qwen3VL-2B	Qwen3VL-4B
Mean	0.2860	0.2501	0.3399	0.3991
Median	0.3161	0.2743	0.4620	0.5611
EoC Prob	0.4999	0.4999	0.4999	0.4999
Fuel Gauge	0.1588	0.1322	0.1186	0.1321

Table 2: Evaluation of the accuracy of fuel level estimation. All number are relative mean absolute error, so smaller is better.

Figure 4: CoT length prediction result with Qwen3-8B model on GPQA-Diamond benchmark. The prediction of our Fuel Gauge evolves as CoT progresses while the baseline fails.

Method	Same Domain, Task-to-Task Generalization, #Allocs $\downarrow$				Image-to-Video & Task-to-Task Generalization, #Allocs $\downarrow$
	GPQA-Diamond		MathVision-m		LongVideoBench-15		LongVideoBench-60
	Qwen3-4B	Qwen3-8B	Qwen3VL-2B	Qwen3VL-4B	Qwen3VL-2B	Qwen3VL-4B	Qwen3VL-2B	Qwen3VL-4B
HF [ 45 ]	491.0	533.0	682.6	544.0	41.79	42.82	44.63	43.62
Fuel Gauge	49.24	39.87	69.43	59.10	18.56	23.10	23.60	27.81
Reduction	9.971 $\times$	13.37 $\times$	9.831 $\times$	9.205 $\times$	2.252 $\times$	1.854 $\times$	1.891 $\times$	1.569 $\times$

Table 3: Results for predictive memory allocation. “#Allocs” is short for the number of required memory allocations. Lower is better.

Figure 5: CoT length and LMM accuracy for different CoT modulation factors

\eta

. Results are obtained with Qwen3-4B model on GPQA-Diamond benchmark. Orange star denotes the baseline case where no CoT modulation is applied. Figure (a) shows that Fuel Gauge controls the CoT length linearly. Then Figure (b) shows that the change in CoT length linearly translates to a change in accuracy. Finally Figure (c) shows that based on the linearity in figures (a) and (b), we can achieve our target and control the accuracy linearly with

\eta

Figure 6: CoT length and LMM accuracy under different CoT modulation factors

\eta

on more benchmarks and models. “OOD” denotes that results are obtained from image-to-video Out-Of-Domain generalization with our Fuel Gauge. “LVB” is short for “LongVideoBench”. By changing

\eta

from -1 to 1, our Fuel Gauge consistently modulates both CoT length and accuracy linearly, thus demonstrating the wide-applicability of our Fuel Gauge. The green star shows the baseline without modulation. Red and orange markers indicate CoT modulation with positive or negative

\eta

, corresponding to gradient ascent or descent (Equation 5 ), yielding longer or shorter CoT.

Modality	$\eta$ to Length	Length to Acc	$\eta$ to Acc
Text $\rightarrow$ Text	0.9936	0.9754	0.9722
Image $\rightarrow$ Image	0.9932	0.9988	0.9866
Image $\rightarrow$ Video	0.9908	0.9835	0.9537

Table 4: Absolute Pearson correlation among the CoT modulation coefficient

\eta

, CoT length and CoT accuracy (“Acc”). “Modality” lists the training and testing modality of our Fuel Gauge.

Figure S11 : CoT length and LMM accuracy for different CoT modulation factors

\eta

. Results are obtained with Qwen3-4B model on multiple benchmarks. Orange star denotes the baseline case where no CoT modulation is applied. Figure (a) shows that Fuel Gauge controls the CoT length linearly. Figure (b) shows that the change in CoT length quasi-linearly translates to a change in accuracy. Finally Figure (c) shows that based on the quasi-linearity in figures (a) and (b), we can achieve our target and control the accuracy quasi-linearly with

\eta

Figure S12 : CoT length and LMM accuracy for different CoT modulation factors

\eta

. Results are obtained with Qwen3-8B model on multiple benchmarks. Orange star denotes the baseline case where no CoT modulation is applied. Figure (a) shows that Fuel Gauge controls the CoT length linearly. Figure (b) shows that the change in CoT length quasi-linearly translates to a change in accuracy. Finally Figure (c) shows that based on the quasi-linearity in figures (a) and (b), we can achieve our target and control the accuracy quasi-linearly with

\eta

Figure S13 : CoT length and LMM accuracy for different CoT modulation factors

\eta

. Results are obtained with Qwen3VL-2B model on multiple benchmarks. Orange star denotes the baseline case where no CoT modulation is applied. Figure (a) shows that Fuel Gauge controls the CoT length linearly. Figure (b) shows that the change in CoT length quasi-linearly translates to a change in accuracy. Finally Figure (c) shows that based on the quasi-linearity in figures (a) and (b), we can achieve our target and control the accuracy quasi-linearly with

\eta

Figure S14 : CoT length and LMM accuracy for different CoT modulation factors

\eta

. Results are obtained with Qwen3VL-4B model on multiple benchmarks. Orange star denotes the baseline case where no CoT modulation is applied. Figure (a) shows that Fuel Gauge controls the CoT length linearly. Figure (b) shows that the change in CoT length quasi-linearly translates to a change in accuracy. Finally Figure (c) shows that based on the quasi-linearity in figures (a) and (b), we can achieve our target and control the accuracy quasi-linearly with

\eta

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S7 : CoT length and LMM accuracy for Intern-S1-mini-8B with different CoT modulation factors

\eta

. Orange star denotes the baseline case where no CoT modulation is applied. Figure (a) shows that Fuel Gauge controls the CoT length linearly. Figure (b) shows that the change in CoT length quasi-linearly translates to a change in accuracy. Finally, Figure (c) shows that based on the linearity in figures (a) and (b), we can achieve our target and control the accuracy quasi-linearly with

\eta

Fuel Level Estimation, rMAE $\downarrow$
Method	MathVision-m	LVB-15	LVB-60
Mean (Section 6.1 )	0.8098	1.4544	1.417
Fuel Gauge	0.1311	0.2583	0.2831
CoT Length Prediction, rMAE $\downarrow$
Mean (Section 6.1 )	0.8550	0.8598	1.130
Fuel Gauge	0.4194	0.4902	0.4525

Appendix

Table S5 : Evaluation results for Intern-S1-mini-8B are reported on MathVision-m, LongVideoBench-15 (LVB-15), and LongVideoBench-60 (LVB-60). MathVision-m demonstrates the generalizability of Fuel Gauge from general tasks to specialized tasks, while LongVideoBench highlights its generalizability across both tasks and modalities ( i.e. , from text-image to text-video). Across all three benchmarks, Fuel Gauge significantly outperforms the baseline methods.

Figure S8 : Examples of predictive KV-cache allocation. The baseline allocator (HF, Section 6.1 ) performs frequent small, on-demand allocations (diagonal line in plots). With our Fuel Gauge, memory needs are predicted and allocated in larger chunks (stair line in plots), reducing allocation frequency and mitigating fragmentation.

Figure S9 : Fuel level estimation accuracy with different number of channels for

f_{\text{sig}}

and

f_{\text{fuel}}

for Qwen3-4B on MMLU. Star denotes the adopted design choice for Fuel Gauge.

Figure S10 : Fuel level estimation accuracy with different sizes of window adopted in

f_{\text{sig}}

for Qwen3-4B on MMLU. Star denotes the adopted design choice for the Fuel Gauge.

Loss Function	Test rMAE $\downarrow$
MSE	0.1323
MAE	0.1311
Smooth L1 ( $\beta=0.1$ )	0.1317
Smooth L1 ( $\beta=0.01$ )	0.1300
Smooth L1 ( $\beta=0.001$ )	0.1311

Appendix

Table S6 : Fuel level estimation accuracy with different loss function in Equation 3 for Qwen3-4B on MMLU. Bold denotes the adopted design choice for the Fuel Gauge.

Model	Throughput	#Parameters
Qwen3-4B	22.4 Token / s	4B
Fuel Gauge [Batch Size=1]	792.7 Token / s	82.24k
Fuel Gauge [Batch Size=32]	11217.3 Token / s	82.24k

Appendix

Table S7 : Overhead of Fuel Gauge on NVIDIA A6000 GPU. Results show that the time and memory spent in Fuel Gauge is negligible.

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 370 papers

May 14, 2026

Stateful Reasoning via Insight Replay

Chain-of-Thought (CoT) reasoning has become a foundation for eliciting multi-step reasoning in large language models, but recent studies show that its benefits do not scale monotonically with chain length: while longer CoT generally enables a model to tackle harder problems, on a given problem, accuracy typically increases with CoT length up to a point, after which it declines. We identify a major cause of this phenomenon: as the CoT grows, the model's attention to critical insights produced earlier in the trace gradually weakens, making those insights progressively less accessible when they are most needed. Therefore, we propose InsightReplay, a stateful reasoning approach in which the model periodically extracts critical insights from its reasoning trace and replays them near the active generation frontier, keeping them accessible as the reasoning scales. Extensive experiments on a

\mathbf{2}\!\times\!\mathbf{3}\!\times\!\mathbf{4}

benchmark grid, covering model scales

\{\text{8B}, \text{30B}\}

, model families

\{\text{Qwen3.5}, \text{DeepSeek-R1-Distill-Qwen}, \text{Gemma-4}\}

, and reasoning benchmarks

\{\text{AIME}, \text{HMMT}, \text{GPQA Diamond}, \text{LiveCodeBench v5}\}

, show that 3-round InsightReplay yields accuracy gains across all 24 settings, with an averaged improvement of

\mathbf{+1.65}

points over standard CoT, and a largest single-setting gain of

\mathbf{+9.2}

points on R1-Distill-32B's LiveCodeBench v5 subset. Our results suggest that the effectiveness of test-time scaling depends not only on how much a model reasons, but also on whether critical intermediate insights remain accessible throughout long reasoning trajectories.

Bin Lei, Caiwen Ding, Jiachen Yang +2

University of Minnesota · 2Simular AI

Memory-Augmented Language Models CoT Reasoning Test-Time Scaling LLM Reasoning

Ask PDF

Aug 30, 2026

Detecting Hidden Chain-of-Thought in Large Language Models with Linguistic, Behavioral, and Mechanistic Indicators

Large language models often answer complex reasoning questions without revealing intermediate steps, raising whether they reason latently or complete patterns. We propose the Hidden CoT Detection Score (HCDS), a comparative behavioral and mechanistic signal measuring whether neutral-prompt behavior aligns more closely with explicit CoT or explicit no- CoT. Here, hidden CoT operationally denotes this neutral-prompt CoT-like alignment; HCDS does not directly observe or prove an unexposed reasoning trace. On GSM8K, HCDS is significantly positive for both Qwen3-4B variants (Thinking

+1.87

p = 1.2 \times 10^{-7}

; Instruct

+1.41

p = 1.9 \times 10^{-4}

), replicates across a different inference stack and quantization within

0.08

(

+1.80

and

+1.45

), and is not significantly positive in seven of eight length-adjusted calibration-control cells. The unadjusted score produces large positive scores on single-step arithmetic and numeric factual lookup. The variants also respond differently to no-CoT instructions: Instruct complies from the prompt alone, whereas Thinking continues reasoning and requires intervention. These findings show stronger, less prompt-conditional CoT-like behavior in the reasoning-tuned model, consistent with but not proof of latent reasoning. HCDS thus investigates latent reasoning without relying on models' self-reported traces.

Armaan Singh, Ryan Trinh Le, Jasmine Kaur +5

1Stanford University · 2York University · 3Facebook

Implicit Reasoning in Language Models Language Model Probing Large Reasoning Models LLM Interpretability LLM Evaluation Language Model Reasoning Evaluation+1

Ask PDF

Apr 15, 2026

LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning

As language models are increasingly deployed for complex autonomous tasks, their ability to reason accurately over longer horizons becomes critical. An essential component of this ability is planning and managing a long, complex chain-of-thought (CoT). We introduce LongCoT, a scalable benchmark of 2,500 expert-designed problems spanning chemistry, mathematics, computer science, chess, and logic to isolate and directly measure the long-horizon CoT reasoning capabilities of frontier models. Problems consist of a short input with a verifiable answer; solving them requires navigating a graph of interdependent steps that span tens to hundreds of thousands of reasoning tokens. Each local step is individually tractable for frontier models, so failures reflect long-horizon reasoning limitations. At release, the best models achieve <10% accuracy (GPT 5.2: 9.8%; Gemini 3 Pro: 6.1%) on LongCoT, revealing a substantial gap in current capabilities. Overall, LongCoT provides a rigorous measure of long-horizon reasoning, tracking the ability of frontier models to reason reliably over extended periods.

Sumeet Ramesh Motwani, Daniel Nichols, Charles London +17

University of Oxford · Lawrence Livermore National Laboratory (LLNL) · MBZUAI +4

Long-Horizon LLM Reasoning CoT Reasoning Long-Horizon Reasoning Long-Context Reasoning in Language Models LLM Evaluation

Ask PDF

Abstract

Efficient Language Model Reasoning

Explore similar work

Stateful Reasoning via Insight Replay

Detecting Hidden Chain-of-Thought in Large Language Models with Linguistic, Behavioral, and Mechanistic Indicators

LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning