Can a Lightweight Multimodal Model Estimate LLM Reasoning Performance? A Study for Compute-Optimal Document Inference

Zishan Ahmad; Vishal Vaddina

cs.AIAug 19, 2026

Can a Lightweight Multimodal Model Estimate LLM Reasoning Performance? A Study for Compute-Optimal Document Inference

Authors: Zishan Ahmad, Vishal Vaddina

Organizations: Phi Labs, Quantiphi

Abstract

Uniformly allocating inference reasoning budgets to LLMs is expensive and prone to over-thinking penalties; especially in document tasks where visual layouts drive complexity. To address this, we introduce BudgetDoc, the first multimodal benchmark providing explicit supervision for model-budget-performance trade-offs across three document tasks. Using BudgetDoc, we train DRB (Document-Reasoning Balancer), an approx. 1B-parameter pre-flight estimator (SigLIP-2 + Qwen3-0.6B) that predicts ordinal model performance across budget levels, achieving a 0.753 weighted F1. When dynamically allocating reasoning budgets across five frontier models and three datasets, DRB matches or improves F1 scores compared to always-maximum-budget baselines in 9 of 15 configurations while drastically reducing cost. Finally, preliminary evaluations demonstrate DRB's potential to generalize to cross-model selection.

Figures & tables

Figure 1: The over-thinking penalty across model families and datasets. Each line shows mean F1 as a function of reasoning budget (Gemini, top row) or reasoning effort tier (GPT, bottom row). For Gemini 2.5 Flash on RVL-CDIP, F1 decreases monotonically from 0.86 (budget

=0

) to 0.81 (budget

=2048

), a 6% penalty from over-allocation. GPT-5.2 exhibits the same pattern on CheckboxQA (0.68 at low

\rightarrow

0.58 at high ). The optimal budget is task- and model-specific, motivating per-sample estimation rather than a fixed ceiling.

Dataset	Model	DRB F1	BL F1	$\Delta$ F1	Adj. DRB Cost	BL Cost	Adj. $\Delta$ Cost	$\Delta$ Lat (s, %)
CDIP	Gemini-2.5-Flash	0.888	0.840	+5.7%	$0.00169	$0.00427	$-$ 48.0%	$+$ 3.6s ( $+$ 27.6%)
	Gemini-3-Flash-Preview	0.890	0.920	$-$ 3.3%	$0.00965	$0.01292	$-$ 21.3%	$+$ 3.8s ( $+$ 25.5%)
	Gemini-3.1-Flash-Lite	0.799	0.762	+4.9%	$0.00482	$0.00752	$-$ 29.0%	$+$ 10.3s ( $+$ 76.4%)
	GPT-5.1	0.815	0.816	$-$ 0.1%	$0.00096	$0.01896	$-$ 92.2%	$-$ 15.5s ( $-$ 40.7%)
	GPT-5.2	0.879	0.869	+1.1%	$0.00093	$0.09611	$-$ 99.0%	$-$ 39.7s ( $-$ 62.3%)
CheckboxQA	Gemini-2.5-Flash	0.559	0.556	+0.4%	$0.00301	$0.00515	$-$ 31.4%	$+$ 3.8s ( $+$ 31.9%)

Table 1: Reasoning budget optimization via DRB estimation on held-out test documents. Baseline: auto-budget for Gemini and high/xhigh for GPT models (the default without an estimator).

\Delta

F1 and Adj.

\Delta

Cost are relative to the baseline, where the adjusted cost dynamically factors in the configuration-level estimator compute overhead.

\Delta

Lat shows absolute seconds followed by percentage change. Green = improvement, red = degradation.

Selection Scope	Dataset	DRB F1	Oracle F1	$\Delta$ F1	Oracle Cost	Adj. DRB Cost	Adj. $\Delta$ Cost
Gemini Family	CDIP	0.897	0.920	$-$ 2.5%	$0.01292	$0.00593	$-$ 54.1%
	CheckboxQA	0.644	0.670	$-$ 3.9%	$0.02126	$0.00513	$-$ 75.9%
	TAT-DQA	0.512	0.568	$-$ 9.9%	$0.00156	$0.00141	$-$ 9.6%
GPT Family	CDIP	0.781	0.869	$-$ 10.1%	$0.09611	$0.00121	$-$ 98.7%
	CheckboxQA	0.622	0.673	$-$ 9.3%	$0.31805	$0.00183	$-$ 99.4%
	TAT-DQA	0.497	0.545	$-$ 9.5%	$0.06084	$0.00061	$-$ 99.0%

Table 2: Exploratory case study: applying DRB estimates to cross-model selection on held-out test documents. Oracle baseline represents the best single model deployed at default auto-budget (Gemini) or maximum reasoning effort (GPT). Adjusted DRB cost dynamically incorporates the configuration-level estimator compute overhead. While cross-family selection triggers expected F1 degradation under this unobserved distribution shift, cost savings remain substantial across all scopes by routing away from expensive high-effort tiers.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Class (F1 bin)	Precision	Recall	F1	Support
0 ( $f_{1}<0.5$ )	0.623	0.491	0.549	448
1 ( $f_{1}\in[0.5,0.6)$ )	0.646	0.361	0.463	87
2 ( $f_{1}\in[0.6,0.7)$ )	0.716	0.671	0.693	143
3 ( $f_{1}\in[0.7,0.8)$ )	0.744	0.569	0.645	174
4 ( $f_{1}\in[0.8,0.9)$ )	0.849	0.656	0.740	154
5 ( $f_{1}\in[0.9,0.99)$ )	0.781	0.653	0.711	98

Appendix

Table 3: DRB (SigLIP-2 + Qwen3-0.6B) per-class estimation performance on the BudgetDoc test set. Class 6 (perfect performance) has the highest F1, driven by strong recall, meaning DRB reliably identifies samples where any budget will succeed.

Dataset	Model (Effort)	DRB F1	BL F1	$\Delta$ F1	Adj. DRB Cost	BL Cost	Adj. $\Delta$ Cost
CDIP	GPT-5.1 (Low)	0.815	0.814	$\pm$ 0.0%	$0.00096	$0.01814	$-$ 91.8%
	GPT-5.1 (Medium)	0.815	0.813	$+$ 0.2%	$0.00096	$0.02148	$-$ 93.1%
	GPT-5.1 (High)	0.815	0.816	$-$ 0.1%	$0.00096	$0.02900	$-$ 94.9%
	GPT-5.2 (Low)	0.879	0.873	$+$ 0.7%	$0.00093	$0.05738	$-$ 97.5%
	GPT-5.2 (Medium)	0.879	0.876	$+$ 0.3%	$0.00093	$0.05851	$-$ 97.5%
	GPT-5.2 (High)	0.879	0.869	$+$ 1.1%	$0.00093	$0.06073	$-$ 97.6%

Appendix

Table 4: DRB versus GPT models across all four reasoning effort tiers. Adj.

\Delta

Cost factors in the dynamic configuration-level compute overhead. Green = improvement, red = degradation.

Cost-Aware Inference

Cost-aware inference allocates models, computation, and serving resources to meet quality goals at lower latency, energy use, and monetary cost.

See all 281 papers

ROI-Reasoning: Rational Optimization for Inference via Pre-Computation Meta-Cognition
Jan 7, 2026Muyang Zhao, Chunhao Tian, Qi Qi +1Cost-Aware Inference Metacognition in Language Models
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation
Aug 12, 2026Rodrigo Guedes de Souza, Alison R. PanissonLLM Evaluation Language Model Generation Evaluation
Federal University of Santa Catarina (UFSC) Brazil
The Shadow Price of Reasoning: Economic Perspective on Optimal Budget Allocation for LLMs
Jun 2, 2026Xu Wan, Speed Zhu, Jianwei Cai +4Cost-Aware Inference Inference-Time Scaling
1Zhejiang University · 2Tecent HY Team · 3Peking University

Jan 7, 2026J/K move · Enter open · S save

ROI-Reasoning: Rational Optimization for Inference via Pre-Computation Meta-Cognition

Muyang Zhao, Chunhao Tian, Qi Qi, Hao Sun

Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China

Cost-Aware Inference Metacognition in Language Models RL for Language Model Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) can achieve strong reasoning performance with sufficient computation, but they do not inherently know how much computation a task requires. We study budgeted inference-time reasoning for multiple problems under a strict global token constraint and formulate it as an Ordered Stochastic Multiple-Choice Knapsack Problem (OS-MCKP). This perspective highlights a meta-cognitive requirement---anticipating problem difficulty and allocating computation strategically. We propose ROI-Reasoning, a two-stage framework that endows LLMs with intrinsic, budget-aware rationality. First, Meta-Cognitive Fine-Tuning teaches models to predict reasoning cost levels and decide whether a problem is worth attempting before generation. Next, Rationality-Aware Reinforcement Learning optimizes sequential decision making under a hard token budget, allowing models to learn long-horizon allocation strategies. Experiments across mathematical reasoning, multiple-choice question answering, and code generation show that ROI-Reasoning improves scores and generally reduces reference-order regret under tight budgets, with up to a 417% relative score improvement over the strongest same-backbone baseline in mathematics.

Abstract

Cost-Aware Inference

Explore similar work

ROI-Reasoning: Rational Optimization for Inference via Pre-Computation Meta-Cognition

Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

The Shadow Price of Reasoning: Economic Perspective on Optimal Budget Allocation for LLMs