CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes

Yufan Wu; Yinghui He; Zhengyi Hu; Lang Wei; Ruichen Li; Qifan Yang; Ting Zhu

cs.CLAug 27, 2026

CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes

Authors: Yufan Wu, Yinghui He, Zhengyi Hu, Lang Wei, Ruichen Li, Qifan Yang, Ting Zhu

Organizations: The Ohio State University · Princeton University

Abstract

Recent advances in inference-time scaling have significantly improved the reasoning performance of large language models (LLMs). However, these methods typically rely on repeated generation or external verification. To address this limitation, we introduce CritICL, a novel inference-time framework that improves reasoning while maintaining high efficiency. Our key insight is that LLM failure modes exhibit structured patterns across model scales within the same family. Instead of treating failures as undesirable outputs, CritICL leverages them as a source of guidance. Specifically, we utilize failure modes derived from weaker models and incorporate them into inference through critique-based in-context examples. We propose two variants: CritICL-dynamic, which adaptively predicts input-specific failure modes and retrieves critiques, and CritICL-static, which uses a global failure mode profile to provide stable guidance. Experimental results show that CritICL consistently outperforms standard in-context learning and achieves performance competitive with or superior to test-time scaling methods, while requiring significantly fewer generations and lower token cost. Code available at: https://github.com/umwyf/CRITICL

Figures & tables

Figure 1: CritICL is a two-stage, inference-time W2SG method. Stage 1: Construct CritBank. We use SLMs to generate responses and reasoning on a dataset. Then we leverage a frontier LLM to produce critiques and assigns failure mode labels. Stage 2: Perform LLM inference with CritICL. For a given query, CritICL-static constructs model-specific failure mode profiles from CritBank, while CritICL-dynamic identifies failure modes relevant to the query. We then performs failure mode-based in-context sample selection from CritBank to provide in-context examples for target LLMs.

Table 1: Performance comparison on Qwen family models. CritICL-dynamic and CritICL-static consistently outperform baselines, achieving improvements of up to 12.9% and 13.4%, respectively. Compared to test-time scaling methods, both approaches achieve competitive accuracy without requiring extensive repeated LLM inference. The evaluated models are Qwen2.5-32B-Instruct and Qwen2.5-72B-Instruct, while CritBank is constructed using Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, and Qwen2.5-7B-Instruct. All results are reported as Pass@1 accuracy unless otherwise specified.

Standard ICL
Method	Generations	Input Tokens	Output Tokens	Total Tokens
Zero-shot	1	312	287	599
1-shot (Rand.)	1	918	294	1212
3-shot (Rand.)	1	2087	306	2393
5-shot (Rand.)	1	3346	289	3635
1-shot (Fixed)	1	905	301	1206

Table 2: Inference cost comparison across methods. We report the average inference tokens per question on the MATH dataset when conducting experiments on Qwen2.5-32B-Instruct. Generations count all model invocations required to produce the final answer, including auxiliary steps such as failure-mode prediction or judge evaluation. CritICL slightly increases input tokens due to critique-based exemplars, but significantly reduces output tokens and avoids repeated generation.

Figure 2: Failure mode distributions across model scales remains highly consistent across scales within Qwen family (top) and Llama family (bottom). Each panel shows the normalized frequency of failure mode categories for models of different sizes. We report aggregate distribution by combining the failure statistics of the three weaker models described in Section 3.1 .

w/o Failure Mode-Based Example Selection
Method	GSM8K			MATH			AMC23			AIME25
Method	Acc	Prec	Rec	Acc	Prec	Rec	Acc	Prec	Rec	Acc	Prec	Rec
Random	87.4	88.1	85.9	53.1	52.2	54.0	22.2	21.4	23.0	13.2	12.5	14.0
Fixed	88.3	87.5	89.1	52.4	53.3	51.2	21.1	20.3	22.0	13.6	14.3	12.8
Semantic	87.9	88.6	86.7	53.8	54.4	52.9	21.9	22.5	20.8	13.0	13.7	12.2
w/ Failure Mode-Based Example Selection

Table 3: Effect of different in-context example selection strategies under a 5-shot setting. We construct the example pool using CritBank generated by smaller Qwen2.5 models (1.5B, 3B, and 7B) and evaluate on Qwen2.5-72B-Instruct.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Family / Transfer	Weak Profile	Spearman $\uparrow$	Kendall $\tau$ $\uparrow$	Top-10 $\uparrow$	JS Dist. $\downarrow$
Qwen $\rightarrow$ Qwen2.5-72B	Qwen2.5-1.5B	0.79	0.61	7/10	0.083
Qwen $\rightarrow$ Qwen2.5-72B	Qwen2.5-3B	0.82	0.65	8/10	0.071
Qwen $\rightarrow$ Qwen2.5-72B	Qwen2.5-7B	0.84	0.67	8/10	0.068
Qwen $\rightarrow$ Qwen2.5-72B	Qwen Aggregate	0.91	0.76	9/10	0.041
Llama $\rightarrow$ Llama-3.1-70B	Llama-3.2-1B	0.74	0.56	7/10	0.091
Llama $\rightarrow$ Llama-3.1-70B	Llama-3.2-3B	0.78	0.60	7/10	0.079

Appendix

Table 4: Quantitative similarity between weak- and strong-model failure-mode distributions. Higher Spearman correlation, Kendall’s

\tau

, and Top-10 overlap indicate stronger agreement, while lower Jensen–Shannon distance indicates greater distributional similarity.

Comparison	Samples	F1 $\uparrow$	Cohen’s $\kappa$ $\uparrow$
GPT-4o-mini vs. GPT-4.1	300	0.84	0.77
GPT-4o-mini vs. Claude-3.5-Sonnet	300	0.81	0.73
GPT-4o-mini vs. Human	100	0.82	0.74
Human vs. Human	100	0.86	0.80

Appendix

Table 5: Agreement between GPT-4o-mini annotations used to construct CritBank and independent LLM or human annotations.

Taxonomy	# Groups	GSM8K	MATH	AMC23	AIME25	Avg.
Coarse-grained	8	94.8	82.7	34.1	23.2	58.7
Fine-grained	20	95.4	84.0	35.4	24.6	59.9
Very fine-grained	45	95.0	83.3	34.8	23.9	59.3

Appendix

Table 6: Effect of failure-mode taxonomy granularity on CritICL performance.

Method	GSM8K	MATH	AMC23	AIME25	Avg.
5-shot ICL	93.8	80.5	32.0	21.0	56.8
Dense correct-exemplar retrieval	94.2	81.1	32.7	21.6	57.4
Generic GPT critique	94.4	81.4	33.1	22.0	57.7
Weak incorrect response only	94.3	81.0	32.5	21.4	57.3
Shuffled failure labels	94.5	81.6	33.0	22.1	57.8
CritICL-static	95.4	84.0	35.4	24.6	59.9

Appendix

Table 7: Source-of-gain ablation. Full CritICL consistently outperforms variants that remove or disrupt failure-mode alignment.

Method	Generations	MATH	AMC23	Avg.
5-shot ICL	1	80.5	32.0	56.3
Consistency@5	5	83.2	35.8	59.5
W2S-AlignTree	5	83.5	35.6	59.6
AdaptMI-style retrieval	1	82.1	33.4	57.8
CritICL-static	1	84.0	35.4	59.7

Appendix

Table 8: Comparison with inference-time weak-to-strong and test-time scaling baselines. CritICL requires only one target-model generation.

Dataset	Best Baseline	CritICL-static	$\Delta$	$p$ -value
GSM8K	$95.0\pm 1.1$	$95.4\pm 1.0$	$+0.4$	0.083
MATH	$83.2\pm 1.0$	$84.0\pm 0.9$	$+0.8$	0.018
AMC23	$35.8\pm 4.7$	$35.4\pm 4.6$	$-0.4$	0.641
AIME24	$26.3\pm 7.8$	$26.5\pm 7.9$	$+0.2$	0.812
AIME25	$24.5\pm 7.5$	$24.6\pm 7.6$	$+0.1$	0.871
Macro Avg.	$52.9\pm 1.6$	$53.2\pm 1.5$	$+0.3$	0.041

Appendix

Table 9: Bootstrap confidence intervals and paired significance tests against the strongest baseline. Values after

\pm

denote 95% bootstrap confidence intervals over evaluation examples.

Target Model	CritBank Source	GSM8K	MATH	AMC23	Avg.
Qwen2.5-72B	5-shot ICL	93.8	80.5	32.0	68.8
Qwen2.5-72B	Dense correct-exemplar retrieval	94.2	81.1	32.7	69.3
Qwen2.5-72B	Llama CritBank, cross-family	94.7	82.3	33.8	70.3
Qwen2.5-72B	Qwen CritBank, same-family	95.4	84.0	35.4	71.6
Llama-3.1-70B	5-shot ICL	89.6	72.8	27.5	63.3
Llama-3.1-70B	Dense correct-exemplar retrieval	90.1	73.5	28.2	63.9

Appendix

Table 10: Cross-family transfer between Qwen and Llama. Cross-family CritBank transfer improves over standard ICL and dense correct-exemplar retrieval, while same-family transfer remains stronger.

Target Domain	CritBank Source	Target Benchmarks	Avg.
Math	Math CritBank	MATH + AMC23	59.7
Math	GPQA CritBank	MATH + AMC23	58.4
Math	Mixed CritBank	MATH + AMC23	59.5
GPQA	Math CritBank	Chem. + Bio. + Phys. + Quantum	72.6
GPQA	GPQA CritBank	Chem. + Bio. + Phys. + Quantum	74.4
GPQA	Mixed CritBank	Chem. + Bio. + Phys. + Quantum	74.1

Appendix

Table 11: Cross-domain transfer between mathematical reasoning benchmarks and GPQA. Domain-specific CritBanks perform best, while mixed-domain CritBanks remain competitive.

Standard ICL
	In-distribution (ID))			Out-of-distribution (OOD))				Overall
Method	GSM8K	MATH	Avg.	AMC23	AIME24	AIME25	Avg.	Avg.
Zero-shot	78.5	45.2	61.9	18.4	12.5	11.2	14.0	38.0
1-shot (Rand.)	82.1	48.0	65.1	19.6	13.4	12.1	15.0	40.1
3-shot (Rand.)	86.3	52.5	69.4	21.5	14.9	13.5	16.6	43.0
5-shot (Rand.)	90.8	58.7	74.8	24.3	16.8	15.2	18.8	46.8

Appendix

Table 12: Performance comparison of Llama family models. The evaluated model is Llama-3.1-70B-Instruct, while the models used to contribute to CritBank include Llama-3.2-1B-Instruct and Llama-3.2-3B-Instruct and Llama-3.1-8B.

Standard ICL
Method	Generations	Input Tokens	Output Tokens	Total Tokens
Zero-shot	1	331	292	623
1-shot (Rand.)	1	947	307	1254
3-shot (Rand.)	1	2176	319	2495
5-shot (Rand.)	1	3518	304	3822
1-shot (Fixed)	1	934	312	1246

Appendix

Table 13: Inference cost comparison on MATH using Qwen2.5-72B-Instruct.

Standard ICL
Method	Generations	Input Tokens	Output Tokens	Total Tokens
Zero-shot	1	298	274	572
1-shot (Rand.)	1	882	281	1163
3-shot (Rand.)	1	1997	294	2291
5-shot (Rand.)	1	3226	279	3505
1-shot (Fixed)	1	867	288	1155

Appendix

Table 14: Inference cost comparison on MATH using LLaMA-3.1-70B-Instruct.

Standard ICL
Method	Generations	Input Tokens	Output Tokens	Total Tokens
Zero-shot	1	276	214	490
1-shot (Rand.)	1	731	228	959
3-shot (Rand.)	1	1604	241	1845
5-shot (Rand.)	1	2612	223	2835
1-shot (Fixed)	1	718	235	953

Appendix

Table 15: Inference cost comparison on GSM8K using LLaMA-3.1-70B-Instruct.

Standard ICL
Method	Chemistry	Biology	Physics	Quantum	Avg.
Zero-shot	60.8	62.5	58.9	55.2	59.4
1-shot (Rand.)	63.4	65.1	61.2	57.6	61.8
3-shot (Rand.)	66.9	68.7	64.5	60.8	65.2
5-shot (Rand.)	69.5	71.0	67.1	63.2	67.7
1-shot (Fixed)	64.1	65.9	61.8	58.3	62.5

Appendix

Table 16: Performance comparison on the GPQA benchmark across different scientific domains, including Chemistry, Biology, Physics, and Quantum Mechanics. The target model is Llama-3.1-70B-Instruct, and CritBank is constructed using weaker LLaMA models. Results show that CritICL consistently improves performance across all domains.

\mathcal{D}(S)\leftarrow\left\{(q,r,\mathcal{C}(q,r))\;\middle|\;\mathcal{L}(q,r)\cap S\neq\emptyset\right\}

Appendix

Algorithm 1 Failure Mode-Based Sample Selection

S_{\text{inst}}(q^{\prime})=\{l_{1},\dots,l_{m}\},\quad m\leq 5

Appendix

Algorithm 2 CritICL-dynamic Inference Workflow

P_{\mathcal{M}}(l),\quad l\in\mathcal{F}

Appendix

Algorithm 3 CritICL-static Inference Workflow

Failure Modes	Explanation
algebraic_equivalence_misinterpretation	Misunderstanding two algebraic expressions that are mathematically equivalent.
algebraic_manipulation_miscalculation	Making computational errors during algebraic transformations.
algebraic_sign_error	Incorrect handling of positive or negative signs in expressions.
arithmetic_operation_mistake	Performing basic arithmetic operations incorrectly.
equation_solving_miscalculation	Making errors while solving equations.
incorrect_factorization	Factoring expressions incorrectly.

Appendix

Table 17: Algebraic and arithmetic failure modes.

Failure Modes	Explanation
incorrect_formula_application	Using an inappropriate or incorrect formula.
inconsistent_formula_usage	Using formulas inconsistently within the same solution.
function_definition_misinterpretation	Misunderstanding how a function is defined or behaves.
geometric_relationship_misinterpretation	Misinterpreting spatial or geometric relationships.
graph_interpretation_error	Drawing incorrect conclusions from a graph or visual representation.
combinatorial_principle_misapplication	Applying incorrect counting principles or combinatorial rules.

Appendix

Table 18: Conceptual and domain-specific failure modes.

Failure Modes	Explanation
problem_intent_misinterpretation	Misunderstanding what the problem is asking.
problem_requirement_misunderstanding	Failing to follow specific problem requirements.
ambiguous_problem_parameters	Misinterpreting unclear or implicitly defined problem parameters.
insufficient_constraint_understanding	Misunderstanding or ignoring problem constraints.
underestimation_of_constraints	Overlooking constraints that affect the solution space.
boundary_value_misinterpretation	Misunderstanding or incorrectly applying boundary conditions.

Appendix

Table 19: Reasoning and problem-understanding failure modes.

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 371 papers

Mar 7, 2026

Improving reasoning at inference time via uncertainty minimisation

Large language models (LLMs) now exhibit strong multi-step reasoning abilities, but existing inference-time scaling methods remain computationally expensive, often relying on extensive sampling or external evaluators. We propose a principled strategy that frames reasoning as uncertainty minimisation and operates at the level of individual thoughts rather than tokens. Our method selects, at each reasoning step, the continuation that maximizes the model's self-certainty, a metric computed from its internal predictive distribution. This approach achieves significant improvement with a small number of samples, relies exclusively on model-internal signals, and applies to open-ended questions as opposed to methods like majority voting. Experiments on MATH500 and GSM8K across multiple model sizes demonstrate that thought-level self-certainty maximization consistently outperforms greedy decoding and matches or exceeds self-consistency under comparable token budgets. Cross-linguistic evaluations further indicate that the method transfers robustly beyond high-resource languages. Furthermore, analysis of self-certainty dynamics reveals that correct reasoning trajectories converge early to stable paths, suggesting that early decisions, likely associated with the planning of the reasoning process, are predictive of final accuracy. Building on this result, we show that self-certainty maximisation applied to the early steps can explain most of the performance gain and provide a simple yet efficient inference-time scaling method.

Nicolas Legrand, Kenneth Enevoldsen, Márton Kardos +1

Center for Humanities Computing Aarhus University, Denmark

Efficient Language Model Reasoning Inference-Time Scaling Language Model Decoding

Ask PDF

Apr 23, 2026

Process Supervision via Verbal Critique Improves Reasoning in Large Language Models

Inference-time scaling for LLM reasoning has focused on three axes: chain depth, sample breadth, and learned step-scorers (PRMs). We introduce a fourth axis, granularity of external verbal supervision, via Verbal Process Supervision (VPS), a training-free framework that uses structured natural-language critique from a stronger supervisor to guide an iterative generate-critique-refine loop up to a round budget R. Across GPQA Diamond, AIME 2025, and LiveCodeBench V6 (covering both closed and open models), VPS yields three key results. First, on GPQA Diamond, GPT-5.4 (High) | GPT-5.4 (Low) reaches 94.9% at R=4, surpassing the 94.1% state of the art without gradient updates. Second, on AIME 2025, VPS enables strong weak-actor rescue, boosting scores from 11.7-26.7% to 63.3-90.0% (up to +63.3 points). Third, at matched compute, VPS outperforms Reflexion by +8.5 to +12.1 points and Self-Consistency@5 by +5.0 pp (GPQA) and +8.3 pp (LiveCodeBench), isolating critique granularity as the key driver. Performance scales with the supervisor-actor capability gap (Pearson r=0.90) and degrades when errors are not linguistically expressible (e.g., code synthesis), motivating hybrid verbal-executable methods. These results establish critique granularity as a new axis of inference-time scaling.

Hao-Yuan Chen

Mindify AI Research · University of London · Senate House, Malet Street, London WC1E 7HU, United Kingdom

Inference-Time Scaling Process Supervision LLM Reasoning

Ask PDF

Mar 5, 2026

$\nabla$ -Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space

Scaling inference-time compute for Large Language Models (LLMs) has unlocked unprecedented reasoning capabilities. However, existing inference-time scaling methods typically rely on inefficient and suboptimal discrete search algorithms or trial-and-error prompting to improve the online policy. In this paper, we propose

\nabla

-Reasoner, an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop to refine the policy on the fly. Our core component, Differentiable Textual Optimization (DTO), leverages gradient signals from both the LLM's likelihood and a reward model to refine textual representations.

\nabla

-Reasoner further incorporates rejection sampling and acceleration design to robustify and speed up decoding. Theoretically, we show that performing inference-time gradient descent in the sample space to maximize reward is dual to aligning an LLM policy via KL-regularized reinforcement learning. Empirically,

\nabla

-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark, while reducing number of model calls by approximately 10-40% compared to strong baselines. Overall, our work introduces a paradigm shift from zeroth-order search to first-order optimization at test time, offering a cost-effective path to amplify LLM reasoning.

Peihao Wang, Ruisi Cai, Zhen Wang +4

The University of Texas at Austin · UC San Diego · TTIC +1

LLM Reasoning Test-Time Optimization Efficient Language Model Reasoning Inference-Time Language Model Alignment

Ask PDF

Abstract

Efficient Language Model Reasoning

Explore similar work

Improving reasoning at inference time via uncertainty minimisation

Process Supervision via Verbal Critique Improves Reasoning in Large Language Models

∇\nabla∇-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space

$\nabla$ -Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space