StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models

Jinghan Tan; Yuanzheng Wang; Lu Chen; Zijun Chen; Yuqian Wang; Maosong Sun

cs.AIAug 24, 2026

StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models

Authors: Jinghan Tan, Yuanzheng Wang, Lu Chen, Zijun Chen, Yuqian Wang, Maosong Sun

Abstract

As large language models are increasingly used in data-scarce and evolving task scenarios, few-shot in-context learning (ICL) has become a key paradigm for task adaptation. However, direct ICL often uses a small set of examples without explicitly abstracting task rules, making it sensitive to example construction. In contrast, human learners often reduce such sensitivity by first summarizing task rules from examples and then applying them to new instances. To evaluate this ability, we propose StrategyBench, which selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility. We further analyze strategy induction from three perspectives: task variation, model configuration, and adaptation setting, covering category-wise differences, generator-executor choices, demonstration design, and SFT-based adaptation. Experiments show that explicit strategy utility differs substantially across task categories and depends on both strategy generation and execution conditions. The benchmark is released at: https://anonymous.4open.science/r/StrategyBench-D53C.

Figures & tables

Figure 1: Comparison between direct few-shot ICL and strategy-based solving. In direct few-shot ICL, the model directly predicts the answer for a new query from the provided input-output examples. In strategy-based solving, the model first induces an explicit task-level strategy from the few-shot examples and then applies the induced strategy to solve the unseen query.

Figure 2: Overall data construction pipeline of StrategyBench . Starting from BIG-Bench, we first filter tasks with the assistance of LLM-based pre-screening and human verification to obtain a suitable task pool. The filtered tasks are then processed along two independent dimensions: dataset splitting, which produces training, ID test, and OOD test subsets, and task categorization, which assigns tasks to coarse-grained categories. Based on the resulting task types and few-shot examples, candidate strategies are generated and further filtered by downstream utility and statistical significance to form the effective strategy set $\mathcal{S}_{\mathrm{eff}}$ and the significant reference strategy set $\mathcal{S}_{\mathrm{sig}}$ .

Stage	#Tasks	#Subtasks	#Examples
Before filtering	108	581	1,614,716
After filtering	44	356	123,845

Table 1: Dataset statistics before and after task filtering.

Strategy Set	#Tasks	#Subtasks	#Strategies
$\mathcal{S}_{\mathrm{eff}}$	22	241	11,993
$\mathcal{S}_{\mathrm{sig}}$	21	221	7,801

Table 2: Statistics of the constructed reference strategy sets.

\mathcal{S}_{\mathrm{eff}}

contains strategies that improve over Direct ICL, while

\mathcal{S}_{\mathrm{sig}}

further retains strategies with statistically significant gains.

Metric	Method	Task Category						BBH
Metric	Method	Numerical	Logic	Language	Spatial	Procedural	Induction	BBH
Corr.	L2M	58.43	42.34	22.21	6.21	9.47	27.50	24.09
	Direct ICL	34.31	11.18	4.40	0.69	0.76	0.55	11.55
	Ours	15.90	38.58	22.11	3.59	0.97	17.03	15.34
	CoT	57.30	43.12	19.43	4.71	19.91	28.99	21.40
	CoT+Ours	69.58	55.32	36.09	8.62	25.95	40.90	37.79

Table 3: Utility comparison across different task categories and BBH. Results are computed over all tasks within each category. Bold values indicate the best performance among the five methods for each metric within the same task category or BBH setting.

Generator	Strategy Quality		Strategy Utility
	Conc.	Fmat.	Exec. 1.7B		Exec. 4B		Exec. 8B
	Conc.	Fmat.	Corr.	Sta samp	Corr.	Sta samp	Corr.	Sta samp
1.7B	90.98	34.88	14.50	48.30	18.48	54.97	15.86	54.49
4B	90.03	25.12	13.27	42.44	19.41	53.66	14.96	53.98
8B	91.57	22.09	14.57	45.03	20.86	56.83	17.74	54.30
14B	92.98	38.84	13.21	48.51	20.83	57.51	16.74	54.73

Table 4: Effect of generator and executor model scales. We compare four strategy generators and evaluate their generated strategies with three executors. Bold values indicate the best result under the same metric and executor setting.

$k$	Strategy Quality		Strategy Utility
$k$	Conc.	Fmat.	Corr.	Sta samp
1	92.67	31.33	17.34	46.51
3	91.57	22.09	20.89	48.69
5	91.54	21.83	21.26	48.06
10	90.77	15.41	21.19	48.41
30	89.83	14.58	25.49	51.29

Table 5: Effect of the number of few-shot examples on strategy generation and downstream application under

k=1,3,5,10,30

. Bold values indicate the best performance across different few-shot settings.

Prompt Type	Strategy Quality		Strategy Utility
Prompt Type	Conc.	Fmat.	Corr.	Sta samp
Structured Prompt	91.60	22.79	20.86	48.55
Free-form Prompt	90.91	57.38	16.96	45.02

Table 6: Comparison between structured and free-form prompts for strategy generation and downstream application. Bold values indicate the better result between the two prompt types under the same metric.

Method	ID					OOD					BBH
	Quality		Utility			Quality		Utility			Quality		Utility
	Conc.	Fmat.	Corr.	Sta samp	Sta rob	Conc.	Fmat.	Corr.	Sta samp	Sta rob	Conc.	Fmat.	Corr.	Sta samp	Sta rob
Ours	92.13	6.68	22.71	59.61	68.08	91.46	21.94	25.42	59.67	68.18	91.77	5.15	15.34	51.90	62.32
Ours+SFT	94.17	62.06	21.33	70.08	77.79	93.03	61.95	23.29	62.89	72.58	93.44	50.35	14.20	66.51	76.41

Table 7: Comparison between Ours and Ours+SFT on ID, OOD, and BBH test sets. Each split reports two strategy quality metrics and three strategy utility metrics. Bold values indicate the best performance between the two variants under the same evaluation setting.

Benchmark / Eval.	Ans. Eval.	Proc. Eval.	Strategy Ind.	Strategy Utility
MMLU ( Hendrycks et al., 2021 )	✓	$\times$	$\times$	$\times$
BBH ( Suzgun et al., 2023 )	✓	$\times$	$\times$	$\times$
GSM8K ( Cobbe et al., 2021 )	✓	$\times$	$\times$	$\times$
Step Verif. ( Lightman et al., 2024 )	✓	✓	$\times$	$\times$
Trace Eval. ( Lee and Hockenmaier, 2025 )	✓	✓	$\times$	$\times$
Ours	✓	✓	✓	✓

Table 8: Comparison of representative benchmarks and process-level evaluations. Ans. Eval., Proc. Eval., Strategy Ind., and Strategy Utility denote answer-level evaluation, process-level evaluation, strategy induction, and strategy utility evaluation, respectively.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Example Value
task	disambiguation_qa
subtask	disambiguation_qa
input	In the following sentences, explain the antecedent of the pronoun. Options: (A) … (B) … (C) Ambiguous.
target	["A"]

Appendix

Table 9: Example of a standardized instance.

Hyperparameter	Value
Base model	Qwen3-8B
Max length	4096
Max steps	2200
Batch size	1
Gradient accumulation	8
Learning rate	$1\times 10^{-4}$

Appendix

Table 10: Main hyperparameter settings for supervised fine-tuning.

Figure 3: Example of the JSONL-style data structure used for strategy supervised fine-tuning. Each instance contains the strategy-generation instruction, the few-shot QA block, and the reference strategy as the supervised target.

Figure 4: Prompt V1 used for strategy generation.

Figure 5: Prompt V2 used for strategy generation.

Figure 6: Overview of the concurrent CoDI framework. CoDI uses StrategyBench as a training and evaluation testbed and leverages an SFT model trained from reference strategies. The framework separates task-level strategy induction from answer execution, serving as a downstream method built on the benchmark resources introduced in this paper.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Strategy-Induct: Task-Level Strategy Induction for Instruction Generation
May 20, 2026Po-Chun Chen, Hen-Hsen Huang, Hsin-Hsi ChenLLM Prompting Few-Shot Prompting
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · AI Research Center (AINTU), National Taiwan University, Taiwan
Data Efficient Sample Selection for In-Context Learning
Sep 6, 2026V Venktesh, Cem Levi, Avishek AnandIn-Context Example Selection In-Context Learning
Stockholm University · SKIM Group B.V. · TU Delft
HERO'S JOURNEY: Testing Complex Rule Induction with Text Games
Jun 1, 2026Anshun Asher Zheng, Kanishka Misra, David I. Beaver +1LLM Agent Evaluation AI Agent Benchmarks
Department of Linguistics The University of Texas at Austin

May 20, 2026J/K move · Enter open · S save

Strategy-Induct: Task-Level Strategy Induction for Instruction Generation

Po-Chun Chen, Hen-Hsen Huang, Hsin-Hsi Chen

Department of Computer Science and Information Engineering, National Taiwan University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · AI Research Center (AINTU), National Taiwan University, Taiwan

LLM Prompting Few-Shot Prompting LLM Reasoning

Open paper AskPDF arXiv

Designing effective task-level prompts is crucial for improving the performance of Large Language Models (LLMs). While prior work on instruction induction demonstrates that LLMs can infer better instructions with limited examples, existing approaches often rely on input-output pairs, where obtaining labeled answers can be difficult or costly. To address this limitation, we propose Strategy-Induct, a framework that derives task-level instructions solely from a small set of example questions without requiring labeled answers. Our approach first prompts the model to generate explicit reasoning strategies for each question, forming (strategy, question) pairs. These pairs are then used to induce a task instruction that guides reasoning. Experiments across multiple tasks and model scales demonstrate that Strategy-Induct outperforms state-of-the-art methods in question-only settings. Furthermore, we observe that jointly utilizing LLMs and Large Reasoning Models across task instruction generation and inference may lead to further performance improvements.

Abstract

LLM Evaluation

Explore similar work

Strategy-Induct: Task-Level Strategy Induction for Instruction Generation

Data Efficient Sample Selection for In-Context Learning

HERO'S JOURNEY: Testing Complex Rule Induction with Text Games