Batch-of-Thought: Cross-Instance Learning for Enhanced LLM Reasoning

Xuan Yang; Furong Jia; Roy Xie; Xi Xiong; Hengwei Bian; Jian Li; Monica Agrawal

cs.AIJan 6, 2026

Batch-of-Thought: Cross-Instance Learning for Enhanced LLM Reasoning

Authors: Xuan Yang, Furong Jia, Roy Xie, Xi Xiong, Hengwei Bian, Jian Li, Monica Agrawal

Organizations: Duke University · ByteDance Inc.

Abstract

Current Large Language Model reasoning systems process queries independently, discarding valuable cross-instance signals such as shared reasoning patterns and consistency constraints. We introduce Batch-of-Thought (BoT), a training-free method that processes related queries jointly to enable cross-instance learning. By performing comparative analysis across queries, BoT identifies high-quality reasoning templates, detects errors through consistency checks, and amortizes computational costs. We instantiate BoT within a multi-agent reflection architecture (BoT-R), where a Reflector performs joint evaluation to unlock mutual information gain unavailable in isolated processing. Experiments across three model families and six benchmarks demonstrate that BoT-R consistently improves accuracy and confidence calibration while reducing inference costs by up to 61%. Our theoretical and experimental analysis reveals when and why batch-aware reasoning benefits LLM systems. Our code is available at https://github.com/xuanyang19/BoT.

Figures & tables

Figure 1: Batch-of-Thought reflection framework. An Actor generates initial responses for a batch of queries. A Reflector then jointly evaluates all responses through comparative analysis, determining whether each should be finalized or refined with feedback.

Model	Method	FraudDet	GPQA	Winogrande	MedQA	PubMedQA	SMS Spam
		$n=1793$	$n=448$	$n=1267$	$n=1273$	$n=1000$	$n=1510$
GPT-4o	ReAct	0.685	0.439	0.872	0.878	0.679	0.796
	Reflection	0.693	0.459	0.879	0.901	0.667	0.854
	BoT-R	0.740 (4)	0.488 (4)	0.890 (8)	0.904 (8)	0.698 (8)	0.887 (8)
Llama-3.3-70B	ReAct	0.635	0.494	0.831	0.783	0.753	0.920
	Reflection	0.679	0.504	0.853	0.797	0.755	0.925

Table 1: Performance comparison of reasoning methods across various base models and datasets (number of queries listed as

n

). Scores represent accuracy. For BoT-R, the number in parentheses denotes the batch size

N\in\{4,8\}

used to obtain the reported result. The best result in each setting is highlighted in bold.

Actor
	SMS_Spam		GPQA		Winogrande
Method	Cost	$\Delta\%$	Cost	$\Delta\%$	Cost	$\Delta\%$
Reflection	$3.61	–	$4.70	–	$2.57	–
BoT (4)	$2.41	33.25%	$3.68	21.71%	$2.27	11.75%
BoT (8)	$2.06	42.96%	$3.42	27.24%	$1.99	22.54%
Reflector

Table 2: Total token cost and relative reduction (

\Delta\%

) for each dataset (GPT-4o).

\Delta\%

is computed for BoT(4) and BoT(8) relative to Reflection cost.

Method	SMS_Spam		GPQA		Winogrande
Method	KS↑	ECE↓	KS↑	ECE↓	KS↑	ECE↓
ReAct	0.256	0.176	0.181	0.372	0.273	0.113
Reflection	0.360	0.104	0.265	0.329	0.376	0.035
BoT-R	0.633	0.063	0.368	0.317	0.442	0.013

Table 3: Confidence calibration across datasets (GPT-4o). Each entry reports the KS statistic between the confidence distributions of correct vs. incorrect answers and the ECE score. Higher KS (

\uparrow

) is better, while lower ECE (

\downarrow

) is better.

Method	med&bio	hum	social	math	sci
ReAct	0.887	0.805	0.915	0.763	0.797
Reflection	0.886	0.825	0.915	0.865	0.843
BoT-R	0.891	0.837	0.922	0.853	0.832

Table 4: MMLU dataset subject-wise accuracy (GPT-4o). The highest score for each subject is in bold.

Dataset	No-batch	Sequential	Semantic
FraudDet	0.693	0.740	0.768
MedQA	0.901	0.904	0.902
PubMedQA	0.667	0.698	0.697
Winogrande	0.879	0.890	0.897
SMS Spam	0.854	0.887	0.902
GPQA	0.459	0.488	0.486

Table 5: Accuracy across batching strategies (GPT-4o). The highest score for each dataset is in bold.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	SMS_Spam				GPQA				Winogrande
	In	Out	Cost	$\Delta$ %	In	Out	Cost	$\Delta$ %	In	Out	Cost	$\Delta$ %
Actor
Reflection	105,936	334,429	$3.61		196,763	421,168	$4.70		115,788	228,050	$2.57
BoT (4)	75,838	221,962	$2.41	33.25%	176,381	324,153	$3.68	21.71%	103,757	200,847	$2.27	11.75%
BoT (8)	65,268	189,556	$2.06	42.96%	146,350	305,647	$3.42	27.24%	91,716	176,136	$1.99	22.54%
Reflector

Appendix

Table 6: Input and output token usage and cost across different methods, decomposed into Actor , Reflector , and Total stages (GPT-4o). 1M input tokens cost

\

2.5

and 1M output tokens cost

$10

.

\Delta%$ is the cost reduction, computed for BoT(4) and BoT(8) relative to Reflection for the same dataset and stage.

Formation total (% of BoT(4) token cost)
Dataset	$\boldsymbol{n}$	Embed	$k$ -means
GPQA	448	$0.0275	$0.00007
Winogrande	1267	$0.0053	$0.00046
SMS Spam	1510	$0.0103	$0.00066
GPQA	$0.028 (0.53%)
Winogrande	$0.006 (0.14%)

Appendix

Table 7: One-off cost of forming semantically coherent batches. Percentages are taken relative to the corresponding BoT(4) total token cost in Table 2 .

Method	Winogrande		GPQA
Method	Acc	Cost	Acc	Cost
ReAct	87.20	$1.92	43.90	$2.28
+ majority vote ( $s{=}3$ )	86.98	$5.71	41.39	$6.75
Reflection	87.90	$5.97	45.90	$7.46
+ majority vote ( $s{=}3$ )	88.86	$23.52	46.65	$34.21
BoT-R	89.00	$3.51	48.80	$5.17

Appendix

Table 8: Majority voting compared with BoT-R (GPT-4o). Accuracy in percent, cost in USD over the full dataset. Majority-vote rows aggregate

s=3

samples at temperature

0.7

; all other rows use temperature

0

. BoT-R uses the batch size reported in Table 1 .

Question	Initial	After BoT-R
Who knew vaccines do not	Mary ✓	Mary ✓
cause autism?	Mary ✓	Mary ✓
Who believed vaccines do	Mary $\times$	Betty ✓
cause autism?	Mary $\times$	Betty ✓

Appendix

Table 9: Case 1. Answered in isolation, the Actor assigns both opposing beliefs to the same character; joint evaluation surfaces the contradiction and corrects the second item.

Batch item	Reflector decision
Organic named-reaction	Accept
Genetic inheritance	Accept
Stem-cell culture	Accept
Diene reactivity	Re-evaluate

Appendix

Table 10: Case 2. Joint evaluation of a four-item batch flags only the answer that lacks mechanism-level support, leaving the other three untouched.

Batch	FraudDet	GPQA	Winogrande	MedQA	PubMedQA	SMS Spam
1 (Reflection)	0.693	0.459	0.879	0.901	0.667	0.854
4	0.740	0.488	0.888	0.895	0.683	0.881
8	0.732	0.471	0.890	0.904	0.698	0.887

Appendix

Table 11: Batch size influence on accuracy (GPT-4o) across the full benchmark suite. Batch size

N=1

is functionally equivalent to per-instance Reflection.

Dataset	$\boldsymbol{N=1}$	$\boldsymbol{N=2}$	$\boldsymbol{N=4}$	$\boldsymbol{N=6}$	$\boldsymbol{N=8}$	$\boldsymbol{N=12}$	$\boldsymbol{N=16}$
Winogrande	0.879	0.886	0.888	0.893	0.890	0.893	0.887
SMS Spam	0.854	0.887	0.881	0.909	0.887	0.912	0.921
GPQA	0.459	0.488	0.488	0.468	0.471	0.472	0.456

Appendix

Table 12: Extended batch size stress test on three representative datasets (GPT-4o). Accuracy is reported across a wider range of batch sizes to evaluate scaling behavior, stability, and context saturation boundaries.

Method	EF-ECE $\downarrow$	Brier $\downarrow$	NLL $\downarrow$	KS $\uparrow$
GPQA
ReAct	51.61%	0.5023	1.7006	0.181
Reflection	44.81%	0.4412	1.5211	0.265
BoT-R	40.66%	0.4015	1.4761	0.368
Winogrande
ReAct	6.58%	0.1107	0.3720	0.273

Appendix

Table 13: Bias-robust calibration metrics (GPT-4o). EF-ECE is equal-frequency ECE with 10 bins; Brier and binary NLL are strictly proper scoring rules; KS is reproduced from Table 3 . Best value per column and dataset in bold.

Dataset	Comparison	Exact $p$
GPQA	ReAct $\rightarrow$ Reflection	0.011
	ReAct $\rightarrow$ BoT-R	0.008
	Reflection $\rightarrow$ BoT-R	0.048
Winogrande	ReAct $\rightarrow$ Reflection	0.007
	ReAct $\rightarrow$ BoT-R	0.001
	Reflection $\rightarrow$ BoT-R	0.324

Appendix

Table 14: Exact McNemar tests on paired per-item outcomes from the deterministic (temperature

0

) runs, GPT-4o.

Dataset	ReAct	Reflection	BoT-R
GPQA	42.53 $\pm$ 1.31	45.58 $\pm$ 0.44	47.44 $\pm$ 1.18
Winogrande	86.77 $\pm$ 0.12	88.23 $\pm$ 0.22	89.27 $\pm$ 0.15

Appendix

Table 15: Run-to-run variance at temperature

0.7

over three seeds (GPT-4o); accuracy in percent, mean

\pm

SD. Reported separately from Table 14 , since stochastic runs cannot enter a paired test.

Subject	ReAct	Reflection	BoT-R
Biology
anatomy	0.907	0.918	0.910
clinical_knowledge	0.913	0.898	0.913
college_biology	0.941	0.931	0.965
college_medicine	0.852	0.855	0.866
high_school_biology	0.959	0.961	0.964

Appendix

Table 16: MMLU per-subject accuracy summary grouped by category (GPT-4o).

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 952 papers

Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoning
Feb 20, 2026Lexiang Tang, Weihao Gao, Bingchen Zhao +4Efficient Language Model Reasoning LLM Reliability
Peking University, Beijing, China · University of Edinburgh, Edinburgh, UK.
Enhancing reasoning accuracy in large language models during inference time
Mar 22, 2026Vinay Sharma, Manish JainLLM Inference LLM Reasoning
One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinement
Apr 28, 2026Yixiao Zhou, Dongzhou Cheng, zhiliang wu +3LLM Alignment RL for Language Model Reasoning
Zhejiang University · Shanghai Innovation Institute · 1Zhejiang University, 2Shanghai Innovation Institute +9

Feb 20, 2026J/K move · Enter open · S save

Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoning

Lexiang Tang, Weihao Gao, Bingchen Zhao, Lu Ma+3

Peking University, Beijing, China · University of Edinburgh, Edinburgh, UK.

Efficient Language Model Reasoning LLM Reliability Contrastive Decoding Numerical Reasoning in Language Models

Open paper AskPDF arXiv

Recent work on test-time scaling for large language model (LLM) reasoning typically assumes that allocating more inference-time computation uniformly improves correctness. However, prior studies show that reasoning uncertainty is highly localized: a small subset of low-confidence tokens disproportionately contributes to reasoning errors and unnecessary output expansion. Motivated by this observation, we propose Thinking by Subtraction, a confidence-driven contrastive decoding approach that improves reasoning reliability through targeted token-level intervention. Our method, Confidence-Driven Contrastive Decoding, detects low-confidence tokens during decoding and intervenes selectively at these positions. It constructs a contrastive reference by replacing high-confidence tokens with minimal placeholders, and refines predictions by subtracting this reference distribution at low-confidence locations. Experiments show that CCD significantly improves accuracy across mathematical reasoning benchmarks while substantially reducing output length, with minimal KV-cache overhead. As a training-free method, CCD enhances reasoning reliability through targeted low-confidence intervention without computational redundancy. Our code will be made available at: https://github.com/bolo-web/CCD.

Abstract

LLM Reasoning

Explore similar work

Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoning

Enhancing reasoning accuracy in large language models during inference time

One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinement