Cascaded Batch Prompting

Sho Hoshino; Peinan Zhang

cs.CLAug 27, 2026

Cascaded Batch Prompting

Authors: Sho Hoshino, Peinan Zhang

Organizations: CyberAgent

Abstract

Although batch prompting makes large language model inference more efficient by processing multiple instances simultaneously, it suffers from unpredictable downstream task performance. We propose cascaded batch prompting, a two-stage approach designed to resolve the unpredictability of conventional batch prompting by disentangling complex reasoning from symbol grounding. Experiments on multiple-choice question answering and natural language inference demonstrate that the proposed method outperforms the standard single prompting baseline while achieving a speedup proportional to batch size, establishing a new state of the art on the Pareto frontier.

Figures & tables

Figure 1: Cascaded batch prompting (top) disentangles reasoning from symbol grounding into a two-stage process, while conventional batch prompting (bottom) handles both in one step.

		Accuracy (%) $\uparrow$
Prompt	Batch Size	MMLU	MNLI-m
GPT-4.1
Single	–	84.39	82.19
Batch	32	85.51	86.14 ∗
Cascaded Batch	32	86.81 ∗	85.09
GPT-4.1-mini

Table 1: Comparison of various prompting strategies. For a fair comparison, we chose the batch size of 32 that performed best for the conventional batch prompting baseline. Bold text highlights the highest scores. The asterisk ( ∗ ) denotes statistical significance (

p<0.05

, detailed in Appendix A ).

Figure 2: Accuracy scaling with batch size. Cascaded batch prompting demonstrates robust performance across tasks and models. Its advantage is most pronounced with GPT-4.1-mini (bottom row), where it consistently outperforms the baselines.

Figure 3: Compared with single prompting, cascaded batch prompting achieves higher accuracy and throughput at the same time, establishing a new Pareto frontier. Meanwhile, conventional batch prompting underperforms, demonstrating its unpredictability.

Figure 4: Cascaded single prompting outperforms the conventional baseline, demonstrating the effectiveness of the two-stage design even without batching.

Prompt	Batch Size	MMLU
		Accuracy (%) $\uparrow$
GPT-4.1-mini
Cascaded Batch
w/o Sanity Check	32	82.96
	128	81.41
w/ Sanity Check	128	81.61

Table 2: Impact of the sanity check on cascaded batch prompting. The sanity check partially recovers performance lost to input and output misalignment, although it cannot fully resolve the performance decline at larger batch sizes.

		Throughput (tokens/sec) $\downarrow$
		MMLU
Prompt	Batch Size	Input	Output
GPT-4.1
Single	–	4,910	47
Batch	32	18,990	965
Cascaded Batch	32	23,341	1,220

Table 3: Cascaded batch prompting introduces a modest cost overhead due to the additional symbol grounding stage.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Listing cascaded batch prompts used in the MCQA task. The placeholders “{question}”, “{options}”, and “{answer}” are replaced with the actual question, its options, and LLM answer.

Panel of Figure 6 — Figure 6: Listing cascaded batch prompts used in the NLI task. The placeholders “{premise}”, “{hypothesis}”, and “{answer}” are replaced with the actual premise, hypothesis, and LLM answer.

Panel of Figure 7 — Figure 7: Listing conventional batch prompts used in the MCQA and NLI tasks, respectively. Unlike cascaded batch, these prompts are tailored with symbol grounding in mind.

LLM Prompting

Large language model prompting studies how instructions, examples, context, and formats steer model behavior at inference time.

See all 431 papers

Batched Contextual Reinforcement: A Task-Scaling Law for Efficient Reasoning
Apr 2, 2026Bangji Yang, Hongbo Ma, Jiajun Fan +1Efficient Language Model Reasoning Efficient Language Model Inference
University of Illinois Urbana-Champaign · Tsinghua University
Batch-wise Adaptive Pruning: Periodic Neuron Activation-Aware Weight Pruning for Language Reasoning Model
Aug 14, 2026Yongmin Kim, Shota Takashiro, Yusuke Iwasawa +2LLM Pruning Efficient Language Model Inference
The University of Tokyo
Intra-Prompt Parallel Decoding for Common-Context Question Answering
Sep 4, 2026Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani +3LLM Inference Acceleration Language Model Decoding
Amazon.com, Inc. · McGill University · Mila +1

Apr 2, 2026J/K move · Enter open · S save

Batched Contextual Reinforcement: A Task-Scaling Law for Efficient Reasoning

Bangji Yang, Hongbo Ma, Jiajun Fan, Ge Liu

University of Illinois Urbana-Champaign · Tsinghua University

Efficient Language Model Reasoning Efficient Language Model Inference LLM Reasoning RL for Language Model Reasoning

Open paper AskPDF arXiv

Large Language Models employing Chain-of-Thought reasoning achieve strong performance but suffer from excessive token consumption that inflates inference costs. Existing efficiency methods such as explicit length penalties, difficulty estimators, or multi-stage curricula either degrade reasoning quality or require complex training pipelines. We introduce Batched Contextual Reinforcement, a minimalist, single-stage training paradigm that unlocks efficient reasoning through a simple structural modification: training the model to solve N problems simultaneously within a shared context window, rewarded purely by per-instance accuracy. This formulation creates an implicit token budget that yields several key findings: (1) We identify a novel task-scaling law: as the number of concurrent problems N increases during inference, per-problem token usage decreases monotonically while accuracy degrades far more gracefully than baselines, establishing N as a controllable throughput dimension. (2) BCR challenges the traditional accuracy-efficiency trade-off by demonstrating a "free lunch" phenomenon at standard single-problem inference. Across both 1.5B and 4B model families, BCR reduces token usage by 15.8% to 62.6% while consistently maintaining or improving accuracy across five major mathematical benchmarks. (3) Qualitative analyses reveal emergent self-regulated efficiency, where models autonomously eliminate redundant metacognitive loops without explicit length supervision. (4) Crucially, we empirically demonstrate that implicit budget constraints successfully circumvent the adversarial gradients and catastrophic optimization collapse inherent to explicit length penalties, offering a highly stable, constraint-based alternative for length control. These results prove BCR practical, showing simple structural incentives unlock latent high-density reasoning in LLMs.

Abstract

LLM Prompting

Explore similar work

Batched Contextual Reinforcement: A Task-Scaling Law for Efficient Reasoning

Batch-wise Adaptive Pruning: Periodic Neuron Activation-Aware Weight Pruning for Language Reasoning Model

Intra-Prompt Parallel Decoding for Common-Context Question Answering