Although batch prompting makes large language model inference more efficient by processing multiple instances simultaneously, it suffers from unpredictable downstream task performance. We propose cascaded batch prompting, a two-stage approach designed to resolve the unpredictability of conventional batch prompting by disentangling complex reasoning from symbol grounding. Experiments on multiple-choice question answering and natural language inference demonstrate that the proposed method outperforms the standard single prompting baseline while achieving a speedup proportional to batch size, establishing a new state of the art on the Pareto frontier.
Figures & tables
Figure 1: Cascaded batch prompting (top) disentangles reasoning from symbol grounding into a two-stage process, while conventional batch prompting (bottom) handles both in one step.
Accuracy (%) ↑
Prompt
Batch Size
MMLU
MNLI-m
GPT-4.1
Single
–
84.39
82.19
Batch
32
85.51
86.14 ∗
Cascaded Batch
32
86.81 ∗
85.09
GPT-4.1-mini
Table 1: Comparison of various prompting strategies. For a fair comparison, we chose the batch size of 32 that performed best for the conventional batch prompting baseline. Bold text highlights the highest scores. The asterisk ( ∗ ) denotes statistical significance ( p<0.05 , detailed in Appendix A ).
Figure 2: Accuracy scaling with batch size. Cascaded batch prompting demonstrates robust performance across tasks and models. Its advantage is most pronounced with GPT-4.1-mini (bottom row), where it consistently outperforms the baselines.
Figure 3: Compared with single prompting, cascaded batch prompting achieves higher accuracy and throughput at the same time, establishing a new Pareto frontier. Meanwhile, conventional batch prompting underperforms, demonstrating its unpredictability.
Figure 4: Cascaded single prompting outperforms the conventional baseline, demonstrating the effectiveness of the two-stage design even without batching.
Accuracy (%) ↑
Prompt
Batch Size
MMLU
GPT-4.1-mini
Cascaded Batch
w/o Sanity Check
32
82.96
128
81.41
w/ Sanity Check
128
81.61
Table 2: Impact of the sanity check on cascaded batch prompting. The sanity check partially recovers performance lost to input and output misalignment, although it cannot fully resolve the performance decline at larger batch sizes.
Throughput (tokens/sec) ↓
MMLU
Prompt
Batch Size
Input
Output
GPT-4.1
Single
–
4,910
47
Batch
32
18,990
965
Cascaded Batch
32
23,341
1,220
Table 3: Cascaded batch prompting introduces a modest cost overhead due to the additional symbol grounding stage.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Listing cascaded batch prompts used in the MCQA task. The placeholders “{question}”, “{options}”, and “{answer}” are replaced with the actual question, its options, and LLM answer.
Figure 6: Listing cascaded batch prompts used in the NLI task. The placeholders “{premise}”, “{hypothesis}”, and “{answer}” are replaced with the actual premise, hypothesis, and LLM answer.
Figure 7: Listing conventional batch prompts used in the MCQA and NLI tasks, respectively. Unlike cascaded batch, these prompts are tailored with symbol grounding in mind.
Large Language Models employing Chain-of-Thought reasoning achieve strong performance but suffer from excessive token consumption that inflates inference costs. Existing efficiency methods such as explicit length penalties, difficulty estimators, or multi-stage curricula either degrade reasoning quality or require complex training pipelines. We introduce Batched Contextual Reinforcement, a minimalist, single-stage training paradigm that unlocks efficient reasoning through a simple structural modification: training the model to solve N problems simultaneously within a shared context window, rewarded purely by per-instance accuracy. This formulation creates an implicit token budget that yields several key findings: (1) We identify a novel task-scaling law: as the number of concurrent problems N increases during inference, per-problem token usage decreases monotonically while accuracy degrades far more gracefully than baselines, establishing N as a controllable throughput dimension. (2) BCR challenges the traditional accuracy-efficiency trade-off by demonstrating a "free lunch" phenomenon at standard single-problem inference. Across both 1.5B and 4B model families, BCR reduces token usage by 15.8% to 62.6% while consistently maintaining or improving accuracy across five major mathematical benchmarks. (3) Qualitative analyses reveal emergent self-regulated efficiency, where models autonomously eliminate redundant metacognitive loops without explicit length supervision. (4) Crucially, we empirically demonstrate that implicit budget constraints successfully circumvent the adversarial gradients and catastrophic optimization collapse inherent to explicit length penalties, offering a highly stable, constraint-based alternative for length control. These results prove BCR practical, showing simple structural incentives unlock latent high-density reasoning in LLMs.
Bangji Yang, Hongbo Ma, Jiajun Fan +1
University of Illinois Urbana-Champaign · Tsinghua University
Large Reasoning Models (LRMs) achieve strong performance on complex tasks through extended chain-of-thought generation, but incur substantial computational costs during inference. In production settings, batched inference is essential for high throughput, yet the existing training-free adaptive pruning methods we evaluate severely degrade in this regime. Because a batch must share a single pruning mask, these methods aggregate activations across samples and then apply threshold-based selection; the threshold, calibrated offline on unaggregated activations, no longer matches the aggregated distribution, so the realized sparsity ratio drifts and accuracy on reasoning tasks collapses under batched inference. In this work, we propose a training-free adaptive pruning method designed specifically for batched inference in LRMs, built on two components. First, we replace threshold-based selection with periodic top-k selection over the aggregated importance scores, which is unaffected by the shift that aggregation induces in the activation distribution, and which runs selection once per update period rather than at every token, preserving the speedup. Second, based on the observation that important neurons re-fire periodically during long reasoning generation, we introduce an activation memory that accumulates importance across update phases so that recurring neurons are retained. Experiments on diverse reasoning benchmarks demonstrate that our method outperforms the previous state-of-the-art adaptive pruning method by 39.7 percentage points in average accuracy at batch size 4 with 50% target sparsity on DeepSeek-R1-Distill-Qwen-7B, and reaches 1.40x speedup over dense inference at 50% actual sparsity.
In common-context question answering (CCQA) tasks, multiple input questions share a common context to base their answers from. However, Large Language Models typically generate each answer using an independent prompt. While existing batching and caching techniques help improve parallelism and reduce repeated computations, the separation of questions across prompts limits the achievable speedup, as modern GPUs are underutilized due to a memory bottleneck during attention. We present Intra-Prompt Parallel Decoding (IPPD), a novel inference method that answers multiple common-context questions in parallel within a single prompt. IPPD directly addresses the bottleneck by efficiently sharing both memory and computation during the attention process, as the next token for every question is decoded in a single inference step. IPPD uses virtual position IDs and attention mask manipulation to generate the same output as standard prompting without requiring fine-tuning or any changes to the LLM architecture. Since all parallelism occurs within a prompt, IPPD is fully compatible with batched inference, even when each prompt features a different context. Our experiments show that IPPD delivers up to 7X the effective throughput as standard decoding without quality degradation, and outperforms prefix caching with PagedAttention in most settings.