Intra-Prompt Parallel Decoding for Common-Context Question Answering

Theodore Glavas; Nikhita Vedula; Dushyanta Dhyani; Antonios Valkanas; Yilun Zhu; Shervin Malmasi

cs.CLSep 4, 2026

Intra-Prompt Parallel Decoding for Common-Context Question Answering

Authors: Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani, Antonios Valkanas, Yilun Zhu, Shervin Malmasi

Organizations: Amazon.com, Inc. · McGill University · Mila · Int. Lab. Learning Systems

Abstract

In common-context question answering (CCQA) tasks, multiple input questions share a common context to base their answers from. However, Large Language Models typically generate each answer using an independent prompt. While existing batching and caching techniques help improve parallelism and reduce repeated computations, the separation of questions across prompts limits the achievable speedup, as modern GPUs are underutilized due to a memory bottleneck during attention. We present Intra-Prompt Parallel Decoding (IPPD), a novel inference method that answers multiple common-context questions in parallel within a single prompt. IPPD directly addresses the bottleneck by efficiently sharing both memory and computation during the attention process, as the next token for every question is decoded in a single inference step. IPPD uses virtual position IDs and attention mask manipulation to generate the same output as standard prompting without requiring fine-tuning or any changes to the LLM architecture. Since all parallelism occurs within a prompt, IPPD is fully compatible with batched inference, even when each prompt features a different context. Our experiments show that IPPD delivers up to 7X the effective throughput as standard decoding without quality degradation, and outperforms prefix caching with PagedAttention in most settings.

Figures & tables

Figure 1: In QA tasks, multiple questions often reference the same context. Traditional batched inference separates each question into a separate prompt, and processes prompts independently to generate the answer. IPPD combines multiple questions and contexts within a single prompt and jointly decodes multiple answers in parallel. This eliminates repeated computations and reduces the memory bottleneck of GPUs during attention to generate the same answers in a fraction of the time.

Figure 2: The attention process between the queries (rows) and keys (columns) is depicted with green cells representing unmasked query-key pairs. In standard batched inference, the sequence contains a common instruction $p$ , shared context $c$ , question $x$ and partial answer $y$ (simplified as one or two tokens for illustrative purposes). The arrows indicate which token is generated using the output of that query row. IPPD concatenates multiple contexts and queries into a single stacked prompt. The attention mask ensures that each query can only attend to past tokens from the relevant context and question, identifiable by color and shade respectively. At inference step $t$ , IPPD generates the $t$ -th token of four answers through a single shared attention operation per layer in this example. Batched inference would require four independent attention calculations to achieve the same result.

			Average token length
Dataset	Number of contexts	Avg. Parallelism	Instruction	Context	Question	Answer
NarrativeQA (5s)	355	29.73	2,754.00	737.45	11.97	5.81
SQuAD 2.0	1,365	8.70	47.00	177.94	12.58	3.94
RACE	1,407	3.51	40.00	344.30	39.81	1.00
LongHealth	20	20.00	73.00	11,719.75	72.50	1.00

Table 1: Dataset statistics with the average token length of each prompt sub-section. (5s) means the instruction

p

includes 5 few-shot examples. Average parallelism is the average number of questions sharing a common context, i.e.,

\bar{M}=\frac{1}{J}\sum_{j=1}^{J}M_{j}

for a dataset with

J

contexts and

M_{j}

questions per context

c_{j}

Model	NarrativeQA (5 shot)		SQuAD 2.0		RACE		LongHealth
Model	ROUGE-L	EM%	F1	EM%	Acc.%	EM%	Acc.%	EM%
Qwen3-32B	$0.771_{+0.000}$	96.5	$0.649_{{\color[rgb]{1,0,0}-0.004}}$	96.9	$91.1_{+0.0}$	99.7	$87.0_{{\color[rgb]{1,0,0}-0.5}}$	99.5
OLMo-2-32B	$0.752_{+0.000}$	97.6	$0.714_{+0.000}$	98.1	$90.6_{+0.0}$	99.8	–	–
Phi4-14B	$0.702_{{\color[rgb]{1,0,0}-0.004}}$	91.3	$0.466_{{\color[rgb]{1,0,0}-0.005}}$	95.8	$88.0_{+0.0}$	99.9	$86.5_{{\color[rgb]{0,1,0}+0.5}}$	99.3
Qwen3-8B	$0.727_{+0.000}$	97.5	$0.598_{{\color[rgb]{1,0,0}-0.001}}$	97.5	$88.3_{{\color[rgb]{1,0,0}-0.1}}$	99.6	$83.3_{+0.0}$	100
Qwen3-4B-Instr.	$0.735_{+0.000}$	95.7	$0.659_{{\color[rgb]{0,1,0}+0.002}}$	98.1	$87.4_{+0.0}$	99.7	$83.0_{{\color[rgb]{0,1,0}+0.5}}$	99.5

Table 2: Quality metrics for standard batched inference, with the delta to IPPD outputs shown in subscript. EM (Exact Match)% measures the percentage of batched inference and IPPD answers matching each other exactly. IPPD provides the same computational path for each token but numerical instabilities result in a near 100% EM rate.

Panel of Figure 3 — Figure 3: Throughput measured in questions answered per second. Normalized throughput is relative to the standard batched inference baseline. IPPD outperforms prefix caching + PagedAttention for both short answer QA tasks (top), and for one of the two multiple-choice tasks (bottom). Darker colored shades represent batched inference (baseline), medium shades represent prefix caching + PagedAttention, and lighter shades represent IPPD.

	Short context	Long context
Single-token answer	IPPD	PC+PA
Multi-token answer	IPPD	IPPD
Long generation	Standard inf.	PC+PA

Table 3: Recommended inference method by answer length and context length, assuming several questions share each context.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method		Prefill		Decode
		First	Subsequent
PC+PA	$dim(q)$	$\|p\|+\|c_{j}\|+\|x_{j,k}\|$	$\|x_{j,k}\|$	$1$
	$dim(K,V)$	$\|p\|+\|c_{j}\|+\|x_{j,k}\|$		$\|p\|+\|c_{j}\|+\|x_{j,k}\|$ $+\|y^{j,k}_{<t}\|$
	Frequency	Once per $x_{j,k}$		Once per $y^{j,k}_{t}$
IPPD	$dim(q)$	$\|p\|+\|c_{j}\|+\sum_{k^{\prime}=1}^{M_{j}}\|x_{j,k^{\prime}}\|$	$\|c_{j}\|+\sum_{k^{\prime}=1}^{M_{j}}\|x_{j,k^{\prime}}\|$	$M_{j}^{*}$
	$dim(K,V)$	$\|p\|+\|c_{j}\|+\sum_{k^{\prime}=1}^{M_{j}}\|x_{j,k^{\prime}}\|$		$\|p\|+\|c_{j}\|$ $+\sum_{k^{\prime}=1}^{M_{j}}(\|x_{j,k^{\prime}}\|+\|y^{j,k^{\prime}}_{<t}\|)$

Appendix

Table 4: Summary of the matrix dimensions for queries

q

and keys/values

K

and

V

for a single attention head, using prefix caching + PagedAttention (PC+PA) and IPPD with one context per prompt.

dim(\cdot)

denotes the dimension that varies with sequence length, ignoring the hidden dimension. Prefill denotes the first forward pass of each prompt, with the First column representing the calculation of the prefix cache. Subsequent prefills leverage this cache to avoid recomputations. The decode column represents the autoregressive decoding stage. The Frequency row indicates how often attention must be performed using these matrix dimensions.

M_{j}

denotes the number of questions in context

c_{j}

. The ∗ indicates that the value may decrease after multiple inference steps, as some answers complete generation before others. The table shows that IPPD’s stacked prompts require larger dimensions but fewer attention calculations.

Model		NarrativeQA (5s)		SQuAD 2.0		RACE		LongHealth
Model		B	C/P	B	C/P	B	C/P	B	C/P
Batched Inf.	Qwen3-32B	8	1	30	1	5	1	1	1
	OLMo-2-32B	8	1	30	1	5	1	1	1
	Phi-4-14B	8	1	30	1	5	1	1	1
	Qwen3-8B	20	1	30	1	5	1	1	1
	Qwen3-4B-Instr.	20	1	30	1	5	1	1	1

Appendix

Table 5: Hyperparameter selection for batch size B and contexts stacked per prompt C/P for IPPD and standard batched inference. Hyperparameters are mostly constant across model sizes, except when VRAM limits require a lower batch size.

Model		NarrativeQA (5s)	LongHealth	SQuAD 2.0	RACE
Model		QPS	QPS	QPS	QPS
Qwen3-32B	Batched Inference	0.48	0.18	2.86	4.52
	PC + PA	7.06	2.82	11.90	5.84
	IPPD	10.74	1.93	15.74	11.78
OLMo-2-32B	Batched Inference	0.54	-	3.47	4.75
	PC + PA	7.15	-	12.54	6.15

Appendix

Table 6: Throughput comparison of all models on four selected CCQA datasets. We compare batched inference, prefix caching + PagedAttention (PC + PA), and our proposed IPPD. We use queries per second (QPS) as the measurement. The highest score for each setting is in bold .

Model	Dataset	Prefill steps	Prefill time	Decode steps	Decode time
Qwen3-32B	NarrativeQA (5s)	20.95 $\times$	37.59 $\times$	10.25 $\times$	8.66 $\times$
	SQuAD 2.0	7.07 $\times$	5.00 $\times$	5.79 $\times$	5.08 $\times$
	RACE	2.80 $\times$	2.58 $\times$	–	–
	LongHealth	20.00 $\times$	9.33 $\times$	–	–
OLMo-2-32B	NarrativeQA (5s)	20.95 $\times$	43.68 $\times$	10.45 $\times$	11.81 $\times$
	SQuAD 2.0	6.19 $\times$	5.86 $\times$	4.02 $\times$	4.18 $\times$

Appendix

Table 7: Prefill and decode step and time reductions from IPPD relative to batched autoregressive inference. Values are multiplicative speedups (higher is better). Dashes denote datasets with single-token outputs or configurations that exceed the model’s maximum sequence length.

Model

Questions / context

PC+PA

IPPD

PC+PA

IPPD

PC+PA

IPPD

PC+PA

IPPD

PC+PA

IPPD

PC+PA

IPPD

NarrativeQA (5s)

Qwen3-32B

5.09

\times

1.52

\times

5.72

\times

2.63

\times

6.73

\times

4.34

\times

7.93

\times

7.20

\times

10.81

\times

12.28

\times

15.76

\times

18.91

\times

OLMo-2-32B

–

Phi-4-14B

6.50

\times

1.57

\times

7.07

\times

2.74

\times

8.38

\times

5.02

\times

9.77

\times

8.84

\times

12.33

\times

15.60

\times

16.60

\times

23.98

\times

Appendix

Table 8: Questions-per-context ablation across datasets and models. Values are throughput speedups relative to batched autoregressive inference (higher is better). The better method for each model and column is in bold , and dashes denote configurations that exceed the model’s maximum sequence length.

Model	Context length
	1 $\times$		2 $\times$		4 $\times$		8 $\times$
	PC+PA	IPPD	PC+PA	IPPD	PC+PA	IPPD	PC+PA	IPPD
NarrativeQA (5s)
Qwen3-32B	15.97 $\times$	18.25 $\times$	20.50 $\times$	17.73 $\times$	20.74 $\times$	13.44 $\times$	18.74 $\times$	12.91 $\times$
OLMo-2-32B	14.39 $\times$	22.07 $\times$	–	–	–	–	–	–
Phi-4-14B	16.05 $\times$	23.16 $\times$	21.46 $\times$	18.85 $\times$	–	–	–	–

Appendix

Table 9: Context-length ablation across datasets and models. Values are throughput speedups relative to batched autoregressive inference (higher is better). The better method for each model and column is in bold , and dashes denote configurations that exceed the model’s maximum sequence length.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,101 papers

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
Sep 22, 2026Md Mostafizer Rahman, Md Faizul Ibne Amin, Md Shahajada Mia +2LLM Inference Efficiency Long-Context Language Model Inference
Lucy Family Institute for Data & Society, University of Notre Dame, IN, USA · The University of Aizu, Aizuwakamatsu, Japan · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, IN, USA
CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference
Mar 30, 2026Chuxu Song, Zhencan Peng, Jiuqi Wei +1LLM Inference Acceleration Long-Context Language Model Inference
Department of Computer Science, Rutgers University · Oceanbase, Ant Group.
PAIR: Memory-Efficient and Fast Inference with Reading--Reasoning Fusion
Oct 12, 2026Sejun Park, Haocheng Xia, Yongjoo ParkMemory-Efficient Inference Long-Context Language Model Inference
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Urbana, IL 61801, USA

Sep 22, 2026J/K move · Enter open · S save

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

Md Mostafizer Rahman, Md Faizul Ibne Amin, Md Shahajada Mia, Yutaka Watanobe+1

Lucy Family Institute for Data & Society, University of Notre Dame, IN, USA · The University of Aizu, Aizuwakamatsu, Japan · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, IN, USA

LLM Inference Efficiency Long-Context Language Model Inference Memory-Efficient Inference KV-Cache Compression Context Compression LLM Context Compression

Open paper AskPDF arXiv

Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing latency, energy consumption, and GPU memory demand as context length scales. Existing soft-compression methods either lack query-guided memory selection at inference time, train without answer-targeted supervision, or couple compression tightly to a specific decoder architecture. We propose a Context-to-Answer-Aligned Memory Compression (CMC) framework, which compresses long input contexts into compact Context Memory Embeddings (CMEs) aligned to any frozen decoder's embedding space, reducing inference costs without modifying decoder weights. CMC introduces a two-tier KV cache that combines question-guided CME selection with a local context window, and trains the compressor with answer-targeted distillation from a frozen LLM. Experiments across nine encoder-decoder combinations and four QA benchmarks show that CMC consistently outperforms the baseline, achieving up to 7.3 EM and 4.0 F1 point gains on SQuAD, while reducing inference time and energy consumption by up to 20% and peak reserved GPU memory by up to 50% at 3,000 generation tokens. Ablation studies confirm that each architectural component and training objective contributes to the performance.

Abstract

LLM Inference Acceleration

Explore similar work

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference

PAIR: Memory-Efficient and Fast Inference with Reading--Reasoning Fusion