$π$-CoT: Prolog-Initialized Chain-of-Thought Prompting for Multi-Hop Question-Answering

Chao Wan; Albert Gong; Mihir Mishra; Carl-Leander Henneking; Claas Beger; Kilian Q. Weinberger

cs.CLJun 25, 2025

$π$ -CoT: Prolog-Initialized Chain-of-Thought Prompting for Multi-Hop Question-Answering

Authors: Chao Wan, Albert Gong, Mihir Mishra, Carl-Leander Henneking, Claas Beger, Kilian Q. Weinberger

Organizations: Cornell University.

Abstract

Chain-of-Thought (CoT) prompting significantly enhances large language models' (LLMs) problem-solving capabilities, but still struggles with complex multi-hop questions, often falling into circular reasoning patterns or deviating from the logical path entirely. This limitation is particularly acute in retrieval-augmented generation (RAG) settings, where obtaining the right context is critical. We introduce Prolog-Initialized Chain-of-Thought ( $π$ -CoT), a novel prompting strategy that combines logic programming's structural rigor with language models' flexibility. $π$ -CoT reformulates multi-hop questions into Prolog queries decomposed as single-hop sub-queries. These are resolved sequentially, producing intermediate artifacts, with which we initialize the subsequent CoT reasoning procedure. Extensive experiments demonstrate that $π$ -CoT matches or outperforms standard RAG and in-context CoT on multi-hop question-answering benchmarks, with the largest gains on questions for which retrieval surfaces the necessary evidence.

Figures & tables

Figure 1: Overview of

\pi

-CoT. Left:

\pi

-CoT executes an LLM-generated Prolog query, using the SLICE module to resolve each sub-query

q_{t}

. Right:

\pi

-CoT uses the passages, notes, and (potentially) answer from the SLICE modules to initialize the CoT prompt for the final LLM call.

Figure 2: SLICE module for fact verification in the RAG setting. At step

t=2

, the module takes in the previous state

S_{1}

containing variable assignments, the current sub-query

q_{2}

, and the corpus

\mathcal{C}

(not shown) as inputs and outputs

S_{2}

. Only the Prolog fact (in green) corresponding to a valid statement is added to a growing knowledge base.

	HotpotQA		2WikiMultiHopQA		MuSiQue
Method	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$
Standard RAG	38.8 ± 2.2	52.6 ± 2.0	37.2 ± 2.2	40.4 ± 2.1	11.0 ± 1.4	18.1 ± 1.5
Self-Ask	19.2 ± 1.8	28.0 ± 1.8	15.8 ± 1.6	21.8 ± 1.7	5.6 ± 1.0	9.1 ± 1.2
IRCoT	40.4 ± 2.2	52.9 ± 2.0	32.4 ± 2.1	42.5 ± 2.0	17.6 ± 1.7	24.5 ± 1.8
$\pi$ -CoT (Ours)	42.0 ± 2.2	59.1 ± 1.9	49.4 ± 2.2	57.5 ± 2.1	15.2 ± 1.6	25.7 ± 1.7

Table 1: Accuracy of prompting-based methods on open-domain QA. We report mean ± 1 standard error for exact match (EM) and F1 score across 500 randomly chosen questions from each dataset. Bold indicates that no other method performs significantly better.

Method	BM25	LLM	Tokens(k)	BM25	LLM	Tokens(k)	BM25	LLM	Tokens(k)
	HotpotQA			2WikiMultiHopQA			MuSiQue
Standard RAG	1	1	3.6	1	1	3.7	1	1	2.4
Self-Ask	3.36	3.36	15	3.44	3.44	16	3.29	3.29	12
IRCoT	3.07	3.07	62	3.47	3.47	59	3.51	3.51	45
$\pi$ -CoT (Ours)	2.82	4.82	18	2.14	4.14	22	3.42	5.42	15

Table 2: Efficiency of prompting-based methods on open-domain QA. We report mean ± 1 standard error number of BM25 queries, LLM calls, and total tokens for the same questions used in Table 1 . We further break down the total tokens into prompt and completion tokens in Table 8 .

	HotpotQA		2WikiMultiHopQA		MuSiQue
Method	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$
Standard RAG	61.2	74.5	58.3	63.2	34.9	44.8
HippoRAG 2	62.6	75.3	65.5	72.0	37.6	49.5
$\pi$ -CoT (Ours)	60.3	76.8	71.1 ‡	79.6 ‡	38.5	56.2 ‡

Table 3: Comparison to the state-of-the-art OpenIE method. We report exact match (EM) and F1 on the splits from Gutiérrez et al. (2025, Tab. 2) . We use Llama-3.3-70B-Instruct for generation and NV-Embed-v2 for embedding passages with

k=5

per query.

Figure 3: Breakdown of $\pi$ -CoT predictions by Prolog execution outcome. Each pie chart shows the distribution of predictions from Table 1 across four categories based on whether Prolog returns an answer ( $S_{T}\neq\emptyset$ vs. $S_{T}=\emptyset$ ) and whether the final answer is correct (green) or incorrect (red). Lighter shades indicate cases where Prolog returns no answer ( $S_{T}=\emptyset$ ), while solid colors indicate successful Prolog execution ( $S_{T}\neq\emptyset$ ).

Dataset	Single-shot missed	Recovered	Rate
HotpotQA	120	37	30.8%
2WikiMultiHopQA	150	34	22.7%
MuSiQue	279	89	31.9%

Table 4: Questions whose gold evidence a single retrieval query misses, and how often

\pi

-CoT’s decomposed retrieval recovers at least one gold passage.

Dataset	Gold retrieved	$n$	RAG	IRCoT	$\pi$ -CoT	$\Delta$ RAG	$\Delta$ IRCoT
HotpotQA	all gold passages retrieved	226	64.2	62.8	73.0	+8.7	+10.2
HotpotQA	some retrieved	180	48.9	51.1	55.7	+6.8	+4.6
HotpotQA	none retrieved	94	31.4	32.4	32.5	+1.0	+0.1
2WikiMultiHopQA	all gold passages retrieved	186	59.6	63.9	79.8	+20.2	+15.9
2WikiMultiHopQA	some retrieved	198	36.5	38.6	56.3	+19.8	+17.7
2WikiMultiHopQA	none retrieved	116	16.2	15.7	23.7	+7.5	+8.0

Table 5: F1 by how much of the annotated gold evidence

\pi

-CoT’s retrieval surfaced. Gains are in percentage points.

	HotpotQA		2WikiMultiHopQA		MuSiQue
Ablation	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$
$\pi$ -CoT (Full)	42.0	59.1	49.4	57.5	15.2	25.7
w/o Passages	37.2	54.4	48.4	56.3	12.6	23.3
w/o Notes	34.0	49.6	47.8	55.7	12.4	22.5
w/o Prolog Answer	19.2	24.1	1.0	1.0	3.2	4.2

Table 6: Ablation analysis of information sources in the final chain-of-thought step. We use the experimental setup from Table 1 . Components are removed cumulatively from top to bottom.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	PW-S	PW-M
Question depth	20	20
Number of family trees	10	100
Max family tree size	50	50
Max family tree depth	20	20
Mode	Easy	Easy
Number of questions per template	10	10

Appendix

Table 7: Configurations for PhantomWiki dataset generation.

	HotpotQA		2WikiMultiHopQA		MuSiQue
Method	P $\times 10^{3}$	C $\times 10^{3}$	P $\times 10^{3}$	C $\times 10^{3}$	P $\times 10^{3}$	C $\times 10^{3}$
Standard RAG	3.46 (0.032)	0.159	3.63 (0.032)	0.106	2.21 (0.073)	0.136
Self-Ask	14.5 (10.5)	0.402	15.7 (10.9)	0.682	11.4 (9.06)	0.750
IRCoT	62.3 (51.5)	0.047	59.2 (44.6)	0.053	45.3 (38.1)	0.057
$\pi$ -CoT	18.0 (5.07)	0.483	21.1 (4.21)	0.543	14.4 (3.28)	0.654

Appendix

Table 8: Token cost of open-domain QA. We report mean

\pm

1 standard error number of prompt tokens P (in thousands) and completion tokens C (in thousands) for the results in Table 2 . We use the vLLM inference engine with prefix caching enabled and report the number of cached tokens in parentheses next to the prompt tokens.

	HotpotQA			2WikiMultiHopQA			MuSiQue
Method	P $\times 10^{3}$	C $\times 10^{3}$	Calls	P $\times 10^{3}$	C $\times 10^{3}$	Calls	P $\times 10^{3}$	C $\times 10^{3}$	Calls
Llama-3.3-70B-Instruct
CoT	2.68	0.110	1	2.10	0.0845	1	3.43	0.116	1
$\pi$ -CoT	12.4	0.412	4.37	11.0	0.437	4.47	16.1	0.553	4.90
DeepSeek-R1-Distill-Qwen-32B
CoT	2.77	0.305	1	2.20	0.298	1	3.57	0.520	1

Appendix

Table 9: Computational cost of in-context QA. We report the mean ± 1 standard error number of prompt tokens P (in thousands), completion tokens C (in thousands), and LLM calls for the results in Table 20 . We use the vLLM inference engine and report cached tokens (in parentheses) when prefix caching is enabled.

Method	mean	median	max	mean	median	max	mean	median	max
	HotpotQA			2WikiMultiHopQA			MuSiQue
Standard RAG	14.8	12.6	44	9.9	8.8	21	10.6	8.6	28
Self-Ask	52.7	33.3	306	448.1	56.1	5068	429.2	36.5	4854
IRCoT	112.9	86.6	320	198.2	88.2	2993	123.0	65.5	331
$\pi$ -CoT (Ours)	64.8	62.1	117	65.8	62.2	94	67.2	62.5	155

Appendix

Table 10: Wall-clock latency of open-domain QA. Seconds per question on the first 50 questions of each split, evaluated one question at a time. We report the mean, median, and maximum because the iterative baselines exhibit strongly right-skewed runtimes.

Stage	HotpotQA	2WikiMultiHopQA	MuSiQue
Query generation	23.4	24.9	22.6
SLICE execution	30.6	30.5	34.1
Final reasoning	10.8	10.4	10.5

Appendix

Table 11: Latency breakdown of

\pi

-CoT. Mean seconds per question by execution stage.

Parsing errors:
Error Type	HotpotQA	2WikiMultiHopQA	MuSiQue
Prolog query parsing error	0.8%	0%	0.8%
Execution parsing error	0.8%	0%	0.8%
Execution errors:
Intermediate predicate existence error	16.2%	19.8%	37.4%
Final predicate existence error	9.2%	0.2%	0.6%

Appendix

Table 12: Breakdown of cases in which Prolog fails to return a solution. Such cases account for 27.0%, 20.0%, and 39.6% of all questions on HotpotQA, 2WikiMultiHopQA, and MuSiQue, respectively. This is a subset of the complete

\pi

-CoT error analysis in Appendix F.2 .

Question	…Ready for It? is a Taylor Swift song from the album
	scheduled to be released on what date?
Prolog Query	album_of_song(...Ready for It?, A1),
	release_date(A1, A2)

Appendix

Table 13: Prolog query parsing error from HotpotQA. The constant “…Ready for It?” is missing double quotes in the Prolog query.

Question	which professional sports team would you not see play a home game
	in the arena where the last place Cream performed?
Prolog Query	last_performance_venue("Cream", A1),
	all_professional_sports_teams(A3),
	+home_teams(A1, A3)

Appendix

Table 14: Prolog query parsing error from MuSiQue. Negations ( + ) are not allowed by our Prolog parser.

Question	What Cantonese slang term can mean both “ghost man”
	and to refer to Westerners?
Response	“Gweilo” or “gwailou”

Appendix

Table 15: Fact extraction parsing error from HotpotQA. The double quotes in the generated answer are incorrectly escaped.

Question	What were both Hawker Hurricane and No. 1455 Flight a part of?
Prolog Query	part_of("Hawker Hurricane", A1),
	part_of("No. 1455 Flight", A2),
	(A1 == A2).
Facts	part_of("Hawker Hurricane", "Royal Air Force (RAF)").
	part_of("Hawker Hurricane", "Royal Yugoslav Air Force (VVKJ)").
	part_of("Hawker Hurricane", "Royal Canadian Air Force").

Appendix

Table 16: Execution error from HotpotQA. The generated Prolog query involves the sub-query (A1 == A2) and cannot be satisfied by the facts in the knowledge base. Although “Royal Air Force (RAF)” and “No. 11 Group RAF” are semantically related, they are not equal under the Prolog operator == . The ground-truth answer is Royal Air Force .

Question	Who wrote Turn Me On , which was performed by the person who also performed Chasing Pirates ?
Prolog Query	performer("Chasing Pirates", A1),
	performer("turn me on", A2),
	A1 = A2 -> writer("turn me on", A3).
Facts	performer("Chasing Pirates", "Norah Jones").
	performer("turn me on", "Sean Smith").
	writer("turn me on", "Greg Lake").

Appendix

Table 17: Execution error from MuSiQue. The generated Prolog query involves a string unification (A1 = A2) and cannot be satisfied by the facts in the knowledge base. The ground-truth answer is John D. Loudermilk.

Failure category	HotpotQA	2Wiki	MuSiQue	Mean F1 range
Prolog query parsing error	0.4%	0.0%	0.2%	0
Fact extraction / verification parsing error	0.6%	0.0%	0.4%	0
Predicate existence error	13.2%	18.2%	35.8%	2–14
Agent failure	0.4%	0.0%	1.2%	0–22
Final CoT overrides correct Prolog answer	1.2%	3.8%	0.2%	0–8
No gold passage retrieved	4.8%	4.6%	16.2%	7–13

Appendix

Table 18: Complete breakdown of

\pi

-CoT predictions counted as incorrect by exact match. Each prediction is assigned to the first applicable failure category under a fixed pipeline-order cascade. Percentages are over all 500 questions in each dataset (2Wiki denotes 2WikiMultiHopQA). The final column reports the range of mean F1 within each category across the three datasets.

	HotpotQA	2WikiMultiHopQA	MuSiQue
Gold passages missed	0.56	0.70	0.53
Gold passages retrieved	0.12	0.17	0.34
Ratio	4.6 $\times$	4.2 $\times$	1.6 $\times$

Appendix

Table 19: Probability of a predicate existence error conditioned on whether the annotated gold passages are retrieved by

\pi

-CoT. The ratio compares the error rate when gold passages are missed to the error rate when they are retrieved.

Llama-3.3-70B-Instruct
	HotpotQA		2WikiMultiHopQA		MuSiQue
Method	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$
CoT	62.4 ± 2.2	77.4 ± 1.6	76.4 ± 1.9	83.9 ± 1.5	51.2 ± 2.2	63.7 ± 1.9
$\pi$ -CoT	58.0 ± 2.2	77.7 ± 1.5	72.8 ± 2.0	82.5 ± 1.5	46.4 ± 2.2	63.1 ± 1.9
DeepSeek-R1-Distill-Qwen-32B
CoT	57.0 ± 2.2	74.2 ± 1.7	75.2 ± 1.9	82.9 ± 1.6	45.8 ± 2.2	57.3 ± 2.0

Appendix

Table 20: Accuracy of prompting-based methods on in-context QA. We report exact match (EM) and F1 score on HotpotQA, 2WikiMultiHopQA, MuSiQue, PhantomWiki with a corpus size of 50 articles (PW-S), and PhantomWiki with a corpus size of 500 articles (PW-M). Bold indicates that

\pi

-CoT significantly outperforms CoT (

p<0.05

) according to a paired samples

t

-test given a dataset and metric. We report the computational cost in Table 9 .

Figure 4: F1 score vs. difficulty, as measured by number of reasoning steps. We use the synthetic PW-S benchmark from Gong et al. (2025) and display mean ± 1 standard error. For each model, we evaluate CoT and

\pi

-CoT prompting.

Llama-3.3-70B-Instruct
#	HotpotQA		2WikiMultiHopQA		MuSiQue		PW-S
Samples	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$
1	63.0	79.8	76.8	84.2	49.6	64.3	55.73	75.43
2	63.2	80.3	77.0	84.1	48.4	62.7	54.00	73.95
4	63.4	80.5	76.0	83.4	50.2	63.9	53.93	74.39
8	63.2	80.2	76.0	83.3	51.0	63.8	53.13	73.83

Appendix

Table 21: Accuracy of CoT with majority voting. We report mean exact match (EM) and F1 score on a subset of the datasets from Table 20 .

Model	Temp.	Top-p	Rep. Penalty	Max Tokens
Llama-3.3-70B-Instruct	0.6	0.9	1.0	4096
DeepSeek-R1-Distill-Qwen-32B	0.6	0.95	1.0	16384

Appendix

Table 22: Sampling hyperparameters for the results in Table 21 .

Question	Query	Target	Definitions
At what age did Cieli di Toscana’s singer become blind?	singer("Cieli di Toscana", A1) age_became_blind(A1, A2)	A2	singer(<literal1>, <answer>) -> The singer of <literal1> is <answer>. ; Who is the singer of <literal1>? age_became_blind(<literal1>, <answer>) -> <literal1> became blind at the age of <answer>. ; At what age did <literal1> become blind?
Cooperative Living Organization is located in a city that is the county seat of what county in Florida?	city("Cooperative Living Organization", A1), county_seat(A1, A2)	A2	- city(<literal1>, <answer>) -> The city where <literal1> is located is <answer>. ; What city is <literal1> located in? - county_seat(<literal1>, <answer>) -> <literal1> is the county seat of <answer>. ; What county is <literal1> the county seat of?
Repertoire of Pluecido Domingo appeared in an opera written by whom which was based on a play by William Shakespeare?	opera("Repertoire of Pluecido Domingo", A1), writer(A1, A2), based_on(A1, A3)	A2	- opera(<literal1>, <answer>) -> The opera in which <literal1> appeared is <answer>. ; What opera did <literal1> appear in? - writer(<literal1>, <answer>) -> The writer of the opera <literal1> is <answer>. ; Who wrote the opera <literal1>? - based_on(<literal1>, <answer>) -> The opera <literal1> was based on a play by William Shakespeare <answer>. ; What play by William Shakespeare was the opera <literal1> based on?
In what 2 countries did both X-Large and A Bathing Ape have stores?	x_large_countries(A1), bape_countries(A2), intersection(A1, A2, A3)	A3	- x_large_countries(<answer>) -> X-Large had stores in <answer> countries. ; In what countries did X-Large have stores? - bape_countries(<answer>) -> A Bathing Ape had stores in <answer> countries. ; In what countries did A Bathing Ape have stores? - intersection(<literal1>, <literal2>, <answer>) -> The common countries between <literal1> and <literal2> are <answer>. ; What are the common countries between <literal1> and <literal2>?
Edmund Robert Harris is the principal benefactor of a museum located where ?	museum("Edmund Robert Harris", A1), location(A1, A2)	A2	- museum(<literal1>, <answer>) -> The museum that <literal1> is the principal benefactor of is <answer>. ; What museum is <literal1> the principal benefactor of? - location(<literal1>, <answer>) -> The location of <literal1> is <answer>. ; Where is <literal1> located?

Appendix

Table 23: 5 examples on generated queries and definitions on HotpotQA.

Question	Query	Target	Definitions
Which film has the director born later, Brutti Di Notte or Bir Türk’E Gönül Verdim?	director("Brutti Di Notte", A1), director("Bir Türk’E Gönül Verdim", A2), date_of_birth(A1, A3), date_of_birth(A2, A4), A3 @> A4 $\rightarrow$ A5	A5	- director(<literal>, <answer>) -> The director of <literal> is <answer>. ; Who is the director of <literal>? - date_of_birth(<literal>, <answer>) -> The date of birth of <literal> is <answer>. ; When was <literal> born?
Where did M. K. Muthu’s father die?	father("M. K. Muthu", A1), place_of_death(A1, A2)	A2	- father(<literal>, <answer>) -> The father of <literal> is <answer>. ; Who is the father of <literal>? - place_of_death(<literal>, <answer>) -> The place of death of <literal> is <answer>. ; Where did <literal> die?
What nationality is Julia Parker (Astrologer)’s husband?	husband("Julia Parker (Astrologer)", A1), nationality(A1, A2)	A2	- husband(<literal>, <answer>) -> The husband of <literal> is <answer>. ; Who is the husband of <literal>? - nationality(<literal>, <answer>) -> The nationality of <literal> is <answer>. ; What is the nationality of <literal>?
Are the directors of both films The Snake Brothers and Kooky from the same country?	director("The Snake Brothers", A1) director("Kooky", A2), country_of_citizenship(A1, A3), country_of_citizenship(A2, A4), A3 == A4 $\rightarrow$ A5	A5	- director(<literal>, <answer>) -> The director of <literal> is <answer>. ; Who is the director of <literal>? - country_of_citizenship(<literal>, <answer>) -> The country of citizenship of <literal> is <answer>. ; What is the country of citizenship of <literal>?
What is the place of birth of the director of film Kiss The Sky (Film) ?	director("Kiss The Sky (Film)", A1), place_of_birth(A1, A2)	A2	- director(<literal>, <answer>) -> The director of <literal> is <answer>. ; Who is the director of <literal>? - place_of_birth(<literal>, <answer>) -> The place of birth of <literal> is <answer>. ; Where was <literal> born?

Appendix

Table 24: 5 examples of generated queries and definitions on 2WikiMultiHopQA.

Question	Query	Target	Definitions
Who was the mother of the person who found the sacred writings that became the Book of Mormon?	founder_of_book_of _mormon(A1), mother(A1, A2)	A2	- founder_of_book_of_mormon(<answer>) -> The person who found the sacred writings that became the Book of Mormon is <answer>. ; Who found the sacred writings that became the Book of Mormon? - mother(<literal>, <answer>) -> The mother of <literal> is <answer>. ; Who is the mother of <literal>?
Who wrote Turn Me On by the Thinking About You performer?	performer("Thinking About You", A1), writer("Turn Me On", A1, A2)	A2	- performer(<literal>, <answer>) -> The performer of <literal> is <answer>. ; Who performed <literal>? - writer(<literal1>, <literal2>, <answer>) -> The writer of <literal1> by <literal2> is <answer>. ; Who wrote <literal1> by <literal2>?
What did M. King Hubbert’s employer announce it was in the process of doing in April 2010?	employer("M. King Hubbert", A1), announcement(A1, "April 2010", A2)	A2	- employer(<literal>, <answer>) -> The employer of <literal> is <answer>. ; Who is the employer of <literal>? - announcement(<literal>, <date>, <answer>) -> <literal> announced it was in the process of doing <answer> on <date>. ; What did <literal> announce it was doing on <date>?
What is the experimental satellite being forerunner to communication satellite of INSAT-4CR’s manufacturer called?	manufacturer("INSAT- 4CR", A1), experimental_satellite (A1, A2)	A2	- manufacturer(<literal>, <answer>) -> The manufacturer of <literal> is <answer>. ; Who is the manufacturer of <literal>? - experimental_satellite(<literal>, <answer>) -> The experimental satellite being the forerunner to the communication satellite of <literal> is <answer>. ; What is the experimental satellite being the forerunner to the communication satellite of <literal>?
The Socialist Autonomous Province of the city where there were mass executions of Danube Swabian population are located in where?	city_with_mass _executions("Danube Swabian", A1), socialist_autonomous _province(A1, A2), location(A2, A3)	A3	- city_with_mass_executions(<literal>, <answer>) -> The city with mass executions of <literal> population is <answer>. ; What city had mass executions of the <literal> population? - socialist_autonomous_province(<literal>, <answer>) -> The Socialist Autonomous Province of <literal> is <answer>. ; What is the Socialist Autonomous Province of <literal>? - location(<literal>, <answer>) -> <literal> is located in <answer>. ; Where is <literal> located?

Appendix

Table 25: 5 examples on generated queries and definitions on MuSiQue.

Question	Query	Target	Definitions
How many children does Shelly Reece have?	aggregate_all(count, distinct(child("Shelly Reece", A1)), A2)	A2	- child(<literal>, <answer>) -> The child of <literal> is <answer>. ; Who is the child of <literal>?
How many friends does the friend of the person whose hobby is aerospace have?	hobby(A1, "aerospace"), friend(A1, A2), aggregate_all(count, distinct(friend(A2, A3)), A4)	A4	- hobby(<answer>, <literal>) -> The hobby of <answer> is <literal>. ; Who is the person whose hobby is <literal>? - friend(<literal>, <answer>) -> The friend of <literal> is <answer>. ; Who is the friend of <literal>?
How many children does the mother of the child of the person whose date of birth is 0985-04-02 have?	dob(A1, "0985-04-02"), child(A1, A2), mother(A2, A3), aggregate_all(count, distinct(child(A3, A4)), A5)	A5	- dob(<answer>, <literal>) -> The date of birth of <answer> is <literal>. ; Who is the person whose date of birth is <literal>? - child(<literal>, <answer>) -> The child of <literal> is <answer>. ; Who is the child of <literal>? - mother(<literal>, <answer>) -> The mother of <literal> is <answer>. ; Who is the mother of <literal>?
How many mothers does the friend of the sister of the sister of the child of the parent of the person whose hobby is mineral collecting have?	hobby(A1, "mineral collecting"), parent(A1, A2), child(A2, A3), sister(A3, A4), sister(A4, A5), friend(A5, A6), aggregate_all(count, distinct(mother(A6, A7)), A8)	A8	- hobby(<answer>, <literal>) -> The hobby of <answer> is <literal>. ; Who is the person whose hobby is <literal>? - parent(<literal>, <answer>) -> The parent of <literal> is <answer>. ; Who is the parent of <literal>? - child(<literal>, <answer>) -> The child of <literal> is <answer>. ; Who is the child of <literal>? - sister(<literal>, <answer>) -> The sister of <literal> is <answer>. ; Who is the sister of <literal>? - friend(<literal>, <answer>) -> The friend of <literal> is <answer>. ; Who is the friend of <literal>? - mother(<literal>, <answer>) -> The mother of <literal> is <answer>. ; Who is the mother of <literal>?
How many children does the sibling of the brother of the brother of the sister of Jamal Song have?	sister("Jamal Song", A1), brother(A1, A2), brother(A2, A3), sibling(A3, A4), aggregate_all(count, distinct(child(A4, A5)), A6)	A6	- sister(<literal>, <answer>) -> The sister of <literal> is <answer>. ; Who is the sister of <literal>? - brother(<literal>, <answer>) -> The brother of <literal> is <answer>. ; Who is the brother of <literal>? - sibling(<literal>, <answer>) -> The sibling of <literal> is <answer>. ; Who is the sibling of <literal>? - child(<literal>, <answer>) -> The child of <literal> is <answer>. ; Who is the child of <literal>?

Appendix

Table 26: 5 examples on generated queries and definitions on PhantomWiki-S.

Question

Did John Updike and Tom Clancy both publish more than 15 bestselling novels?

Query

bestselling_novels("John Updike", A1),

bestselling_novels("Tom Clancy", A2),

(A1 @> 15 → B1 = "Yes" ; B1 = "No"),

(A2 @> 15 → B2 = "Yes" ; B2 = "No"),

Appendix

Table 27: One example on the execution trace of a question on HotpotQA.

Question

Who is Wisigard’s father-in-law?

Query

spouse("Wisigard", A1),

father(A1, A2)

Definitions

spouse(<literal> , <answer>) → The spouse of <literal> is <answer>. ; Who is the spouse of <literal>? father(<literal> , <answer>) → The father of <literal> is <answer>. ; Who is the father of <literal>?

Appendix

Table 28: One example on the execution trace of a question on 2WikiMultiHopQA.

Question

In what region of the country containing A Luoi is the birthplace of John Phan located?

Query

contains("A Luoi", A1),

birthplace("John Phan", A2),

region(A1, A2, A3)

Definitions

Appendix

Table 29: 1 example on the execution trace of a question on MuSiQue.

Dataset	Metric	$F(3,1497)$	$p$
HotpotQA	EM	52.58	$2.6\times 10^{-32}$
HotpotQA	F1	94.43	$5.7\times 10^{-56}$
2WikiMultiHopQA	EM	67.89	$3.7\times 10^{-41}$
2WikiMultiHopQA	F1	83.60	$5.2\times 10^{-50}$
MuSiQue	EM	21.13	$2.1\times 10^{-13}$
MuSiQue	F1	41.22	$1.4\times 10^{-25}$

Appendix

Table 30: Omnibus repeated-measures ANOVA over the four methods of Table 1 . All six tests reject the null hypothesis of equal means.

Comparison	$\Delta$	95% CI	$p$	$\Delta$	95% CI	$p$
	EM			F1
$\pi$ -CoT $-$ Standard RAG	+3.2	[-4.4, +10.8]	0.679	+6.6	[-0.5, +13.6]	0.078
$\pi$ -CoT $-$ IRCoT	+1.6	[-6.0, +9.2]	0.900	+6.2	[-0.8, +13.3]	0.104
$\pi$ -CoT $-$ Self-Ask	+22.8	[+15.2, +30.4]	$<$ 0.001	+31.1	[+24.1, +38.2]	$<$ 0.001
Standard RAG $-$ IRCoT	-1.6	[-9.2, +6.0]	0.900	-0.3	[-7.4, +6.7]	0.900
Standard RAG $-$ Self-Ask	+19.6	[+12.0, +27.2]	$<$ 0.001	+24.6	[+17.5, +31.6]	$<$ 0.001

Appendix

Table 31: Tukey HSD pairwise comparisons on HotpotQA, in percentage points, with simultaneous 95% confidence intervals. Bold indicates significance at

\alpha=0.05

after correction for multiple comparisons;

p

values reported as

<

0.001 are upper bounds.

Comparison	$\Delta$	95% CI	$p$	$\Delta$	95% CI	$p$
	EM			F1
$\pi$ -CoT $-$ Standard RAG	+12.2	[+4.8, +19.6]	$<$ 0.001	+17.1	[+9.9, +24.3]	$<$ 0.001
$\pi$ -CoT $-$ IRCoT	+17.0	[+9.6, +24.4]	$<$ 0.001	+14.8	[+7.6, +22.0]	$<$ 0.001
$\pi$ -CoT $-$ Self-Ask	+33.6	[+26.2, +41.0]	$<$ 0.001	+35.7	[+28.5, +42.9]	$<$ 0.001
Standard RAG $-$ IRCoT	+4.8	[-2.6, +12.2]	0.346	-2.3	[-9.5, +4.9]	0.827
Standard RAG $-$ Self-Ask	+21.4	[+14.0, +28.8]	$<$ 0.001	+18.6	[+11.4, +25.9]	$<$ 0.001

Appendix

Table 32: Tukey HSD pairwise comparisons on 2WikiMultiHopQA, in percentage points, with simultaneous 95% confidence intervals. Bold indicates significance at

\alpha=0.05

after correction for multiple comparisons;

p

values reported as

<

0.001 are upper bounds.

Comparison	$\Delta$	95% CI	$p$	$\Delta$	95% CI	$p$
	EM			F1
$\pi$ -CoT $-$ Standard RAG	+4.2	[-1.1, +9.5]	0.175	+7.7	[+2.0, +13.3]	0.003
$\pi$ -CoT $-$ IRCoT	-2.4	[-7.7, +2.9]	0.632	+1.2	[-4.4, +6.9]	0.900
$\pi$ -CoT $-$ Self-Ask	+9.6	[+4.3, +14.9]	$<$ 0.001	+16.7	[+11.0, +22.3]	$<$ 0.001
Standard RAG $-$ IRCoT	-6.6	[-11.9, -1.3]	0.008	-6.4	[-12.1, -0.8]	0.018
Standard RAG $-$ Self-Ask	+5.4	[+0.1, +10.7]	0.044	+9.0	[+3.4, +14.7]	$<$ 0.001

Appendix

Table 33: Tukey HSD pairwise comparisons on MuSiQue, in percentage points, with simultaneous 95% confidence intervals. Bold indicates significance at

\alpha=0.05

after correction for multiple comparisons;

p

values reported as

<

0.001 are upper bounds.

Stratum	$n$	RAG	Self-Ask	IRCoT	$\pi$ -CoT	$\Delta$ RAG	$\Delta$ IRCoT
all gold passages retrieved	226	64.2	31.7	62.8	73.0	+8.7	+10.2
some retrieved	180	48.9	26.9	51.1	55.7	+6.8	+4.6
none retrieved	94	31.4	21.0	32.4	32.5	+1.0	+0.1
bridge	406	49.1	26.0	49.3	56.3	+7.2	+7.0
comparison	94	67.4	36.6	68.1	71.3	+3.9	+3.2

Appendix

Table 34: F1 by stratum on HotpotQA, for all four methods. The upper block groups by retrieval coverage, the lower by question structure. Gains are in percentage points.

Stratum	$n$	RAG	Self-Ask	IRCoT	$\pi$ -CoT	$\Delta$ RAG	$\Delta$ IRCoT
all gold passages retrieved	186	59.6	31.7	63.9	79.8	+20.2	+15.9
some retrieved	198	36.5	17.9	38.6	56.3	+19.8	+17.7
none retrieved	116	16.2	12.5	15.7	23.7	+7.5	+8.0
compositional	218	30.6	21.4	44.8	46.5	+16.0	+1.8
comparison	119	49.7	24.0	50.6	76.9	+27.1	+26.3
bridge_comparison	112	44.8	11.8	18.0	50.2	+5.3	+32.1

Appendix

Table 35: F1 by stratum on 2WikiMultiHopQA, for all four methods. The upper block groups by retrieval coverage, the lower by question structure. Gains are in percentage points.

Stratum	$n$	RAG	Self-Ask	IRCoT	$\pi$ -CoT	$\Delta$ RAG	$\Delta$ IRCoT
all gold passages retrieved	66	33.7	15.8	49.2	56.4	+22.6	+7.1
some retrieved	236	19.4	10.4	23.0	28.1	+8.6	+5.0
none retrieved	198	11.2	5.2	18.0	12.7	+1.5	-5.3
2-hop	282	24.1	11.4	33.2	35.3	+11.2	+2.1
3-hop	140	9.1	4.9	15.1	14.4	+5.3	-0.7
4-hop	78	12.4	8.0	9.9	11.5	-0.9	+1.6

Appendix

Table 36: F1 by stratum on MuSiQue, for all four methods. The upper block groups by retrieval coverage, the lower by question structure. Gains are in percentage points.

Dataset	Gold retrieved	$n$	$\Delta$ RAG	Norm.	$\Delta$ IRCoT	Norm.
HotpotQA	all gold passages retrieved	226	+8.7	+24.4	+10.2	+27.3
HotpotQA	some retrieved	180	+6.8	+13.2	+4.6	+9.3
HotpotQA	none retrieved	94	+1.0	+1.5	+0.1	+0.1
2WikiMultiHopQA	all gold passages retrieved	186	+20.2	+49.9	+15.9	+44.0
2WikiMultiHopQA	some retrieved	198	+19.8	+31.2	+17.7	+28.9
2WikiMultiHopQA	none retrieved	116	+7.5	+8.9	+8.0	+9.5

Appendix

Table 37: Coverage-stratified gains in percentage points and normalized by the baseline’s remaining performance headroom. The normalized gains decrease monotonically as retrieval coverage decreases.

Multi-Hop QA

Multi-hop question answering studies how systems combine evidence across documents, facts, or modalities to answer questions requiring linked reasoning.

See all 228 papers

May 27, 2026

ConRAG: Consensus-Driven Evidence Flow for Multi-Hop Question Answering

Retrieval-augmented generation (RAG) has emerged as a promising paradigm for enhancing large language models (LLMs) on multi-hop question answering (QA), which requires reasoning over evidence from multiple documents. However, current multi-hop RAG approaches still struggle with evidence selection and information transfer, i.e., retrieving irrelevant passages or failing to pass intermediate information to subsequent reasoning steps, thereby hindering coherent multi-hop reasoning. To this end, we propose ConRAG, a consensus-driven evidence flow framework that effectively boosts LLM performance on complex multi-hop QA. In short, the core of ConRAG is to select relevant evidence via consensus-driven multi-view ranking, and leverage a slot-based mechanism to transfer intermediate information between dependent reasoning steps. Extensive experiments across three multi-hop QA benchmarks show that ConRAG surpasses the baselines by a clear margin. Encouragingly, on Gemma-4-31B-it, ConRAG outperforms the strongest baseline by 5.8 points in average score, while reducing total end-to-end runtime and token consumption by 75.9% and 67.3%, respectively.

Yikai Zhu, Kunfeng Chen, Qihuang Zhong +2

School of Computer Science, Wuhan University, Wuhan, China

Multi-Hop QA Retrieval-Augmented Generation

Ask PDF

May 13, 2026

Retrieval is Cheap, Show Me the Code: Executable Multi-Hop Reasoning for Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) has become a standard approach for knowledge-intensive question answering, but existing systems remain brittle on multi-hop questions, where solving the task requires chaining multiple retrieval and reasoning steps. Key challenges are that current methods represent reasoning through free-form natural language, where intermediate states are implicit, retrieval queries can drift from intended entities, and errors are detected by the same model that produces them making self-reflection an unreliable, ungrounded signal. We observe that multi-hop question answering is a typical form of step-by-step computation, and that this structured process aligns closely with how code-specialized language models are trained to operate. Motivated by this, we introduce \pyrag, a framework that reformulates multi-hop RAG as program synthesis and execution. Instead of free-form reasoning trajectories, \pyrag represents the reasoning process as an executable Python program over retrieval and QA tools, exposing intermediate states as variables, producing deterministic feedback through execution, and yielding an inspectable trace of the entire reasoning process. This formulation further enables compiler-grounded self-repair and execution-driven adaptive retrieval without any additional training. Experiments on five QA benchmarks (PopQA, HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle) show that \pyrag consistently outperforms strong baselines under both training-free and RL-trained settings, with especially large gains on compositional multi-hop datasets. Our code, data and models are publicly available at https://github.com/GasolSun36/PyRAG.

Jiashuo Sun, Jimeng Shi, Yixuan Xie +10

University of Illinois Urbana-Champaign · Hong Kong University of Science and Technology · Texas A&M University +1

Multi-Hop QA Retrieval-Augmented Generation Retrieval-Augmented Reasoning

Ask PDF

Aug 5, 2026

D $^2$ F-ReAG: Dynamic Decomposition and Filtering for Multi-Hop Reasoning-Augmented Generation

Large language models (LLMs) often generate inaccurate answers due to their reliance on static internal knowledge. Retrieval-augmented generation (RAG) addresses this limitation by integrating external knowledge and excelling at single-hop queries. However, it struggles with multi-hop questions that require cross-document reasoning. Existing methods, such as graph structured RAG or question decomposition, often lack dynamic decomposition and effective filtering, which leads to lower efficiency and accuracy. To overcome these limitations, we propose Dynamic Decomposition and Filtering for Multi-Hop Reasoning-Augmented Generation (D2F-ReAG), a novel paradigm that adaptively controls reasoning depth by judging the reliability of the root-level reasoning. If the root reasoning is reliable, the model directly generates the answer. Otherwise, the question is logically decomposed into sub-questions, and the verified reasoning derived from these sub-questions is used to refine the root reasoning. Experiments on three multi-hop benchmarks demonstrate the effectiveness of our method in handling complex multi-hop questions.

Jiaoyang Li, Junhao Ruan, Shengwei Tang +4

Northeastern University, Shenyang, China · Kunming University of Science and Technology, Kunming, China · NiuTrans Research, Shenyang, China

Multi-Hop QA Retrieval-Augmented Generation Adaptive RAG Question Decomposition Question Answering Multi-Hop Reasoning

Ask PDF

Abstract

Multi-Hop QA

Explore similar work

ConRAG: Consensus-Driven Evidence Flow for Multi-Hop Question Answering

Retrieval is Cheap, Show Me the Code: Executable Multi-Hop Reasoning for Retrieval-Augmented Generation

D2^22F-ReAG: Dynamic Decomposition and Filtering for Multi-Hop Reasoning-Augmented Generation

D $^2$ F-ReAG: Dynamic Decomposition and Filtering for Multi-Hop Reasoning-Augmented Generation