cs.CLApr 2, 2026
SaveSAFE: An LLM-as-Verifier Framework for Evidence-Grounded Multi-Hop Reasoning
Organizations: Seoul National University, South Korea
Abstract
Multi-hop QA benchmarks often reward Large Language Models (LLMs) for spurious correctness, where models reach correct answers through invalid intermediate reasoning. We propose SAFE, an LLM-as-verifier framework for evidence-grounded multi-hop QA. Rather than judging only the final answer after generation, SAFE verifies reasoning during generation by checking intermediate steps against the provided passages and previous reasoning trajectory. To make this process checkable, SAFE decomposes reasoning into atomic, evidence-grounded units represented with Knowledge Graph (KG) triples. At train-time, SAFE verifies benchmark supervision under KG-grounded constraints and constructs reliable verifier training data. At inference-time, an external verifier checks each generated step, identifies invalid reasoning, and provides correction feedback before errors propagate. Across three multi-hop QA benchmarks, SAFE improves accuracy by 8.8 pp on average. These results show that evidence-grounded multi-hop QA benefits from shifting LLM-based evaluation from post-hoc answer judgment to stepwise reasoning verification.
Figures & tables
| Dataset | Split | Total | Unverifiable | Ratio |
|---|---|---|---|---|
| 2Wiki | Train | 167,454 | 7,452 | 4.45% |
| Val | 12,576 | 901 | 7.16% | |
| HotpotQA | Train | 90,447 | 2,376 | 2.63% |
| Val | 7,345 | 227 | 3.09% | |
| MuSiQue | Train | 19,902 | 2,149 | 10.80% |
| Val | 2,412 | 342 | 14.18% |
Table 1: Statistics of instances flagged as unverifiable by our KG-grounded benchmark verification pipeline.
| Model | Method | 2Wiki | HotpotQA | MuSiQue | Average | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| EM | F1 | Acc | EM | F1 | Acc | EM | F1 | Acc | EM | F1 | Acc | |||
| Without Feedback | ||||||||||||||
| Llama 3 8B | IRCoT † | 36.1 | 52.3 | - | 50.8 | 65.7 | - | 27.4 | 36.9 | - | 38.1 | 51.6 | - | - |
| TRACE † | 45.5 | 56.4 | - | 55.1 | 70.0 | - | 33.4 | 40.1 | - | 44.7 | 55.5 | - | - | |
| Gemma 3 12B | No Verification | 50.7 | 65.1 | 79.4 | 51.1 | 68.1 | 86.6 | 41.7 | 56.4 | 67.3 | 47.8 | 63.2 | 77.8 | - |
| Qwen 2.5 14B | 55.4 | 71.0 | 87.2 | 49.5 | 68.4 | 87.9 | 40.4 | 54.1 | 67.1 | 48.4 | 64.5 | 80.7 | - | |
Table 2: Results on three multi-hop QA benchmarks. Evaluation metrics are Exact Match, F1 score, and LLM-as-judge accuracy. denotes the improvement of SAFE over the best baseline (No Verification or Self-Verification) in terms of average accuracy. indicates results directly reported from the original paper ( 3 ) .
| Model | Setting | 2Wiki | Hotpot | MSQ | Avg |
|---|---|---|---|---|---|
| Gemma 12B | Diagnosis only | 86.5 | 88.0 | 71.2 | 81.9 |
| Guidance only | 90.9 | 88.8 | 76.9 | 85.5 | |
| Full | 91.8 | 90.0 | 76.7 | 86.2 | |
| Llama 8B | Diagnosis only | 83.7 | 83.6 | 66.5 | 77.9 |
| Guidance only | 89.5 | 89.5 | 77.2 | 85.4 | |
| Full | 90.7 | 90.3 | 75.3 | 85.4 |
Table 3: Ablation of verifier feedback formats. Guidance-only feedback provides the strongest signal, while the full format combines diagnosis and guidance for more stable correction.
| Drop Category | Gemma 3 12B | Llama 3.1 8B | Qwen 3 8B |
|---|---|---|---|
| No Drop | 86.2 | 85.4 | 86.3 |
| Procedural | 85.2 (-1.0) | 82.9 (-2.5) | 84.4 (-1.9) |
| Attribution | 84.4 (-1.8) | 82.8 (-2.6) | 84.5 (-1.8) |
| Logical | 85.5 (-0.7) | 84.3 (-1.1) | 85.0 (-1.3) |
Table 4: Performance degradation (average score) under different error category removals. Values in parentheses indicate absolute change ( ) from the No Drop setting.
| Model | Training Data | 2Wiki | Hotpot | MSQ | Avg |
|---|---|---|---|---|---|
| Gemma-3-12B | Verified | 91.8 | 90.0 | 76.7 | 86.2 |
| + Unverifiable | 88.5 (-3.3) | 89.7 (-0.3) | 74.9 (-1.8) | 84.4 (-1.8) | |
| Qwen2.5-14B | Verified | 90.2 | 90.1 | 76.6 | 85.6 |
| + Unverifiable | 88.0 (-2.2) | 87.9 (-2.2) | 70.9 (-5.7) | 82.3 (-3.3) | |
| Llama-3.1-8B | Verified | 90.7 | 90.3 | 75.3 | 85.4 |
| + Unverifiable | 88.2 (-2.5) | 87.5 (-2.8) | 72.3 (-3.0) | 82.7 (-2.7) |
Table 5: Effect of adding unverifiable instances to verifier training data. Training with unverifiable supervision consistently degrades accuracy, showing the importance of train-time benchmark verification.
| Model | No Verification | Self-Verification | SAFE (20B) | SAFE (120B) |
|---|---|---|---|---|
| Gemma3 12B | 78.0 | 73.0 | 84.7 | 89.7 |
| Qwen3 8B | 83.7 | 76.3 | 86.7 | 89.0 |
| Llama 3.1 8B | 66.3 | 56.7 | 78.7 | 83.0 |
Table 6: Performance with weaker teacher model’s verification signal (gpt-oss-20b). Results are averaged over 100 examples per benchmark.
| Model | Method | 2Wiki | Hotpot | MSQ | Avg | |
|---|---|---|---|---|---|---|
| Gemini 3.1 FL | No Verification | 94.0 | 90.0 | 71.0 | 85.0 | – |
| Self-Verification | 91.0 | 86.0 | 71.0 | 82.7 | – | |
| SAFE | 94.0 | 94.0 | 76.0 | 88.0 | +3.0 | |
| GPT-5.4 Mini | No Verification | 85.0 | 93.0 | 82.0 | 86.7 | – |
| Self-Verification | 92.0 | 87.0 | 74.0 | 84.3 | – | |
| SAFE | 97.0 | 93.0 | 80.0 | 90.0 | +3.3 |
Table 7: Subset evaluation on stronger reasoning models. Results are averaged over 100 examples per benchmark.
| Model | Dataset | W/O Retrieval | W/ Retrieval |
|---|---|---|---|
| Gemma 3 12B | 2Wiki | 44 | 92 |
| HotpotQA | 55 | 90 | |
| MuSiQue | 43 | 81 | |
| Qwen 3 8B | 2Wiki | 31 | 95 |
| HotpotQA | 57 | 88 | |
| MuSiQue | 42 | 77 |
Table 8: Performance of with/without retrieval under incomplete evidence setting. Results are averaged over 100 examples per benchmark.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Error Type | Definition |
| Procedural (Attribution & Logical Steps) | Overthinking † | Engaging in excessive extraction or derivation steps even after the target node ( ) has been reached. |
| Inefficiency † | Generating textual steps that do not correspond to any valid KG operation (i.e., no node or edge traversal). | |
| Off-topic † | Extracting entities or traversing paths that form a disconnected subgraph, not contributing to the target path. | |
| Redundancy † | Repeating the extraction of a node or traversal of an edge already documented in previous steps. | |
| Attribution (Attribution Steps) | Unsupported | Incorporating an entity or relationship that is completely absent from the evidence subgraph . |
| Premature Attribution | Attempting to connect a valid entity to while bypassing the necessary intermediate edge . |
Table 9: Feedback categories used by SAFE for stepwise verification. Each invalid reasoning step is assigned one mutually exclusive error type, which determines the feedback provided for correction. Procedural errors marked with are checked first because they concern the structure and progress of the reasoning path before evidence attribution and logical consistency are assessed.
Figure 1: Overview of our KG-grounded benchmark verification pipeline. The example shows an unverifiable reasoning path, where the fact is supported by a passage about North Carolina’s Piedmont rather than Virginia’s Piedmont . Our pipeline detects such evidence-grounding gaps and removes the instance from the verified set.
| Question | Gold Passages (Truncated) | Answer | Reasoning Analysis |
|---|---|---|---|
| Which film has the director who is older, Koeputkiaikuinen Ja Simon Enkelit or Indiana Jones And The Temple Of Doom? | [1] Koeputkiaikuinen… directed, written and starring Spede Pasanen… [2] Indiana Jones and the Temple of Doom… directed by Steven Spielberg… [3] Spede Pasanen (10 April 1930 – 7 Sept 2001) was a Finnish film director… [4] Steven Spielberg (born December 18, 1946) is an American filmmaker… | Koeputkiaikuinen Ja Simon Enkelit | The passages provide all necessary components for the multi-hop chain: (1) identifies Pasanen and Spielberg as respective directors, and (2) provides their birth years (1930 vs. 1946), enabling a correct age comparison. |
| What judge who was on the show “Dancing with the Stars” was born on the 31st of March, 1963? | [1] Paul Joseph Mercurio (born 31 March 1963) is an Australian actor, dancer, and TV presenter… [2] …with the exception of Paul Mercurio who wanted to focus more on the Australian version of “Dancing with the Stars”. | Paul Joseph Mercurio | The passages provides a clear multi-hop chain: (1) identifies the specific birth date for Paul Joseph Mercurio, and (2) confirms his role as a judge on the mentioned TV show, allowing for an unambiguous match. |
| Beena Sarwar is the editor of a peace initiative sponsored by a newspaper based in what city? | [1] Beena Sarwar is the Pakistan Editor of the Aman ki Asha initiative… jointly sponsored by the Jang group… [2] The Daily Jang is an Urdu newspaper based in Karachi, Pakistan. | Karachi, Pakistan | The passages provide a complete, self-contained chain: (1) connects Beena Sarwar to the Aman ki Asha initiative, (2) identifies Jang group as the sponsor, and (3) explicitly states the newspaper’s base city as Karachi. |
Table 10: False positive examples from our KG-grounded benchmark verification pipeline. These valid instances are incorrectly flagged as unverifiable, showing that misclassifications can occur on standard, well-structured reasoning paths rather than only on unusually complex queries. This suggests that the pipeline does not simply remove examples based on reasoning difficulty.
Figure 2: Representative benchmark instances flagged as unverifiable by our KG-grounded benchmark verification pipeline. The examples show an incorrect ground-truth answer that contradicts the provided evidence (top), an ambiguous question without a unique answer (middle), and insufficient context where the gold passages do not support a complete evidence-grounded reasoning path (bottom).
Figure 3: Diagram of SAFE at inference-time. The Generator produces or refines an atomic reasoning step, while the Verifier detects errors and provides feedback. This step-level feedback loop continues until the reasoning terminates.
Figure 4: Distribution of diagnostic error categories in the verifier training set. The inner ring shows broad categories, while the outer ring shows specific error types, illustrating coverage of diverse failure modes used for stepwise verification.
Figure 5: Example verifier training instance. The verifier receives the question, provided passages, previous reasoning steps, and the current step, then outputs an error type, diagnosis, and corrective guidance. For brevity, distractor passages are omitted from the original 10-passage context, and step-type tags are removed.
| Dataset | Step-Level Instances | Unique Questions |
|---|---|---|
| 2Wiki | 5,476 | 2,292 |
| HotpotQA | 3,063 | 1,330 |
| MuSiQue | 2,976 | 1,438 |
| Total | 11,515 | 5,060 |
Table 12: Statistics of the verifier training dataset. Step-level instances correspond to individual verifier inputs constructed from reasoning prefixes and current steps. Because SAFE verifies reasoning iteratively, a single question can produce multiple step-level training instances.
Figure 6: Positional distribution of error steps in the verifier training data. Errors appear across different reasoning stages, providing supervision for detecting unsupported steps at multiple points during generation.
| Component | Total | Cached | Computed | Savings |
|---|---|---|---|---|
| Generator | 43.8M | 32.9M | 10.9M | 75.2% |
| Evaluator | 100.4M | 95.4M | 5.0M | 95.0% |
| Overall | 144.2M | 128.3M | 15.9M | 89.0% |
Table 13: Inference cost measured in total and cached tokens. In SAFE , prefix KV caching eliminates 89.0% of redundant computations, and the marginal increase remains practically negligible.
| Question | Are both Missisa Lake and Lake Eden located in the same country? |
|---|---|
| Passages | [1] Eden is a CDP in Erie County, New York, United States. (Distractor) [6] Missisa Lake is a lake in Ontario, Canada. [10] Lake Eden is a small, recreational lake in Alberta, Canada. |
| Reasoning Step | Step 1: According to Passage 1, Lake Eden is located in New York. |
| Self-Verification | error_type: Correct (No Error) diagnosis: The step correctly extracts the location of Lake Eden from Passage 1. guidance: Find the location of Missisa Lake from the provided passages. |
| Ours | error_type: Contradictory diagnosis: The step misreads Passage 10 by claiming Lake Eden is in New York, whereas Passage 10 explicitly states that Lake Eden is in Alberta, Canada. This directly contradicts the retrieved passage. guidance: Correct the attribution by stating that Lake Eden is located in Alberta, Canada, based on Passage 10. |
Table 14: Qualitative comparison between self-verification and SAFE . The self-verification baseline is misled by lexical overlap with the distractor term “Eden” in Passage 1 and fails to distinguish it from the target entity “Lake Eden.” In contrast, SAFE detects the entity mismatch and provides evidence-grounded corrective guidance.
| Question: Which film has the director who died earlier, Un’Estate Ai Caraibi or Her Honor, The Governor ? Passages (Truncated): [2] Chester “Chet” Withey… died 6 October 1939. [4] Her Honor, the Governor is a 1926… film starring Pauline Frederick, directed by Chester Withey… [5] Carlo Vanzina (13 March 1951 – 8 July 2018) was an Italian film director, producer and screenwriter… [7] Un’estate ai Caraibi is a 2009 Italian comedy film directed by Carlo Vanzina… | |
|---|---|
| Self-Verification | Ours |
| Step 1: According to Passage 2, Chester “Chet” Withey died on Oct 6, 1939. error_type: Correct (No Error) diagnosis: The step correctly extracts the death date of Chester Withey. guidance: Find the death date of Carlo Vanzina. | Step 1: According to Passage 2, Chester “Chet” Withey died on Oct 6, 1939. error_type: Premature Attribution diagnosis: Missing the preceding step to identify that Withey is the director of “Her Honor, The Governor”. guidance: First, explicitly identify the director of “Her Honor, The Governor” based on Passage 4. |
| Step 2: According to Passage 5, Carlo Vanzina died on July 8, 2018. (Accepted) Step 3: Comparing the death date of Chester Withey (October 6, 1939) with the death date of Carlo Vanzina (July 8, 2018), Chester Withey died earlier. (Accepted) | Step 1 (Retry): According to Passage 4, Chester Withey directed “Her Honor, The Governor”. (Accepted) Step 2-4: Identifies Vanzina as director, then collects death dates for both directors. (Accepted) |
| Step 4: ####ANSWER: Chester Withey Analysis: Because the feedback model did not enforce the director-to-film connection in Step 1, the generator lost track of the final goal (finding the film title ) and mistakenly output the director’s name instead. | Step 5: Compares death dates of two directors. Step 6: ####ANSWER: Her Honor, The Governor Analysis: The verifier’s guidance explicitly instructed: “State which film has the director who died earlier… using format ####ANSWER: <film_title>.” . This ensured task compliance and correct entity type. |
Table 15: Detailed trace comparison between self-verification and SAFE . Self-verification loses entity consistency through premature attribution, whereas SAFE detects the missing link and provides stepwise corrective guidance to keep the reasoning path evidence-grounded and aligned with the required answer type.
| Question: In 1900, what was the population of the second largest city in the state where the Oh Yeah performer is from? Passages (Truncated): [2] Tucson is the largest city in southern Arizona, the second largest in the state after Phoenix. [6] By 1900, 7,531 people lived in the city Tucson. [9] Oh Yeah is a 1962 album by jazz musician Charles Mingus. [10] Charles Mingus was born in Nogales, Arizona. Ground Truth Answer: 7531 |
|---|
| Self-Verification Reasoning Trace (Summarized) |
| Step 4: According to Passage 2, Phoenix is the second largest city in Arizona after Tucson. |
| error_type: Off-topic diagnosis: The question asks for the population of the second largest city… While this step correctly identifies Phoenix as the second largest city in Arizona , it doesn’t address the core requirement of finding the population in 1900. guidance: Find the population of Phoenix in 1900 from the passages. |
| Step 9: According to Passage 2, Phoenix is the largest city in Arizona, but it does not state the population of Phoenix in 1900. |
| error_type: Redundancy diagnosis: This step simply reiterates information already established in Step 8. guidance: Find the population of Phoenix in 1900 from external sources. |
| Step 10: According to the United States Census Bureau , the population of Phoenix, Arizona in 1900 was 10611 . |
Table 16: Detailed trace of reasoning collapse in the self-verification baseline. The self-verifier shifts the target from the requested entity, Tucson , to an irrelevant entity, Phoenix , and then instructs the generator to use external knowledge when the required evidence is not found in the provided passages. This example illustrates how self-verification can amplify unsupported reasoning when the evaluator is not grounded in the evidence.
| Model | Verifier Training | 2Wiki | HotpotQA | MuSiQue | Average |
|---|---|---|---|---|---|
| Gemma-3-12B | Synthetic Only | 88.6 | 89.4 | 71.6 | 83.2 |
| + Refined | 91.8 (+3.2) | 90.0 (+0.6) | 76.7 (+5.1) | 86.2 (+3.0) | |
| Qwen2.5-14B | Synthetic Only | 90.2 | 89.7 | 75.7 | 85.2 |
| + Refined | 90.2 (+0.0) | 90.1 (+0.4) | 76.6 (+0.9) | 85.6 (+0.4) | |
| Llama-3.1-8B | Synthetic Only | 86.5 | 84.8 | 66.7 | 79.3 |
| + Refined | 90.7 (+4.2) | 90.3 (+5.5) | 75.3 (+8.6) | 85.4 (+6.1) |
Table 17: Ablation of targeted refinement for verifier training. “Synthetic Only” uses only diagnostic error-injected examples, while “+ Refined” adds hard cases collected through analysis-driven targeted refinement. Values in parentheses indicate absolute changes relative to Synthetic Only.
Figure 7: Cost-accuracy comparison between SAFE and the Self-Verification baseline. We evaluate the inference efficiency across different max_steps ( ) and max_retries ( ). While the Self-Verification exhibits a severe cost explosion without meaningful accuracy improvements, SAFE achieves superior accuracy with minimal computational overhead.
Figure 8: Cost-accuracy trade-off of SAFE . Annotations indicate the configuration, where is max_steps and is max_retries.
Figure 9: Distribution of error types detected by SAFE . Procedural errors, such as Redundancy and Off-topic, account for the majority of failures (64.1%), suggesting that many errors arise from invalid reasoning structure rather than missing factual knowledge.
Figure 10: Distribution of detected errors across reasoning steps. A large fraction of failures occurs within the first three steps (59.0%), supporting the use of inference-time verification to detect and correct unsupported steps before they propagate.
Figure 11: Reasoning step generation prompt.
Figure 12: Evaluation prompt.
Figure 13: Final Answer Generation prompt.
Figure 14: LLM judge prompt.
Figure 15: Plan generation prompt for 2WikiMultiHop dataset.
Figure 16: Plan generation prompt for HotpotQA dataset.
Figure 17: Plan generation prompt for MuSiQue dataset.
Figure 18: Ideal reasoning generation prompt.
Figure 19: Error Injection prompt.
Figure 20: Triple extraction prompt.
Figure 21: Triple gleaning prompt.
Figure 22: Entity resolution prompt.
Figure 23: Logical path discovery prompt.