Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models

Xiaojie Gu; Sherry T. Tong; Aosong Feng; Sophia Simeng Han; Jinghui Lu; Yingjian Chen; Yusuke Iwasawa; Yutaka Matsuo

cs.CLMar 17, 2026

Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models

Authors: Xiaojie Gu, Sherry T. Tong, Aosong Feng, Sophia Simeng Han, Jinghui Lu, Yingjian Chen, Yusuke Iwasawa, Yutaka Matsuo, +3 more

Organizations: The University of Tokyo · Yale University · Stanford University · Xiaomi EV · Smartor LLC · Soongsil University

Abstract

Evaluating the reasoning abilities of large language models (LLMs) solely from final answers can obscure failures in intermediate steps, especially in multi-hop QA benchmarks without step-level annotations. To address this gap, we introduce Omanic, an open-domain 4-hop QA benchmark designed not only to measure final-answer accuracy but also to diagnose where reasoning breaks down. Omanic contains 10,296 machine-generated training examples (OmanicSynth) and 967 expert-reviewed human-annotated evaluation examples (OmanicBench), with each evaluation question decomposed into single-hop sub-questions, intermediate answers, and structured graph topologies. Experiments with proprietary and open-source LLMs show that Omanic is challenging, while step-wise analysis reveals a later-hop bottleneck, factual knowledge floor, and error propagation along reasoning chains. Fine-tuning on OmanicSynth transfers to six reasoning and mathematics benchmarks, yielding a 7.41-point average gain and validating its effectiveness as supervision for reasoning-capability transfer. We release the data at https://huggingface.co/datasets/li-lab/Omanic and the code at https://github.com/XiaojieGu/Omanic.

Figures & tables

Reasoning Graph	Multi-hop Question	Single-hop Composition
	In the country of citizenship of the author of Candida, which political party was founded the same number of years before 1968 as the number of distinct 3-member committees that can be formed from a group of 7 candidates? Fine Gael [A: Fine Gael B: Labour Party C: Fianna Fáil D: Sinn Féin]	1. Who is the author of Candida? Bernard Shaw 2. What is the country of citizenship of Bernard Shaw ? Ireland 3. How many distinct 3-member committees can be formed from a group of 7 candidates? 35 4. In Ireland , which political party was founded 35 years before 1968? Fine Gael

Table 1: An example illustrating the multi-hop reasoning graph and its step-by-step question decomposition.

	MCQ	Exact Match	F1-Score
Proprietary LLMs
GPT-5.4	49.22	22.85	32.22
GPT-5.4 ${}_{\texttt{CoT}}$	70.84	27.09	43.58
Claude-Sonnet-4.6	55.43	20.73	37.15
Claude-Sonnet-4.6 ${}_{\texttt{CoT}}$	73.11	14.81 †	32.27 †
Gemini-3.1-flash-lite	44.88	23.47	32.31

Table 2: Performance comparison of LLMs on OmanicBench. Bold marks the best result per metric. † : discuss in Appendix C.2 .

	Step 1	Step 2	Step 3	Step 4
Direct
GPT-5.4	85.63	86.97	84.80	66.08
Claude-Sonnet-4.6	90.59	88.31	87.49	68.46
Gemini-3.1-Flash-Lite	89.35	87.38	86.04	66.18
CoT
GPT-5.4	93.80	95.66	92.66	79.63

Table 3: Single-hop step-level MCQ accuracy of selected models on OmanicBench. Red values indicate the lowest result for each model.

Figure 1: Comparison of accuracy across benchmarks between the vanilla model and its counterparts fine-tuned on OmanicSynth.

Figure 2: Results averaged across LLMs under Direct and CoT prompting. Left: Multi-hop accuracy by number of single-hop errors. Right: Step-wise error rates under independent and chain evaluation.

Figure 3: Answer entropy analysis for vanilla and SFT model. Left: Final answer entropy on full multi-hop questions. Right: Per-hop answer entropy under gold-context evaluation.

Appendix figures & tables48 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Overview of Omanic construction pipeline, where rectangles denote entities and circles denote single-hop questions.

Figure 5: Annotation interface for single-hop question.

Figure 6: Annotation interface for 4-hop question.

Single-hop Sub-questions
	Factuality	Distractor	Fluency
Mean ${}_{\text{STD}}$	$4.62_{0.54}$	$4.31_{0.72}$	$4.47_{0.61}$
4-hop Questions
	Logic	Consistency	Obscurity
Mean ${}_{\text{STD}}$	$4.53_{0.58}$	$4.41_{0.65}$	$4.18_{0.79}$
	Fluency	Complexity

Appendix

Table 4: Human annotation scores on a 1–5 scale over 967 test instances. Each cell reports the mean score with the standard deviation in subscript (mean

{}_{\text{std}}

Reasoning Graph	Multi-hop Question	Single-hop Composition
	If a political regime began in 1969 and lasted for a number of years equal to 7 times the count of Apollo moon landing missions that occurred during the presidency of Eisenhower’s vice president, and if the U.S. was one of 2 major powers that recognized this regime’s leader early on, what was the other major power? Soviet Union [A: United Kingdom B: France C: Soviet Union D: West Germany]	1. Who served as Eisenhower’s vice president? Nixon 2. How many successful Apollo moon landing missions occurred while Nixon was president of the U.S.? 6 3. If a political regime lasted for 6 times 7 years starting from 1969, whose face was most closely associated with Libya’s government during this period? Gaddafi 4. If the U.S. was one major power that recognized Gaddafi ’s government at an early date, and there were 2 major powers total that did so early on, what was the other major power besides the U.S.? Soviet Union
	Great American Ball Park is the home stadium of an MLB team that has won the World Series multiple times. Multiply their total championship count by the number of players permanently banned for fixing the 1919 World Series in the Black Sox Scandal. The carrier named after the president with that ordinal number replaced which vessel at Naval Station Yokosuka, Japan, in 2015? USS George Washington [A: USS Nimitz B: USS Carl Vinson C: USS George Washington D: USS John C. Stennis]	1. In which U.S. city is Great American Ball Park located? Cincinnati 2. How many World Series championships have the Cincinnati Reds won in total? 5 3. Eight Chicago White Sox players were permanently banned from baseball for their role in fixing the 1919 World Series (the Black Sox Scandal). Multiply this number by the Cincinnati Reds’ total World Series championships ( 5 ). Which U.S. President held the ordinal number equal to this product? Ronald Reagan 4. USS Ronald Reagan replaced which aircraft carrier as the U.S. Navy’s forward-deployed vessel at Naval Station Yokosuka, Japan, in 2015? USS George Washington

Appendix

Table 5: Example multi-hop reasoning graph and its step-by-step question decomposition.

Dataset	Open Domain	# Hops	Explicit step-wise chain	Topology Annotated	Math reasoning	Expert-review
HotpotQA-SubQ Tang et al. (2021)	Yes	2	Yes	No	No	No
MuSiQue Trivedi et al. (2022)	Yes	2-4	No	No	No	Yes
FanOutQA Zhu et al. (2024)	Yes	5-6	Yes	No	No	Yes
MoreHopQA Schnitzler et al. (2024)	No	1-5	Yes	No	Yes	Yes
CofCA Wu et al. (2025)	Yes	2-4	No	No	No	No
SynthWorlds Gu et al. (2026)	No	2-4	No	No	No	No

Appendix

Table 6: Qualitative comparison between OmanicBench and representative multi-hop or reasoning benchmarks. “# Hops” reports the supported hop range; for MuSiQue and CofCA, around 80% of the questions are 2-hop. “Explicit step-wise chain” indicates whether a benchmark provides explicit step-level annotations, such as decomposed sub-questions and intermediate answers, for diagnosing reasoning failures. “Topology Annotated” indicates whether a benchmark explicitly defines and annotates reasoning topologies (e.g., chain, tree, DAG) for its questions.

Panel of Figure 7 — Figure 7: Three reasoning graph topologies used in Omanic for organizing 4-hop questions.

Figure 8: Domain distribution of single-hop sub-questions across OmanicSynth, OmanicBench, and the overall dataset.

Figure 9: Reasoning graph topology distribution across OmanicSynth, OmanicBench, and the overall dataset.

Setting	Template
Direct multi-choice question	Select the correct option from four candidates and return only the answer letter (A/B/C/D).
Direct open-ended	Answer as concisely as possible and provide only the final answer without explanation.
CoT multi-choice question	Think step by step and end the response with “The answer is X”, where X is A, B, C, or D.
CoT open-ended	Think step by step and write the final answer on a separate line in the format “FINAL ANSWER: <answer>”.

Appendix

Table 7: Evaluation templates for direct and CoT prompting.

Parameter	Value
Cutoff length	2048
Per-device batch size	32
Gradient accumulation	2
Learning rate	$1.0\times 10^{-5}$
Epochs	3.0
LR scheduler	Cosine

Appendix

Table 8: Hyperparameters for Qwen3-8B Full SFT.

Parameter	Value
Train batch size	512
Max prompt length	1024
Max response length	1024
Actor learning rate	$1.0\times 10^{-5}$
PPO mini-batch size	512
PPO micro-batch size / GPU	32

Appendix

Table 9: Hyperparameters for Qwen3-8B GRPO training.

Parameter	Value
LoRA rank	8
Cutoff length	2048
Per-device batch size	32
Gradient accumulation	2
Learning rate	$1.0\times 10^{-4}$
Epochs	3.0

Appendix

Table 10: Hyperparameters for LLaMA-3.3-70B LoRA SFT.

	Direct	CoT
OpenAI
GPT-5.4	13	451
GPT-5.2	15	502
GPT-5.1	12	1,126
GPT-4o	16	1,314
Anthropic

Appendix

Table 11: Average output length under Direct and CoT prompting.

	MCQ	Exact Match	F1-Score
OpenAI
GPT-5.2	38.51	18.20	29.09
GPT-5.2 ${}_{\texttt{CoT}}$	65.98	34.13	46.93
GPT-5.1	39.40	18.55	26.62
GPT-5.1 ${}_{\texttt{CoT}}$	71.68	39.96	51.18
GPT-4o	39.50	16.65	28.93

Appendix

Table 12: Expanded proprietary LLMs results on OmanicBench.

	MCQ	Exact Match	F1-Score
Meta
LLaMA-3.3-70B	40.04	11.77	20.47
LLaMA-3.3-70B ${}_{\texttt{CoT}}$	59.57	31.33	39.43
LLaMA-3-8B	25.23	4.96	8.78
LLaMA-3-8B ${}_{\texttt{CoT}}$	42.50	16.55	22.78
LLaMA-3-70B	38.47	12.10	18.84

Appendix

Table 13: Expanded Open-source LLMs results on OmanicBench.

Model	Step 1	Step 2	Step 3	Step 4
GPT-4o	89.76	66.39	72.08	49.02
GPT-5.1	87.49	70.63	72.49	51.71
GPT-5.2	82.01	71.04	73.73	53.67
Claude-Sonnet-4	89.97	89.04	86.35	67.32
Claude-Opus-4	94.62	91.93	90.49	72.80
Claude-Opus-4.1	95.35	92.55	90.69	72.08

Appendix

Table 14: Step-level accuracy under direct prompting on OmanicBench.

Model	Step 1	Step 2	Step 3	Step 4
GPT-4o	92.55	94.11	89.35	75.49
GPT-5.1	94.83	94.73	91.93	78.70
GPT-5.2	90.69	94.11	90.18	77.77
Claude-Sonnet-4	93.38	96.79	92.45	80.77
Claude-Opus-4	96.07	95.66	92.66	79.94
Claude-Opus-4.1	96.79	95.97	92.86	80.97

Appendix

Table 15: Step-level accuracy under CoT prompting on OmanicBench.

Figure 10: Multi-hop accuracy by number of single-hop errors for GPT-5.2.

Figure 11: Multi-hop accuracy by number of single-hop errors for GPT-5.1.

Figure 12: Multi-hop accuracy by number of single-hop errors for GPT-4o.

Figure 13: Multi-hop accuracy by number of single-hop errors for GPT-5.4.

Figure 14: Multi-hop accuracy by number of single-hop errors for Claude-Sonnet-4.5.

Figure 15: Multi-hop accuracy by number of single-hop errors for Claude-Opus-4.5.

Figure 16: Multi-hop accuracy by number of single-hop errors for Claude-Opus-4.1.

Figure 17: Multi-hop accuracy by number of single-hop errors for Claude-Sonnet-4.

Figure 18: Multi-hop accuracy by number of single-hop errors for Claude-Opus-4.

Figure 19: Multi-hop accuracy by number of single-hop errors for Claude-Sonnet-4.6.

Figure 20: Multi-hop accuracy by number of single-hop errors for Gemini-3-Flash-Preview.

Figure 21: Multi-hop accuracy by number of single-hop errors for Gemini-2.5-Flash.

Figure 22: Multi-hop accuracy by number of single-hop errors for Gemini-2.5-Flash-Lite.

Figure 23: Multi-hop accuracy by number of single-hop errors for Gemini-3.1-flash-lite.

Figure 24: Multi-hop accuracy by number of single-hop errors for Qwen3-Max.

Figure 25: Step-wise error rates under independent and chain evaluation for GPT-5.2.

Figure 26: Step-wise error rates under independent and chain evaluation for GPT-5.1.

Figure 27: Step-wise error rates under independent and chain evaluation for GPT-4o.

Figure 28: Step-wise error rates under independent and chain evaluation for GPT-5.4.

Figure 29: Step-wise error rates under independent and chain evaluation for Claude-Sonnet-4.5.

Figure 30: Step-wise error rates under independent and chain evaluation for Claude-Opus-4.5.

Figure 31: Step-wise error rates under independent and chain evaluation for Claude-Opus-4.1.

Figure 32: Step-wise error rates under independent and chain evaluation for Claude-Sonnet-4.

Figure 33: Step-wise error rates under independent and chain evaluation for Claude-Opus-4.

Figure 34: Step-wise error rates under independent and chain evaluation for Claude-Sonnet-4.6.

Figure 35: Step-wise error rates under independent and chain evaluation for Gemini-3-Flash-Preview.

Figure 36: Step-wise error rates under independent and chain evaluation for Gemini-2.5-Flash.

Figure 37: Step-wise error rates under independent and chain evaluation for Gemini-2.5-Flash-Lite.

Figure 38: Step-wise error rates under independent and chain evaluation for Gemini-3.1-flash-lite.

Figure 39: Step-wise error rates under independent and chain evaluation for Qwen3-Max.

Explore similar work

SAFE: An LLM-as-Verifier Framework for Evidence-Grounded Multi-Hop Reasoning
Apr 2, 2026Daeyong Kwon, Soyoung Yoon, Seung-won HwangMulti-Hop QA LLM Answer Verification
Seoul National University, South Korea
Learning from Synthetic Data Improves Multi-hop Reasoning
Mar 2, 2026Anmol Kabra, Yilun Yin, Albert Gong +6RL for Language Model Reasoning Multi-Hop Reasoning
Cornell University · University of Cambridge · Stanford University
BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models
Mar 16, 2026Yuzhe TangLLM Evaluation Implicit Reasoning in Language Models
Georgia Institute of Technology

Apr 2, 2026J/K move · Enter open · S save

SAFE: An LLM-as-Verifier Framework for Evidence-Grounded Multi-Hop Reasoning

Daeyong Kwon, Soyoung Yoon, Seung-won Hwang

Seoul National University, South Korea

Multi-Hop QA LLM Answer Verification LLM Grounding

Open paper AskPDF arXiv

Multi-hop QA benchmarks often reward Large Language Models (LLMs) for spurious correctness, where models reach correct answers through invalid intermediate reasoning. We propose SAFE, an LLM-as-verifier framework for evidence-grounded multi-hop QA. Rather than judging only the final answer after generation, SAFE verifies reasoning during generation by checking intermediate steps against the provided passages and previous reasoning trajectory. To make this process checkable, SAFE decomposes reasoning into atomic, evidence-grounded units represented with Knowledge Graph (KG) triples. At train-time, SAFE verifies benchmark supervision under KG-grounded constraints and constructs reliable verifier training data. At inference-time, an external verifier checks each generated step, identifies invalid reasoning, and provides correction feedback before errors propagate. Across three multi-hop QA benchmarks, SAFE improves accuracy by 8.8 pp on average. These results show that evidence-grounded multi-hop QA benefits from shifting LLM-based evaluation from post-hoc answer judgment to stepwise reasoning verification.

Abstract

Explore similar work

SAFE: An LLM-as-Verifier Framework for Evidence-Grounded Multi-Hop Reasoning

Learning from Synthetic Data Improves Multi-hop Reasoning

BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models