Evaluating the reasoning abilities of large language models (LLMs) solely from final answers can obscure failures in intermediate steps, especially in multi-hop QA benchmarks without step-level annotations. To address this gap, we introduce Omanic, an open-domain 4-hop QA benchmark designed not only to measure final-answer accuracy but also to diagnose where reasoning breaks down. Omanic contains 10,296 machine-generated training examples (OmanicSynth) and 967 expert-reviewed human-annotated evaluation examples (OmanicBench), with each evaluation question decomposed into single-hop sub-questions, intermediate answers, and structured graph topologies. Experiments with proprietary and open-source LLMs show that Omanic is challenging, while step-wise analysis reveals a later-hop bottleneck, factual knowledge floor, and error propagation along reasoning chains. Fine-tuning on OmanicSynth transfers to six reasoning and mathematics benchmarks, yielding a 7.41-point average gain and validating its effectiveness as supervision for reasoning-capability transfer. We release the data at https://huggingface.co/datasets/li-lab/Omanic and the code at https://github.com/XiaojieGu/Omanic.
Figures & tables
Reasoning Graph
Multi-hop Question
Single-hop Composition
In the country of citizenship of the author of Candida, which political party was founded the same number of years before 1968 as the number of distinct 3-member committees that can be formed from a group of 7 candidates? Fine Gael [A: Fine Gael B: Labour Party C: Fianna Fáil D: Sinn Féin]
1. Who is the author of Candida? Bernard Shaw 2. What is the country of citizenship of Bernard Shaw ? Ireland 3. How many distinct 3-member committees can be formed from a group of 7 candidates? 35 4. In Ireland , which political party was founded 35 years before 1968? Fine Gael
Table 1: An example illustrating the multi-hop reasoning graph and its step-by-step question decomposition.
MCQ
Exact Match
F1-Score
Proprietary LLMs
GPT-5.4
49.22
22.85
32.22
GPT-5.4 CoT
70.84
27.09
43.58
Claude-Sonnet-4.6
55.43
20.73
37.15
Claude-Sonnet-4.6 CoT
73.11
14.81 †
32.27 †
Gemini-3.1-flash-lite
44.88
23.47
32.31
Table 2: Performance comparison of LLMs on OmanicBench. Bold marks the best result per metric. † : discuss in Appendix C.2 .
Step 1
Step 2
Step 3
Step 4
Direct
GPT-5.4
85.63
86.97
84.80
66.08
Claude-Sonnet-4.6
90.59
88.31
87.49
68.46
Gemini-3.1-Flash-Lite
89.35
87.38
86.04
66.18
CoT
GPT-5.4
93.80
95.66
92.66
79.63
Table 3: Single-hop step-level MCQ accuracy of selected models on OmanicBench. Red values indicate the lowest result for each model.
Figure 1: Comparison of accuracy across benchmarks between the vanilla model and its counterparts fine-tuned on OmanicSynth.
Figure 2: Results averaged across LLMs under Direct and CoT prompting. Left: Multi-hop accuracy by number of single-hop errors. Right: Step-wise error rates under independent and chain evaluation.
Figure 3: Answer entropy analysis for vanilla and SFT model. Left: Final answer entropy on full multi-hop questions. Right: Per-hop answer entropy under gold-context evaluation.
Appendix figures & tables48 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Overview of Omanic construction pipeline, where rectangles denote entities and circles denote single-hop questions.
Figure 5: Annotation interface for single-hop question.
Figure 6: Annotation interface for 4-hop question.
Single-hop Sub-questions
Factuality
Distractor
Fluency
Mean STD
4.620.54
4.310.72
4.470.61
4-hop Questions
Logic
Consistency
Obscurity
Mean STD
4.530.58
4.410.65
4.180.79
Fluency
Complexity
Appendix
Table 4: Human annotation scores on a 1–5 scale over 967 test instances. Each cell reports the mean score with the standard deviation in subscript (mean std ).
Reasoning Graph
Multi-hop Question
Single-hop Composition
If a political regime began in 1969 and lasted for a number of years equal to 7 times the count of Apollo moon landing missions that occurred during the presidency of Eisenhower’s vice president, and if the U.S. was one of 2 major powers that recognized this regime’s leader early on, what was the other major power? Soviet Union [A: United Kingdom B: France C: Soviet Union D: West Germany]
1. Who served as Eisenhower’s vice president? Nixon 2. How many successful Apollo moon landing missions occurred while Nixon was president of the U.S.? 6 3. If a political regime lasted for 6 times 7 years starting from 1969, whose face was most closely associated with Libya’s government during this period? Gaddafi 4. If the U.S. was one major power that recognized Gaddafi ’s government at an early date, and there were 2 major powers total that did so early on, what was the other major power besides the U.S.? Soviet Union
Great American Ball Park is the home stadium of an MLB team that has won the World Series multiple times. Multiply their total championship count by the number of players permanently banned for fixing the 1919 World Series in the Black Sox Scandal. The carrier named after the president with that ordinal number replaced which vessel at Naval Station Yokosuka, Japan, in 2015? USS George Washington [A: USS Nimitz B: USS Carl Vinson C: USS George Washington D: USS John C. Stennis]
1. In which U.S. city is Great American Ball Park located? Cincinnati 2. How many World Series championships have the Cincinnati Reds won in total? 5 3. Eight Chicago White Sox players were permanently banned from baseball for their role in fixing the 1919 World Series (the Black Sox Scandal). Multiply this number by the Cincinnati Reds’ total World Series championships ( 5 ). Which U.S. President held the ordinal number equal to this product? Ronald Reagan 4. USS Ronald Reagan replaced which aircraft carrier as the U.S. Navy’s forward-deployed vessel at Naval Station Yokosuka, Japan, in 2015? USS George Washington
Appendix
Table 5: Example multi-hop reasoning graph and its step-by-step question decomposition.
Dataset
Open Domain
# Hops
Explicit step-wise chain
Topology Annotated
Math reasoning
Expert-review
HotpotQA-SubQ Tang et al. (2021)
Yes
2
Yes
No
No
No
MuSiQue Trivedi et al. (2022)
Yes
2-4
No
No
No
Yes
FanOutQA Zhu et al. (2024)
Yes
5-6
Yes
No
No
Yes
MoreHopQA Schnitzler et al. (2024)
No
1-5
Yes
No
Yes
Yes
CofCA Wu et al. (2025)
Yes
2-4
No
No
No
No
SynthWorlds Gu et al. (2026)
No
2-4
No
No
No
No
Appendix
Table 6: Qualitative comparison between OmanicBench and representative multi-hop or reasoning benchmarks. “# Hops” reports the supported hop range; for MuSiQue and CofCA, around 80% of the questions are 2-hop. “Explicit step-wise chain” indicates whether a benchmark provides explicit step-level annotations, such as decomposed sub-questions and intermediate answers, for diagnosing reasoning failures. “Topology Annotated” indicates whether a benchmark explicitly defines and annotates reasoning topologies (e.g., chain, tree, DAG) for its questions.
Figure 7: Three reasoning graph topologies used in Omanic for organizing 4-hop questions.
Figure 8: Domain distribution of single-hop sub-questions across OmanicSynth, OmanicBench, and the overall dataset.
Figure 9: Reasoning graph topology distribution across OmanicSynth, OmanicBench, and the overall dataset.
Setting
Template
Direct multi-choice question
Select the correct option from four candidates and return only the answer letter (A/B/C/D).
Direct open-ended
Answer as concisely as possible and provide only the final answer without explanation.
CoT multi-choice question
Think step by step and end the response with “The answer is X”, where X is A, B, C, or D.
CoT open-ended
Think step by step and write the final answer on a separate line in the format “FINAL ANSWER: <answer>”.
Appendix
Table 7: Evaluation templates for direct and CoT prompting.
Parameter
Value
Cutoff length
2048
Per-device batch size
32
Gradient accumulation
2
Learning rate
1.0×10−5
Epochs
3.0
LR scheduler
Cosine
Appendix
Table 8: Hyperparameters for Qwen3-8B Full SFT.
Parameter
Value
Train batch size
512
Max prompt length
1024
Max response length
1024
Actor learning rate
1.0×10−5
PPO mini-batch size
512
PPO micro-batch size / GPU
32
Appendix
Table 9: Hyperparameters for Qwen3-8B GRPO training.
Parameter
Value
LoRA rank
8
Cutoff length
2048
Per-device batch size
32
Gradient accumulation
2
Learning rate
1.0×10−4
Epochs
3.0
Appendix
Table 10: Hyperparameters for LLaMA-3.3-70B LoRA SFT.
Direct
CoT
OpenAI
GPT-5.4
13
451
GPT-5.2
15
502
GPT-5.1
12
1,126
GPT-4o
16
1,314
Anthropic
Appendix
Table 11: Average output length under Direct and CoT prompting.
MCQ
Exact Match
F1-Score
OpenAI
GPT-5.2
38.51
18.20
29.09
GPT-5.2 CoT
65.98
34.13
46.93
GPT-5.1
39.40
18.55
26.62
GPT-5.1 CoT
71.68
39.96
51.18
GPT-4o
39.50
16.65
28.93
Appendix
Table 12: Expanded proprietary LLMs results on OmanicBench.
MCQ
Exact Match
F1-Score
Meta
LLaMA-3.3-70B
40.04
11.77
20.47
LLaMA-3.3-70B CoT
59.57
31.33
39.43
LLaMA-3-8B
25.23
4.96
8.78
LLaMA-3-8B CoT
42.50
16.55
22.78
LLaMA-3-70B
38.47
12.10
18.84
Appendix
Table 13: Expanded Open-source LLMs results on OmanicBench.
Model
Step 1
Step 2
Step 3
Step 4
GPT-4o
89.76
66.39
72.08
49.02
GPT-5.1
87.49
70.63
72.49
51.71
GPT-5.2
82.01
71.04
73.73
53.67
Claude-Sonnet-4
89.97
89.04
86.35
67.32
Claude-Opus-4
94.62
91.93
90.49
72.80
Claude-Opus-4.1
95.35
92.55
90.69
72.08
Appendix
Table 14: Step-level accuracy under direct prompting on OmanicBench.
Model
Step 1
Step 2
Step 3
Step 4
GPT-4o
92.55
94.11
89.35
75.49
GPT-5.1
94.83
94.73
91.93
78.70
GPT-5.2
90.69
94.11
90.18
77.77
Claude-Sonnet-4
93.38
96.79
92.45
80.77
Claude-Opus-4
96.07
95.66
92.66
79.94
Claude-Opus-4.1
96.79
95.97
92.86
80.97
Appendix
Table 15: Step-level accuracy under CoT prompting on OmanicBench.
Figure 10: Multi-hop accuracy by number of single-hop errors for GPT-5.2.
Figure 11: Multi-hop accuracy by number of single-hop errors for GPT-5.1.
Figure 12: Multi-hop accuracy by number of single-hop errors for GPT-4o.
Figure 13: Multi-hop accuracy by number of single-hop errors for GPT-5.4.
Figure 14: Multi-hop accuracy by number of single-hop errors for Claude-Sonnet-4.5.
Figure 15: Multi-hop accuracy by number of single-hop errors for Claude-Opus-4.5.
Figure 16: Multi-hop accuracy by number of single-hop errors for Claude-Opus-4.1.
Figure 17: Multi-hop accuracy by number of single-hop errors for Claude-Sonnet-4.
Figure 18: Multi-hop accuracy by number of single-hop errors for Claude-Opus-4.
Figure 19: Multi-hop accuracy by number of single-hop errors for Claude-Sonnet-4.6.
Figure 20: Multi-hop accuracy by number of single-hop errors for Gemini-3-Flash-Preview.
Figure 21: Multi-hop accuracy by number of single-hop errors for Gemini-2.5-Flash.
Figure 22: Multi-hop accuracy by number of single-hop errors for Gemini-2.5-Flash-Lite.
Figure 23: Multi-hop accuracy by number of single-hop errors for Gemini-3.1-flash-lite.
Figure 24: Multi-hop accuracy by number of single-hop errors for Qwen3-Max.
Figure 25: Step-wise error rates under independent and chain evaluation for GPT-5.2.
Figure 26: Step-wise error rates under independent and chain evaluation for GPT-5.1.
Figure 27: Step-wise error rates under independent and chain evaluation for GPT-4o.
Figure 28: Step-wise error rates under independent and chain evaluation for GPT-5.4.
Figure 29: Step-wise error rates under independent and chain evaluation for Claude-Sonnet-4.5.
Figure 30: Step-wise error rates under independent and chain evaluation for Claude-Opus-4.5.
Figure 31: Step-wise error rates under independent and chain evaluation for Claude-Opus-4.1.
Figure 32: Step-wise error rates under independent and chain evaluation for Claude-Sonnet-4.
Figure 33: Step-wise error rates under independent and chain evaluation for Claude-Opus-4.
Figure 34: Step-wise error rates under independent and chain evaluation for Claude-Sonnet-4.6.
Figure 35: Step-wise error rates under independent and chain evaluation for Gemini-3-Flash-Preview.
Figure 36: Step-wise error rates under independent and chain evaluation for Gemini-2.5-Flash.
Figure 37: Step-wise error rates under independent and chain evaluation for Gemini-2.5-Flash-Lite.
Figure 38: Step-wise error rates under independent and chain evaluation for Gemini-3.1-flash-lite.
Figure 39: Step-wise error rates under independent and chain evaluation for Qwen3-Max.
Multi-hop QA benchmarks often reward Large Language Models (LLMs) for spurious correctness, where models reach correct answers through invalid intermediate reasoning. We propose SAFE, an LLM-as-verifier framework for evidence-grounded multi-hop QA. Rather than judging only the final answer after generation, SAFE verifies reasoning during generation by checking intermediate steps against the provided passages and previous reasoning trajectory. To make this process checkable, SAFE decomposes reasoning into atomic, evidence-grounded units represented with Knowledge Graph (KG) triples. At train-time, SAFE verifies benchmark supervision under KG-grounded constraints and constructs reliable verifier training data. At inference-time, an external verifier checks each generated step, identifies invalid reasoning, and provides correction feedback before errors propagate. Across three multi-hop QA benchmarks, SAFE improves accuracy by 8.8 pp on average. These results show that evidence-grounded multi-hop QA benefits from shifting LLM-based evaluation from post-hoc answer judgment to stepwise reasoning verification.
Reinforcement Learning (RL) has been shown to significantly boost reasoning capabilities of large language models (LLMs) in math, coding, and multi-hop reasoning tasks. However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by LLM-based verifiers. All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-prone and costly, and LLM-based verifiers are inaccurate and slow. In this work, we investigate a cheaper alternative: RL fine-tuning on rule-generated synthetic data for multi-hop reasoning tasks. We discover that LLMs fine-tuned on synthetic data perform significantly better on popular real-world question-answering benchmarks, despite the synthetic data containing only fictional knowledge. On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to compose knowledge -- a fundamental and generalizable reasoning skill. Our work highlights rule-generated synthetic reasoning data as a free and scalable resource to improve LLM reasoning capabilities.
Anmol Kabra, Yilun Yin, Albert Gong +6
Cornell University · University of Cambridge · Stanford University
Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.