Micro-Macro Retrieval: Reducing Long-Form Hallucination in Large Language Models

Yujie Feng; Jian Li; Zhihan Zhou; Pengfei Xu; Yujia Zhang; Xiaoyu Li; Xiaohui Zhou; Alan Zhao

cs.CLApr 10, 2026

Micro-Macro Retrieval: Reducing Long-Form Hallucination in Large Language Models

Authors: Yujie Feng, Jian Li, Zhihan Zhou, Pengfei Xu, Yujia Zhang, Xiaoyu Li, Xiaohui Zhou, Alan Zhao,

Organizations: Solar System of OVB, Tencent, China · The Hong Kong Polytechnic University, Hong Kong S.A.R. · Jilin University, China

Abstract

Large Language Models (LLMs) achieve impressive performance across many tasks but remain prone to hallucination, especially in long-form generation where redundant retrieved contexts and lengthy reasoning chains amplify factual errors. Recent studies highlight a critical phenomenon: the closer key information appears to the model outputs, the higher the factual accuracy. However, existing retrieval-augmented language models (RALMs) lack effective mechanisms to ensure this proximity - external evidence is injected into reasoning via multi-turn retrieval, but this cannot ensure key information stays close to the outputs. We propose Micro-Macro Retrieval (M2R), a novel retrieve-while-generate framework to fill this gap. At the macro level, M2R retrieves coarse-grained evidence from external sources; at the micro level, it extracts essential results from a key information repository built during reasoning and reuses them while generating answers. This design directly addresses the key-information-to-output proximity bottleneck, effectively reducing hallucination in long-form tasks. M2R is trained with a curriculum learning-based reinforcement learning strategy using customized rule-based rewards, enabling stable acquisition of retrieval and grounding skills. Extensive experiments across different benchmarks demonstrate the effectiveness of M2R, especially in lengthy-context settings.

Figures & tables

Figure 1: Overview of the M 2 R framework. During the reasoning phase, M 2 R performs macro retrieval and stores answer-aligned facts into an internal key-information repository. During the answer phase, the model invokes micro retrieval to fetch the stored facts and place them close to the generated answer tokens.

Qwen2.5-3B-Instruct
Model	HotpotQA		2Wiki		MuSiQue		Bamboogle
Model	EM	LJ	EM	LJ	EM	LJ	EM	LJ
Naive Generation	12.05	18.45	10.09	19.79	2.62	6.08	5.01	8.50
Naive RAG	21.04	37.23	13.82	23.08	4.14	10.32	14.43	20.00
Iter-RetGen	24.63	42.22	14.75	28.86	6.91	13.43	16.03	22.61
IRCoT	23.63	40.60	12.50	23.54	4.39	10.83	17.60	26.13

Table 1: Exact Match (EM, %) and LLM-as-a-Judge (LJ, %) results on multi-hop question answering benchmarks. The best results are highlighted in bold.

Figure 2: Performance on HotpotQA under multi-question inference settings. Here, HotpotQA-2Q denotes concatenating two questions into a single inference instance.

Variant	EM (%)	LJ (%)
Full M 2 R	24.12	35.44
- One-shot Grounding	23.38	34.72

Table 2: Ablation study on M 2 R.

Panel of Figure 3 — Figure 3: Reward dynamics of M 2 R during reinforcement learning.

Dataset	Think	Answer	Total	Min	Max
HotpotQA	3.7	1.4	5.1	3	6
2Wiki	4.5	1.7	6.2	3	10
MuSiQue	5.7	1.9	7.6	4	9
Bamboogle	3.5	1.3	4.8	2	6

Table 3: Average number of model invocations per query.

Dataset	Avg Invocations	Inference Time (s)
HotpotQA	5.1	$\approx$ 4.7
2Wiki	6.2	$\approx$ 5.2
MuSiQue	7.6	$\approx$ 6.8
Bamboogle	4.8	$\approx$ 4.6

Table 4: Measured inference time of M 2 R (Qwen2.5-3B + SGLang, 4

\times

A100).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Effect of answer position on model accuracy (figure taken from Liu et al. (2023) ). Accuracy declines sharply as the answer-bearing evidence appears in the middle of the context.

Prompt Template

You will be given a question and its ground truth answer list where each item can be a ground truth answer. Provided a pred_answer, you need to judge if the pred_answer correctly answers the question based on the ground truth answer list. You should first give your rationale for the judgement, and then give your judgement result (i.e., correct or incorrect). Here is the criteria for the judgement: 1. The pred_answer doesn’t need to be exactly the same as any of the ground truth answers, but should be semantically same for the question. 2. Each item in the ground truth answer list can be viewed as a ground truth answer for the question, and the pred_answer should be semantically same to at least one of them. question: {question} ground truth answers: {gt_answer} pred_answer: {pred_answer} The output should in the following json format:

json { "rationale": "your rationale for the judgement, as a text", "judgement": "your judgement result, can only be ` correct ` or ` incorrect ` " }

Your output:

Appendix

Table 5: Prompt for LLM-as-a-Judge.

Parameter	Value
Learning Rate	1e-6
Train Batch Size	256
Number of Training Epochs	2
Number of Rollout	5
Rollout Temperature	1.0
KL Loss Coefficient	0.001

Appendix

Table 6: Implementation details of M 2 R .

Llama-3.1-8B-Instruct
Method	HotpotQA	2Wiki	MuSiQue	Bamboogle
Naive Generation	18.1	24.7	4.4	11.6
Naive RAG	34.2	31.3	9.8	20.9
COFT	38.4	39.5	15.8	32.5
SURE	39.3	42.0	18.7	37.8
ReSearch	42.2	45.8	20.9	43.1

Appendix

Table 7: Exact Match results on additional model families.

Method	HotpotQA-3Q	2Wiki-3Q	MuSiQue-3Q	Bamboogle-3Q
Naive Generation	13.1	17.5	4.3	7.1
Naive RAG	20.6	16.9	5.1	15.2
COFT	24.5	27.2	12.6	22.0
SURE	28.3	31.5	11.3	25.5
ReSearch	30.2	33.9	14.2	28.0
M 2 R	32.0	35.8	17.9	30.6

Appendix

Table 8: Performance under multi-question reasoning (3Q and 5Q).

Retrieve-Top- $k$	Naive RAG	ReSearch	M 2 R
3	13.5	37.2	38.3
5 (default)	13.8	38.9	40.1
8	13.6	38.0	39.4

Appendix

Table 9: Ablations on retrieve-top-

k

and chunk size for FlashRAG.

Dataset	Input Tokens	Output Tokens (ReSearch)	Output Tokens (M 2 R)
HotpotQA	25	416	432
MuSiQue	31	483	505
2Wiki	37	440	478
Bamboogle	21	376	389

Appendix

Table 10: Token statistics during inference. M 2 R produces slightly longer outputs due to micro retrieval, but the inserted key facts are compact and answer-aligned, improving grounding and final accuracy.

Inference Time (s)	HotpotQA	2Wiki	MuSiQue	Bamboogle
ReSearch	$\approx$ 4.3	$\approx$ 4.8	$\approx$ 6.3	$\approx$ 4.2
M 2 R	$\approx$ 4.7	$\approx$ 5.2	$\approx$ 6.8	$\approx$ 4.6

Appendix

Table 11: Measured inference latency of M 2 R compared to ReSearch.

Setting	Avg Invocations	Inference Time (s)
1Q	7.6	$\approx$ 6.8
2Q	13.8	$\approx$ 14.1
3Q	19.7	$\approx$ 22.3

Appendix

Table 12: Latency scaling under multi-question reasoning.

Dataset	Avg Tokens	Min	Max
HotpotQA	62	24	108
2Wiki	73	28	121
MuSiQue	88	32	139
Bamboogle	55	18	95

Appendix

Table 13: Size of the key-information repository measured in tokens.

Training Strategy	HotpotQA	2Wiki	MuSiQue	Bamboogle
Naive RAG	21.0	13.8	4.1	14.3
Direct Optimization	13.2	8.4	3.9	10.8
Curriculum Learning (ours)	38.7	40.1	20.9	39.6

Appendix

Table 14: Comparison of training strategies, showing the importance of curriculum learning.

Question: Guest 101 wants to book any available suite room and needs to know both the total price after VIP discount and the room number with the lowest payable amount.

Answer: 180.0, Room 301

<think> First, I need to find an available suite room. Then I will calculate the price and apply the VIP discount for Guest 101. </think>

<macro_tool_call>

"name": "get_available_rooms", "room_type": "suite"

</macro_tool_call>

<macro_result> "number": 301, "type": "suite", "price": 200.0, "available": True; "number": 302, "type": "suite", "price": 220.0, "available": True </macro_result>

Appendix

Table 15: A simplified case study of M 2 R on a test instance.

Prompt Template For M 2 R

You are a helpful assistant. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. During the reasoning process, you have access to a set of tools you can use to assist with the user query, referred to as macro retrievals . These macro retrievals are enclosed within <macro_tool_call></macro_tool_call> tags. You may conduct multiple rounds of function calls, and in each round, you can call one or more functions. The results of the macro function calls will be given back to you after execution, and you can continue to call functions until you get the final answer for the user’s question. Additionally, during the reasoning process, whenever you obtain the answer to the user’s question, you must store it as a key-value pair in a key information dictionary using the <key_info_save></key_info_save> tag. • The format must strictly follow JSON, e.g.: {"target_value": "value"} • The stored key-value pairs must be directly relevant to the final answer. Finally, once you have obtained the answer and stored the key information, proceed to the answering phase. At this stage, do not call any further functions. Before writing the final answer sentence, you must first perform micro retrieval to fetch the answer from the key information dictionary. The final answer must be based only on the results of micro retrieval, rather than answering independently. Notes for micro retrieval: • Micro retrieval must be enclosed within <micro_tool_call></micro_tool_call> tags. • You may query multiple items at once or issue requests in batches. • The results of micro retrieval will be provided after execution. • If the micro retrieval fails, you must simply state that the result could not be retrieved, and must not fabricate an answer independently. Every value from the key information dictionary that appears in the final answer must be enclosed in \boxed{}.

Appendix

Table 16: Template for M 2 R.

LLM Hallucination Mitigation

Large language model hallucination mitigation develops ways to detect and reduce unsupported, fabricated, or inconsistent outputs.

See all 179 papers

Building Reliable Long-Form Generation via Hallucination Rejection Sampling
Jun 2, 2026Lin Li, Georgia Channing, Suhaas M Bhat +2Long-Form Document Generation LLM Hallucination Mitigation
Department of Computer Science, University of Oxford, Oxford, UK · Department of Engineering, University of Oxford, Oxford, UK · Oxford Digital Health Labs, Nuffield Department of Women’s and Reproductive Health, University of Oxford, Oxford, UK
Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs
Jan 5, 2026Amirali Ebrahimzadeh, Seyyed M. SaliliLong-Context Retrieval LLM Hallucination Mitigation
Department of Electrical Engineering & Computer Science, University of Michigan, Ann Arbor, MI 48109, USA · Independent Researcher, Ann Arbor, MI 48106, USA
Think Through Uncertainty: Improving Long-Form Generation Factuality via Reasoning Calibration
Apr 13, 2026Xin Liu, Lu WangLLM Hallucination Mitigation Language Model Calibration
Computer Science and Engineering, University of Michigan, Ann Arbor, MI, USA

Jun 2, 2026J/K move · Enter open · S save

Building Reliable Long-Form Generation via Hallucination Rejection Sampling

Lin Li, Georgia Channing, Suhaas M Bhat, Gabriel Davis Jones+1

Department of Computer Science, University of Oxford, Oxford, UK · Department of Engineering, University of Oxford, Oxford, UK · Oxford Digital Health Labs, Nuffield Department of Women’s and Reproductive Health, University of Oxford, Oxford, UK

Long-Form Document Generation LLM Hallucination Mitigation LLM Inference Rejection Sampling Text Generation

Open paper AskPDF arXiv

Large language models (LLMs) have achieved remarkable progress in open-ended text generation, yet they remain prone to hallucinating incorrect or unsupported content, which undermines their reliability. This issue is exacerbated in long-form generation due to hallucination snowballing, a phenomenon where early errors propagate and compound into subsequent outputs. To address this challenge, we propose a novel inference-time hallucination mitigation framework, named Segment-wise HAllucination Rejection Sampling (SHARS), which uses an arbitrary hallucination detector to identify and reject hallucinated segments during generation and resample until faithful content is produced. By retaining only confident information and building subsequent generations upon it, the framework mitigates hallucination accumulation and enhances factual consistency. To instantiate this framework, we adopt semantic uncertainty as the detector and introduce several vital modifications to address its limitations and better adapt it to long-form text. Our method enables models to self-correct hallucinations without requiring external resources such as web search or knowledge bases, while remaining compatible with them for future extensions. Empirical evaluations on standardized hallucination benchmarks demonstrate that our method substantially reduces hallucinations in long-form generation while preserving or even improving the informativeness of generation. Code is available at: https://github.com/TreeLLi/hallucination-rejection-sampling.

Abstract

LLM Hallucination Mitigation

Explore similar work

Building Reliable Long-Form Generation via Hallucination Rejection Sampling

Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

Think Through Uncertainty: Improving Long-Form Generation Factuality via Reasoning Calibration