The tendency for hallucination in current large language models (LLMs) negatively impacts dialogue systems. Such hallucinations produce factually incorrect responses that may mislead users and undermine system trust. Existing refinement methods for dialogue systems typically operate at the response level, overlooking the fact that a single response may contain multiple verifiable or unverifiable facts. To address this gap, we propose Fine-Refine, a fine-grained refinement framework that decomposes responses into atomic units, verifies each unit using external knowledge, assesses fluency via perplexity, and iteratively corrects granular errors. We evaluate factuality across the HybriDialogue and OpendialKG datasets in terms of factual accuracy (fact score) and coverage (Not Enough Information Proportion), and experiments show that Fine-Refine substantially improves factuality, achieving up to a 7.63-point gain in dialogue fact score, with a small trade-off in dialogue quality.
Figures & tables
Figure 1: Overall framework of the proposed fine-grained refinement. Starting from the dialogue, the original LLM-generated response contains several factually incorrect atomic units. The right part describes the assessment details. After assessing these atomic units, the LLM refines the response accordingly.
Methods
PersonaChat
Topical-Chat
Maintains Ctx.
Natural
Maintains Ctx.
Natural
r
ρ
r
ρ
r
ρ
r
ρ
Humans
0.613
0.612
0.472
0.484
0.557
0.560
0.486
0.487
Llama-3.3-70B
0.521
0.561
0.297
0.317
0.457
0.512
0.447
0.453
Qwen3-32B
0.529
0.509
0.325
0.325
0.486
0.494
0.450
0.484
Gemma3-27B
0.552
0.556
0.333
0.370
0.481
0.492
0.465
0.466
Table 1: We report correlations (r: Pearson, ρ : Spearman) on PersonaChat and Topical-Chat. Humans denotes the average of three human–human correlations, and model scores are averaged against humans. Maintains Ctx. denotes Maintains Context. Best results are in bold.
Methods
HybriDialogue
OpendialKG
Fact
NEIP ↓
Maintains Ctx.
Natural
Fact
NEIP ↓
Maintains Ctx.
Natural
G-Retriever ( He et al., 2024 )
70.29
55.77
2.824
2.788
84.79
62.65
2.895
2.927
Self-RAG ( Asai et al., 2024 )
79.74
46.39
2.162
2.238
84.06
49.53
2.462
2.418
Mistral-7B-Instruct-v0.3 ( Jiang et al., 2024 )
77.41
50.28
2.935
2.948
86.79
49.60
2.988
2.914
+ Self-Refine 3 ( Madaan et al., 2023 )
79.65
63.10
2.894
2.755
77.12
59.20
2.893
2.652
+ Fine-Refine 3 (Ours)
80.62
45.97
2.879
2.775
86.81
46.52
2.945
2.627
Table 2: Performance across HybriDialogue and OpendialKG. Fact denotes the dialogue fact score. Bold indicates the best per category. Superscripts indicate the number of refinement iterations.
Model
Iteration
HybriDialogue
OpendialKG
Fact
NEIP ↓
Maintains Ctx.
Natural
Fact
NEIP ↓
Maintains Ctx.
Natural
Qwen3-8B
Self-Refine 1
61.31
59.70
1.036
1.038
74.47
76.48
1.252
1.331
Self-Refine 2
74.70
59.38
2.854
2.871
84.68
58.99
2.925
2.902
Self-Refine 3
75.66
61.99
2.875
2.868
85.14
60.65
2.952
2.902
Qwen3-8B
Fine-Refine 1
79.39
50.08
2.842
2.881
87.36
57.62
2.970
2.961
Fine-Refine 2
80.74
49.43
2.820
2.844
87.55
57.15
2.965
2.953
Table 3: Qwen3-8B results on HybriDialogue and OpendialKG with refinement iteration counts in superscript.
Model
Setting
HybriDialogue
OpendialKG
Fact
NEIP ↓
Maintains Ctx.
Natural
Fact
NEIP ↓
Maintains Ctx.
Natural
Mistral-7B-Instruct-v0.3
Fine-Refine 1
78.99
46.78
2.901
2.876
86.31
48.22
2.969
2.735
Only Response
77.87
49.99
2.928
2.919
85.77
51.13
2.987
2.804
Only Fact
79.08
47.38
2.904
2.844
86.90
48.30
2.973
2.677
Only Fluency
76.75
48.19
2.918
2.890
86.77
49.65
2.971
2.797
Llama-3.1-8B-Instruct
Fine-Refine 1
83.21
44.94
2.754
2.605
87.53
40.85
2.891
2.614
Table 4: Ablation on a single component of feedback for HybriDialogue and OpendialKG with LLM-based evaluation.
Type
Content
Dialogue History
Speaker A: What is the decan all about? Speaker B: The decan are 36 groups of stars (small constellations) used in the Ancient Egyptian astronomy. Speaker A: How many signs are there?
Original Response
Speaker B: There are 36 signs in the decan, each representing a group of stars. ( Dialogue fact score : 0.5) Maintains Context : 2.99, natural : 2.56)
Atomic Unit 1
Speaker B says there are 36 signs in the decan. ( Fact : False, Fluency score : 0)
Atomic Unit 2
Speaker B says each sign represents a group of stars. ( Fact : True, Fluency score : 1)
Refined Response
Speaker B: There are 36 decans, each representing a group of stars. ( Dialogue fact score : 1) ( Maintains Context : 2.04, natural : 2.02)
Table 5: A case study of dialogue evaluation and refinement collected from Qwen3-8B and corresponding Fine-Refine .