Large Language Models (LLMs) are prone to generating hallucinated content, which compromises their reliability in knowledge-intensive tasks. To address this challenge without sacrificing creativity, we propose HARPO, a reinforcement learning framework designed to jointly optimize faithfulness and creativity. HARPO incorporates a Hallucination-Aware Generative Reward Model (HA-GRM), trained via verifiable feedback, to assess both faithfulness and writing quality. A Selective Activation Mechanism (SAM) activates writing rewards only for outputs judged hallucination-free by HA-GRM, while a data curriculum progressively shifts training from creative writing to hallucination-centric tasks. On RAGTruth, our Qwen3-4B-based HA-GRM achieves a response-level F1 score of 78.08%, compared with 66.37% for the supervised fine-tuning baseline. Experiments on Qwen2.5 and Qwen3 models from 1.7B to 8B parameters show improvements in both faithful generation and writing quality. On Qwen3-4B, HARPO reduces the HA-GRM-judged hallucination rate on MultiHopRAG from 3.29% to 1.02%, while increasing the Arena-Hard-v2.0 creative-writing score from 16.95% to 27.54%.
Figures & tables
Figure 1: The HARPO Framework. Left: The Data Curriculum Strategy smoothly shifts the training distribution from creative to faithful tasks via cosine annealing. Right: The Selective Activation Mechanism activates writing rewards only for responses judged hallucination-free by HA-GRM.
Model Name
RAGTruth (Summ)
RAGTruth (QA)
RAGTruth (D2T)
Overall
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Open-Source Models (8B - 671B)
Qwen3-8B
50.14
87.25
63.69
36.81
88.12
51.93
79.76
91.19
85.09
60.50
89.82
72.30
Qwen3-32B
42.76
91.18
58.22
33.64
91.88
49.25
80.12
94.28
86.62
56.54
93.20
70.39
Qwen3-235B-A22B-Instruct
40.25
96.55
56.81
28.65
95.62
44.09
66.55
99.48
79.75
48.99
98.19
65.37
Qwen3-235B-A22B-Thinking
37.94
97.01
54.55
28.95
95.00
44.38
71.84
98.44
83.07
50.03
97.55
66.14
Table 1: Performance (Precision, Recall and F1) of hallucination evaluation on the test split of RAGTruth. We separate Qwen3-4B based models, larger open-source models, and proprietary models into different blocks.
Model Name
Summ
QA
D2T
Overall
Qwen3-4B
41.43
34.00
38.08
37.33
HA-GRM ( SFT )
45.94
61.26
30.13
44.58
HA-GRM ( Response Only )
48.53
48.74
39.31
44.43
HA-GRM ( Full )
52.10
53.63
47.26
50.38
Table 2: Span-level hallucination detection performance (span-F1) on the RAGTruth test set. Bold indicates the best score in each column.
Model Name
RAGTruth
Auto-J
Average
Qwen3-4B
74.58
57.22
65.90
w/ Hallucination
78.41
57.31
67.86
w/ Hallucination and Writing
78.08
59.91
69.00
Table 3: Impact of multi-objective training on hallucination detection (RAGTruth) and general preference evaluation (Auto-J) test sets.
Dataset
Domain
# of Samples
Training data
RAGTruth
Summarization
2,217
HaluEval
Question Answering
1,452
Arena-Human-Preference-140k
Creative Writing
3,000
Testing data
HHEM
Summarization
831
Table 4: Statistics of the training and testing datasets utilized in the HARPO experiments.
Model Name
Hallucination Control ↓
Creative Writing ↑
General ↑
HHEM
MultiHopRAG
Arena-Hard-v2.0 (Creative Writing)
Arena-Hard-v2.0 (Hard Prompts)
Qwen2.5 series
Qwen2.5-3B
11.07
4.69
2.33
2.07
w/ HARPO (Ours)
2.41
2.86
2.66
2.30
Qwen2.5-7B
5.05
1.88
4.68
4.63
w/ HARPO (Ours)
1.56
0.86
6.78
4.93
Table 5: Comparison of HARPO against base models and baselines. ( ↓ ) indicates lower is better, ( ↑ ) indicates higher is better. HARPO lowers HA-GRM-judged hallucination rates and improves writing scores relative to the corresponding base models.