Atomic Consistency Preference Optimization for Long-Form Question Answering

Jingfeng Chen; Raghuveer Thirukovalluru; Junlin Wang; Kaiwei Luo; Bhuwan Dhingra

cs.CLMay 14, 2025

Atomic Consistency Preference Optimization for Long-Form Question Answering

Authors: Jingfeng Chen, Raghuveer Thirukovalluru, Junlin Wang, Kaiwei Luo, Bhuwan Dhingra

Abstract

Large Language Models (LLMs) often produce factoid hallucinations - plausible yet incorrect answers. A common mitigation strategy is model alignment, which improves factual accuracy by training on curated (factual, non-factual) pairs. However, this approach often relies on a stronger model (e.g., GPT-4) or an external knowledge base to assess factual correctness that may not always be accessible. Addressing this, we propose Atomic Consistency Preference Optimization (ACPO), a self-supervised preference-tuning method that enhances factual accuracy without external supervision. ACPO leverages atomic consistency signals (i.e., the agreement of individual facts across multiple stochastic responses) to identify high- and low-quality data pairs for model alignment. Despite being fully self-supervised, ACPO outperforms the strong supervised alignment baseline by 1.95 points averaged across Phi-3 and Llama3 on the LongFact and BioGen datasets, demonstrating its effectiveness in improving factual reliability without relying on external models or knowledge bases.

Figures & tables

Figure 1: ACPO data curation pipeline. Steps 1–5 (Top): Generate stochastic responses for a question, Split and extract atomic facts; Cluster the atomic facts; Identify consistent and non-consistent clusters; Score responses based on cluster consistency. Step 6-7 (Bottom): Highest and lowest scoring curated as preference pairs; DPO alignment.

	Llama-3-8B-Instruct				Phi-3-mini-4k-instruct
	LongFact		BioGen		LongFact		BioGen
Method	Score	#Claim	Score	#Claim	Score	#Claim	Score	#Claim
RawModel	79.8	121.2	55.9	61	78	90.2	41.7	88.4
FactAlign	83.3	119.9	57.1	56.7	81.2	113.8	47.1	100.9
ACPO	82.1	143.8	58	68	84.6	70.7	51.8	67.1

Table 1: Factscore accuracy for ACPO FactAlign and other baselines on LongFact, Bios datasets. FactAlign and ACPO were trained on the train set of LongFact. #Claim is the average number of claims produced by the model.

Training Data	(#Consistent: #Non-Consistent )
Phi-3-mini-4k-instruct	(1:3)
Llama-3-8B-Instruct	(1:1.8)

Table 2: Ratio of consistent to non-consistent data across training sets.

	LLama-3-8b-Instruct						Phi-3-mini-4k-instruct
	Length		LongFact		BioGen		Length		LongFact		BioGen
Method	P	NP	Score	#C	Score	#C	P	NP	Score	#C	Score	#C
ACPO	478	457	82.1	143.8	58	68	307	327	84.6	70.7	51.8	67.1
ACPO (5,5)	466	449	79.6	150.6	57.3	83.2	287	295	85.1	72.5	49.4	64.3
ACPO (5,4+1)	466	461	80.2	140.2	58.2	73.9	287	285	85.4	75.6	50.5	65.4
ACPO (5,3+2)	466	468	81.4	127.8	59.7	68	287	279	84.9	75.2	50.6	67.1

Table 3: Using length as an additional criterion to balance preferred and non-preferred data leads improves #Claims, thereby increasing recall. Results are better for Llama3 on BioGen and for Phi-3 on LongFact.

Method	F1	#Claim
RawModel	75.88	100.85
FactAlign	80.32	103.33
ACPO (Ours)	83.84	120.17

Table 4: Longfact F1 score with Llama-3-8b-Instruct

	Phi-3-mini-4k-instruct
			BioGen
Model	Inf.	$\text{Score}(R_{i})$	Score	#Claims
RawModel	Direct	-5.61	42.9	85.3
RawModel	ASC	4.10	45.7	84.5
ACPO (It. 1)	Direct	1.27	51.8	67.1
ACPO (It. 1)	ASC	7.97	54	70.7

Table 5: Performance with ASC at inference time. ACPO with direct inference outperforms RawModel , showing the importance of ASC in training. Applying ASC at inference further boosts performance, with Iteration 1 performing best. Later iterations add no gains but still beat RawModel while generating more #Claims, highlighting ASC’s value for alignment data.

Panel of Table 6 — Table 6: Comparison of computational time across different methods for preference pair construction. Our ACPO employs a clustering-based approach, whereas Sentence+GPT-4 treats sentences as atomic facts and verifies them using GPT-4, and GPT-4 (SKT) both generates and verifies atomic facts through GPT-4.

	BioGen
Method	Score	#Claim
ACPO	51.8	67.1
ACPO (5,5)	49.4	64.3
ACPO w/o NC Penalty	46.9	99.7
Longest Preferred	41.3	97.6
Shortest Preferred	42.8	10.6

Table 7: Stronger preference signals in ACPO perform better than weaker ones in ACPO(5,5) . Not penalizing non-consistent atomic facts yields worse alignment. Favoring short or long responses harms factual precision, highlighting the value of ACPO ’s alignment strategy.

Method	Score	#Claims
ACPO ( $\Theta=2$ )	51.8	67.1
ACPO ( $\Theta=3$ )	50.4	48.7

Table 8: ACPO performance under different

\Theta

values.

Temp.	0	0.3	0.6	0.9	1.2	1.5	ACPO
Score	42.9	43.2	41.7	38.4	36.3	33.3	51.8
#Claim	85.3	85.2	88.4	93	99.9	111	67.1

Table 9: Self-supervised alignment in ACPO (temp=0) outperforms temperature scaling—a simple post-hoc calibration method—across multiple temperatures.

Question \| "What is the De Beers diamond monopoly controversy?
Preferred	Non-Preferred
The De Beers diamond monopoly controversy centers … by the De Beers company, … had a near-complete monopoly over the global diamond market. Several key details and examples include: 1. De Beers’ monopoly began in the late 19th century when Cecil Rhodes, a British businessman and mining magnate, founded the company in South Africa in 1888. 2. By the early 20th century, De Beers … and controlled over 90% of the world’s rough diamond production and trade. 3. The company’s … stockpiling diamonds and controlling … to maintain high prices and ensure a stable supply. … 7. In 2005, the Antitrust Division of the United States Department of Justice filed an antitrust lawsuit … price-fixing and maintaining a monopoly in the diamond market. …	The De Beers diamond monopoly controversy revolves around the company’s control over the global diamond market in the … early 21st centuries . De Beers, a British company founded in 1888 by Cecil Rhodes, came to dominate the diamond industry by controlling diamond production and distribution. … De Beers’ monopoly was challenged by a South African-based company, Cut Stone, led by Ernest Oppenheimer, grandson of …, Cecil Rhodes . Cut Stone accused De Beers of fixing diamond prices and controlling the market. The controversy intensified when the South African government, …, announced plans to nationalize the diamond industry in 1981. … In 1998, a class action lawsuit was filed against De Beers in the United States by the American Antitrust Institute (AAI) ….

Table 10: Preferred and Non-Preferred responses curated by ACPO . Green highlights indicate phrases verified to be correct, while red highlights mark incorrect phrases according to Wikipedia. In this example, terms like Cutstone, the nationalization claim, AAI are all hallucinated.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Llama-3-8B-Instruct				Phi-3-mini-4k-instruct
	LongFact		BioGen		LongFact		BioGen
Method	Score	#Claim	Score	#Claim	Score	#Claim	Score	#Claim
RawModel	79.5	121.6	55.3	61.1	79.8	91.1	42.9	85.3
FactAlign	83.1	118.2	57.6	58.1	82.6	112.2	48.4	97.3
ACPO	82.1	143.8	58	68	84.6	70.7	51.8	67.1

Appendix

Table 11: ACPO vs other methods at temperature=0 during inference

Datasets	Model	(Train Test)	Response Number	ACS	ARC
LongFact	LLaMA-3	(2097,233)	30	290	23
LongFact	Phi-3.5-mini	(2097,233)	30	245	14

Appendix

Table 12: Summary of the training dataset statistics. Response Number denotes the number of initial responses generated per question. ACS (Average Cluster Size) represents the average number of clusters formed per question based on atomic fact clustering. ARC (Average Response Coverage) indicates the average number of clusters that each response contributes to

LLM Hallucination Mitigation

Large language model hallucination mitigation develops ways to detect and reduce unsupported, fabricated, or inconsistent outputs.

See all 178 papers

FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness
Apr 11, 2026Xiaoning Dong, Chengyan Wu, Yajie Wen +5LLM Alignment Factual Knowledge in Language Models
Tsinghua University, Institute for Interdisciplinary Information Sciences · Shanghai Qi Zhi Institute · South China Normal University, School of Electronic Science and Engineering +2
Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparison
Oct 13, 2026Pratuat Amatya, Vinay Setty, V Venktesh +1LLM Alignment Factual Knowledge in Language Models
Factiverse AS · University of Stavanger · Stockholm University +1
Differentiable Conformal Training for LLM Reasoning Factuality
Apr 22, 2026Nathan Hittesdorf, Marco Salzetta, Lu ChengLLM Hallucination Mitigation Conformal Prediction
Department of Computer Science, University of Illinois at Chicago, Chicago, United States · Department of Physics, University of Illinois at Urbana-Champaign, Urbana, United States

Apr 11, 2026J/K move · Enter open · S save

FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness

Xiaoning Dong, Chengyan Wu, Yajie Wen, Yu Chen+4

Tsinghua University, Institute for Interdisciplinary Information Sciences · Shanghai Qi Zhi Institute · South China Normal University, School of Electronic Science and Engineering+2

LLM Alignment Factual Knowledge in Language Models

Open paper AskPDF arXiv

Large Language Models (LLMs) can generate factually inaccurate content even if they have corresponding knowledge, which critically undermines their reliability. Existing approaches attempt to mitigate this by incorporating uncertainty in QA prompt during training, but these numerical scores lack the semantic richness for LLM to properly understand its internal states of trustworthiness and honestness, leading to insufficient factuality alignment. We introduce FAITH (Factuality Alignment through Integrating Trustworthiness and Honestness), a post-training framework for factuality alignment that integrates natural-language uncertainty signals with external knowledge. Specifically, we augment training datasets by computing confidence scores and semantic entropy from LLM outputs and mapping them into a knowledge state quadrant that describes the model's internal knowledge possession (trustworthiness) and answering behaviors (honestness) in natural language. Based on this enhanced data, we design a reward function that considers both correctness and uncertainty signals, and fine-tune the LLM using the Proximal Policy Optimization (PPO) algorithm. To further mitigate weakly grounded responses, we design a retrieval-augmented module that retrieves relevant external passages, improving the consistency between internal and external knowledge representations. Extensive experiments on four knowledge-intensive benchmarks demonstrate that FAITH enhances the factual accuracy and truthfulness of LLMs.

Abstract

LLM Hallucination Mitigation

Explore similar work

FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness

Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparison

Differentiable Conformal Training for LLM Reasoning Factuality