FARCA: Fact-Aligned Reliability-Aware Credit Assignment for Reinforcement Learning with Factual Supervision

Qiming Xie; Wenjie Zheng; Xiangqing Shen; Rui Xia

cs.CLAug 25, 2026

FARCA: Fact-Aligned Reliability-Aware Credit Assignment for Reinforcement Learning with Factual Supervision

Authors: Qiming Xie, Wenjie Zheng, Xiangqing Shen, Rui Xia

Organizations: School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, China

Abstract

To reduce the hallucination risk caused by outcome-driven rewards in large language models trained through reinforcement learning with verifiable rewards, existing mitigation approaches introduce process-level factual supervision. However, due to coarse-grained aggregation of factual signals and the lack of reliability assessment for these signals, they create a mismatch between fact verification and policy updates. We term this noisy factual credit assignment and decompose it into two aspects: credit localization ambiguity and credit reliability ambiguity. To address these issues, we propose FARCA (Fact-Aligned Reliability-Aware Credit Assignment), a policy optimization framework that transforms factual supervision into localized, reliability-weighted token-level training signals. FARCA achieves fine-grained credit localization by aligning the granularity of fact verification with that of policy updates. It further introduces counterfactual evidence attribution, which uses the dependence of a factual judgment on key evidence as an empirical proxy for verification reliability to compute reliability weights. These weights modulate factual rewards and local policy advantages, reducing the influence of potentially unreliable signals on policy optimization. Experiments across different models and multiple factual reasoning benchmarks show that FARCA significantly improves model factuality while preserving general reasoning capabilities.

Figures & tables

Figure 1: (a) Percentages of rollouts and reasoning steps with mixed-correctness atomic facts, and of sign conflicts between fact labels and their shared credit. (b) Accuracy of three verifiers across quartiles constructed separately for each reliability proxy.

Figure 2: Overview of the proposed FARCA framework.

Method	Hallucination				Math
Method	SimpleQA	TruthfulQA	HalluQA	HaluEval-QA	AIME2026	AIME2025	MATH-500	GSM8K
Qwen2.5-3B-Instruct
Zero-shot	1.33	30.41	16.44	23.56	0.00	0.00	32.20	65.96
GRPO (Math only)	0.79	18.12	9.56	10.22	3.33	3.33	54.80	77.18
GRPO (Full)	2.52	42.59	22.67	25.05	3.33	3.33	53.22	75.82
KnowRL	2.42	41.98	20.67	25.57	0.00	0.00	51.80	75.59

Table 1: Performance comparison on hallucination and math benchmarks. Best results are in bold, and all tied second-best results are underlined.

Variant	Hallucination				Math
Variant	SimpleQA	TruthfulQA	HalluQA	HaluEval-QA	AIME2026	AIME2025	MATH-500	GSM8K
FARCA	3.56	45.41	25.78	26.67	6.67	10.00	63.20	84.46
w/o token provenance	3.23	44.52	23.94	25.90	6.67	6.67	61.00	82.87
w/o reliability estimation	3.04	43.88	24.51	26.28	3.33	6.67	61.40	82.26
w/o continuous factual score	3.31	44.78	25.40	26.06	6.67	10.00	62.80	83.85

Table 2: Component ablation results of FARCA.

Figure 3: Hyperparameter sensitivity of FARCA to the reliability temperature

\tau

(score change from

\tau=0.20

Figure 4: Fact matching rate and the percentage of tokens covered by matched fact spans during training.

Figure 5: Proportions of verification signals grouped by

\Delta

relative to

\mu

and by fallback status over FARCA training, with mean continuous weight

w

Figure 6: Effect of reliability-aware soft interpolation on the advantages of fact-covered tokens. (a) Directional outcome of the interpolated advantage relative to the original advantage. (b) Per-step mean and median trajectories of the normalized shift over training.

Figure 7: Normalized token-level correction of local–global credit conflicts during FARCA training.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Instruction: You are an atomic fact extractor. Given a SENTENCE, extract all atomic facts from it. An atomic fact is the smallest, self-contained, and independently verifiable factual statement. 1. Each atomic fact must be a complete and self-contained statement that can be understood independently. 2. Remove subjective opinions, speculation, reasoning steps, and discourse connectors; only keep factual content. 3. If the SENTENCE does not contain any verifiable facts (e.g., pure reasoning, mathematical derivations, or subjective judgments), return an empty list. 4. For each atomic fact, provide its corresponding source_span , which must be an exact substring of the SENTENCE. 5. Atomic facts should be ordered according to their appearance in the SENTENCE. 6. The source_span should include only the minimal, distinctive part corresponding to that fact. Avoid repeating shared subjects or prefixes across multiple facts. 7. Your task is to do this for the SENTENCE under “Your Task.” Some examples have been provided for you to learn how to do this task. EXAMPLE #1: SENTENCE: Beijing is the capital of the United States, Tiananmen is one of the most famous landmarks of Beijing. ATOMIC FACTS:

json {"atomic_facts": [{"fact": "Beijing is the capital of the United States", "source_span": "Beijing is the capital of the United States"}, {"fact": "Tiananmen is one of the most famous landmarks of Beijing", "source_span": "Tiananmen is one of the most famous landmarks of Beijing"}]}

EXAMPLE #2: SENTENCE: Therefore, we can conclude that this reasoning is correct. ATOMIC FACTS: json {"atomic_facts": []} EXAMPLE #3: SENTENCE: Tiananmen Square in Beijing is the largest city square in the world and attracts millions of visitors each year. ATOMIC FACTS:

json {"atomic_facts": [{"fact": "Tiananmen Square in Beijing is the largest city square in the world", "source_span": "Tiananmen Square in Beijing is the largest city square in the world"}, {"fact": "Tiananmen Square in Beijing attracts millions of visitors each year", "source_span": "attracts millions of visitors each year"}]}

EXAMPLE #4: SENTENCE: Beijing is the capital of the United States, its landmark building is Tiananmen. ATOMIC FACTS:

json {"atomic_facts": [{"fact": "Beijing is the capital of the United States", "source_span": "Beijing is the capital of the United States"}, {"fact": "Beijing’s landmark building is Tiananmen", "source_span": "its landmark building is Tiananmen"}]}

EXAMPLE #5: SENTENCE: Marie Curie was a famous chemist, physicist, and writer. ATOMIC FACTS:

json {"atomic_facts": [{"fact": "Marie Curie was a famous chemist", "source_span": "Marie Curie was a famous chemist"}, {"fact": "Marie Curie was a famous physicist", "source_span": "physicist"}, {"fact": "Marie Curie was a famous writer", "source_span": "writer"}]}

Your Task: SENTENCE: [SENTENCE] ATOMIC FACTS:

Appendix

Table 3: Prompt used for atomic fact extraction and source-span localization. The placeholder [SENTENCE] is replaced with each verifiable sentence in the reasoning trajectory.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 802 papers

Semifactual Credit-Augmented Policy Optimization
Sep 30, 2026Junshu Pan, Zhizhang Fu, Shulin Huang +5Prompt Sensitivity RL for Language Model Reasoning
Zhejiang University · Westlake University · Shanghai Innovation Institute +1
FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness
Apr 11, 2026Xiaoning Dong, Chengyan Wu, Yajie Wen +5LLM Alignment Factual Knowledge in Language Models
Tsinghua University, Institute for Interdisciplinary Information Sciences · Shanghai Qi Zhi Institute · South China Normal University, School of Electronic Science and Engineering +2
BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation
Jun 14, 2026Ning Li, Zixuan Guo, Yan Xu +7RL for Language Models LLM Hallucination Mitigation
1Shanghai Jiao Tong University · 2Tencent · 3The

Sep 30, 2026J/K move · Enter open · S save

Semifactual Credit-Augmented Policy Optimization

Junshu Pan, Zhizhang Fu, Shulin Huang, Yiran Ding+4

Zhejiang University · Westlake University · Shanghai Innovation Institute+1

Prompt Sensitivity RL for Language Model Reasoning Credit Assignment in RL Step-Level Credit Assignment in RL Reinforcement Learning with Verifiable Rewards Causal Interventions in Language Models+1

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet their predictions remain sensitive to task-irrelevant prompt features. We investigate this sensitivity through semifactual prompt interventions that preserve the underlying problem and its answer. Our analysis reveals substantial variation in token-level sensitivity and shows that suppressing high-drift token candidates during decoding improves reasoning accuracy without updating model weights. These findings highlight a limitation of Group Relative Policy Optimization (GRPO), which assigns the same outcome-derived advantage to every response token and may reinforce potential spurious dependence alongside useful reasoning. Motivated by this observation, we introduce Semifactual Credit-Augmented Policy Optimization (SCAPO), a causally inspired variant of GRPO that incorporates semifactual stability into token-level credit assignment. SCAPO measures token probability drift for fixed responses under semifactual interventions and uses normalized stability scores to reduce advantages for relatively unstable tokens during early training, while granting no additional credit for stability alone. On Qwen3-4B-Base and Qwen3-1.7B-Base, SCAPO improves AIME 2024-2026 accuracy over GRPO by 5.63 and 4.17 percentage points, respectively. At both model scales, SCAPO achieves the best results on most evaluated mathematics benchmarks and all evaluated out-of-distribution benchmarks among the compared methods. These results suggest that semifactual stability provides an effective training signal for improving reasoning and generalization through finer-grained credit assignment in RLVR. The code is available at https://github.com/DtYXs/SCAPO.

Abstract

LLM Reliability

Explore similar work

Semifactual Credit-Augmented Policy Optimization

FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness

BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation