SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection

Jiawei Dong; Zilong Bai; Pengtian zhu; Peng Zhou

cs.CLOct 23, 2026

SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection

Authors: Jiawei Dong, Zilong Bai, Pengtian zhu, Peng Zhou

Organizations: Patsnap Pte. Ltd., Suzhou, China

Abstract

Large Language Models (LLMs) have gained widespread adoption in various natural language processing tasks, but suffer from hallucination issues where they generate unfaithful or inconsistent content. While Reinforcement Learning with Verifiable Rewards (RLVR) has shown promising applications in reasoning tasks, its application to span-level hallucination detection is challenged by localization ambiguity, reward sparsity on hard examples, and uncontrolled length inflation. To address these issues, we introduce Socratic Guided Policy Optimization (SOPO), a multi-turn RL framework for robust hallucination detection. The key innovation of SOPO is a Socratic feedback mechanism, where a teacher model guides the reasoning trajectories during rollout by providing indirect hints inferred from the ground truth. This teacher-guided approach enables the model to generate more effective reasoning trajectories for hard examples, thus alleviating reward sparsity. Furthermore, we incorporate Turn-based Advantage Scaling to penalize verbosity bias, encouraging the model to develop efficient and autonomous reasoning capabilities. Empirical results on RAGTruth demonstrate that SOPO (4B) achieves state-of-the-art performance, surpassing leading proprietary models (e.g., GPT-5) and standard RL baselines, while obtaining higher reported mean scores than GRPO on four out-of-domain benchmarks, with mixed evidence against CAPO. We release our training and evaluation code alongside the 1.7B and 4B SOPO models at https://github.com/patsnap/SOPO.

Figures & tables

Table 4

Method	Training	Span-F1	Case-F1
SOPO	Interactive RL	$63.34\pm 0.22$	$84.65\pm 0.21$
SOPO-FLOP	Interactive RL	$62.96\pm 0.24$	$83.72\pm 0.32$
80B-CoT-Distill	Teacher-trace SFT	$50.09\pm 0.77$	$79.05\pm 0.39$

Table 6: Overall RAGTruth F1 (%, mean

\pm

standard deviation over 10 selected full-set, single-turn inference runs per checkpoint). All students share the cold-start initialization.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	SFT (Cold-start)	RL
Learning Rate	$7\times 10^{-6}$	$1\times 10^{-6}$
Batch Size	16	64
Epochs	3	4
Warmup Epoch	1	1
LR Scheduler	Cosine	Constant
Optimizer	Adam	Adam

Appendix

Table 7: Hyperparameter of SFT and RL training.

Statistic	GRPO	SOPO	Difference
Generation Length (multi-turn sum)	2700.1	2529.4	-6.3%
Dynamic Sampling Rate ( $\rho$ )	41.30%	48.02%	+6.7%
Turns Per Trajectory	1	1.57	-
Teacher Gen Length Per Feedback	-	172.8	-
Student Gen Length Per Turn	2700.1	1534.4	-43.2%

Appendix

Table 8: Trajectory statistics.

Training Phase	Component	GRPO	SOPO	Difference
Forward ( $2ND$ )	Student	9.37	8.62	-8.0%
Forward ( $2ND$ )	Teacher	0.00	3.46	-
Backward ( $6ND$ )	Student	11.61	12.87	+10.6%
Total		20.98	24.95	+18.9%

Appendix

Table 9: FLOPs calculation (ExaFLOPs).

Metric	GRPO	SOPO	Difference
Training Time (Hours)	17.88	15.82	-11.5%
GPU Hours (NVIDIA-A800)	286.08	316.40	+10.6%

Appendix

Table 10: Wall-clock time and GPU hours.

SOPO checkpoint	Baseline	SOPO mean	Base mean	$\boldsymbol{\Delta}$ F1	95% CI for $\Delta$	One-sided $p$
Overall span-F1
SOPO	CAPO	63.34	59.75	+3.59	[1.52, 5.75]	0.0005
SOPO	GRPO	63.34	59.49	+3.85	[1.33, 6.57]	0.0008
SOPO-FLOP	CAPO	62.96	59.75	+3.21	[0.98, 5.38]	0.0031
SOPO-FLOP	GRPO	62.96	59.49	+3.47	[0.93, 6.00]	0.0039
SOPO-GPUh	CAPO	62.74	59.75	+2.99	[0.95, 5.07]	0.0016

Appendix

Table 12: Repeated-inference RAGTruth results (%, 10 runs per method). Nominal paired-bootstrap 95% CIs and one-sided

p

-values use 10,000 replicates (seed 42), drawing one run per pool and resampling matched test examples. Pools were selected by span-F1 (including baseline runs from multiple evaluation campaigns); case-F1 uses the same runs. Estimates are conditional on these pools and do not measure training-seed variability.

Checkpoint	PsiloQA span-F1	Math-Shepherd span-F1	AggreFact mean BAcc	HaluEval accuracy
SOPO	$73.90\pm 0.36$	$89.02\pm 0.49$	$75.93\pm 0.15$	$69.77\pm 0.03$
SOPO-FLOP	$74.17\pm 0.20$	$87.59\pm 0.62$	$75.23\pm 0.10$	$69.23\pm 0.06$
CAPO	$72.67\pm 0.18$	$85.89\pm 0.60$	$75.29\pm 0.11$	$69.19\pm 0.19$
GRPO	$72.51\pm 0.17$	$85.68\pm 0.61$	$72.55\pm 0.90$	$67.52\pm 0.35$

Appendix

Table 13: OOD primary-metric means

\pm

standard deviations (%) over 10 inference runs per checkpoint or baseline group. These are repeated inference evaluations, not independent training seeds.

Benchmark	SOPO checkpoint	Baseline	$\boldsymbol{\Delta}$	95% CI for $\Delta$	One-sided $p$
PsiloQA	SOPO	CAPO	+1.28	$[-0.26,2.92]$	0.0524
	SOPO	GRPO	+1.47	$[-0.05,3.12]$	0.0282
	SOPO-FLOP	CAPO	+1.48	$[-0.15,3.19]$	0.0353
	SOPO-FLOP	GRPO	+1.66	$[0.08,3.43]$	0.0212
Math-Shepherd	SOPO	CAPO	+3.04	$[0.35,5.99]$	0.0131
	SOPO	GRPO	+3.19	$[-0.03,6.73]$	0.0267

Appendix

Table 14: OOD primary-metric sample-paired bootstrap comparisons (percentage points). Reported

\Delta

, 95% CIs, and one-sided

p

-values pair the median-scoring run from each method’s 10 inference runs and use

B=10{,}000

paired example-resampling replicates (seed 42). These nominal intervals and

p

-values are conditional on the paired inference runs.

p<10^{-4}

indicates zero nonpositive differences among the bootstrap replicates.

Figure 8: Probability density histogram of QA, data2text and summary task on RAGTruth Test set.

Hallucination Detection

Hallucination detection identifies unsupported, fabricated, or evidence-inconsistent content in language and multimodal model outputs.

See all 78 papers

RAGognizer: Hallucination-Aware Fine-Tuning via Detection Head Integration
Apr 17, 2026Fabian Ridder, Laurin Lessel, Malte SchillingHallucination in Language Models RAG Robustness
Computer Science Department, University of Münster, Münster, Germany
BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation
Jun 14, 2026Ning Li, Zixuan Guo, Yan Xu +7RL for Language Models LLM Hallucination Mitigation
1Shanghai Jiao Tong University · 2Tencent · 3The
Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLMs
Jul 21, 2026Xiaomeng Hu, Jiaqi Hu, Hao Chen +4RL for Language Model Reasoning Hallucination in Language Models
Zhejiang University

Apr 17, 2026J/K move · Enter open · S save

RAGognizer: Hallucination-Aware Fine-Tuning via Detection Head Integration

Fabian Ridder, Laurin Lessel, Malte Schilling

Computer Science Department, University of Münster, Münster, Germany

Hallucination in Language Models RAG Robustness LLM Hallucination Mitigation Retrieval-Augmented Generation Language Modeling LLM Hallucination Detection+1

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) is widely used to augment the input to Large Language Models (LLMs) with external information, such as recent or domain-specific knowledge. Nonetheless, current models still produce closed-domain hallucinations and generate content that is unsupported by the retrieved context. Current detection approaches typically treat hallucination as a post-hoc problem, relying on black-box consistency checks or probes over frozen internal representations. In this work, we demonstrate that hallucination detection based on internal state representation can also serve as a direct training signal. We introduce RAGognize, a dataset of naturally occurring closed-domain hallucinations with token-level annotations, and RAGognizer, a hallucination-aware fine-tuning approach that integrates a lightweight detection head into an LLM, allowing for the joint optimization of language modeling and hallucination detection. This joint objective forces the model to improve the separability of its internal states regarding hallucinations while simultaneously learning to generate well-formed and meaningful responses. Across multiple benchmarks, RAGognizer achieves state-of-the-art token-level hallucination detection while substantially reducing hallucination rates during generation, without degrading language quality or relevance.

Abstract

Hallucination Detection

Explore similar work

RAGognizer: Hallucination-Aware Fine-Tuning via Detection Head Integration

BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation

Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLMs