Triviality Corrected Endogenous Reward

Xinda Wang; Zhengxu Hou; Yangshijie Zhang; Bingren Yan; Jialin Liu; Chenzhuo Zhao; Zhibo Yang; Bin-Bin Yang

cs.CLApr 13, 2026

Triviality Corrected Endogenous Reward

Authors: Xinda Wang, Zhengxu Hou, Yangshijie Zhang, Bingren Yan, Jialin Liu, Chenzhuo Zhao, Zhibo Yang, Bin-Bin Yang,

Organizations: Peking University · Alibaba Group · Lanzhou University

Abstract

Reinforcement learning for open-ended text generation is constrained by the lack of verifiable rewards, necessitating reliance on judge models that require either annotated data or powerful closed-source models. Inspired by recent work on unsupervised reinforcement learning for mathematical reasoning using confidence-based endogenous rewards, we investigate whether this principle can be adapted to open-ended writing tasks. We find that directly applying confidence rewards leads to Triviality Bias: the policy collapses toward high-probability outputs, reducing diversity and meaningful content. We propose TCER (Triviality Corrected Endogenous Reward), which addresses this bias by rewarding the relative information gain between a specialist policy and a generalist reference policy, modulated by a probability-dependent correction mechanism. Across multiple writing benchmarks and model architectures, TCER achieves consistent improvements without external supervision. Furthermore, TCER also transfers effectively to mathematical reasoning, validating the generality of our approach across different generation tasks.

Figures & tables

Figure 1: TCER computes rewards using both base and actor models, emphasizing high-entropy meaningful content over EndoR. After RL Training, TCER shows gentler entropy decay with enhanced exploration.

Figure 2: Overview of TCER training pipeline and reward comparison. (a) Training workflow: SFT on high-quality data to obtain $\pi_{s}$ , followed by Reference Augmented GRPO with TCER that incorporate information gain $\phi$ and gating weight $w$ . (b) Sentence-level reward visualization on a rollout: token-level rewards are averaged within each sentence comparing TCER and EndoR. (Rewards are average log-probs; higher is closer to $0$ .)

General Closed-source Models
Model	LB	HB-A	HB-B	WB-A	WB-B	WB-C	WB-D	WB-E	WB-F
GPT-4o	83.1	83.7	87.6	74.4	73.4	74.3	77.9	75.8	78.0
Claude 3.5	89.3	82.9	88.3	59.5	57.6	56.3	59.3	62.0	67.7
Claude 3.7	97.8	83.9	93.2	78.2	77.9	76.5	79.3	79.2	80.8
General Open-source Models
Llama3.1-8B-Instruct	60.3	45.5	41.3	47.5	45.6	43.7	42.3	48.6	50.8

Table 1: Performance comparison on writing benchmarks. We evaluate models across LongBench-Write, HelloBench subsets, and WritingBench domains. Starting from Qwen2.5-7B-Instruct, each training stage (SFT, RL-EndoR, RL-TCER) demonstrates progressive improvements.

DeepWriting Dataset
Model	LB	HB-A	HB-B	WB-A	WB-B	WB-C	WB-D	WB-E	WB-F
DeepWriter-8B	85.6	77.8	85.7	72.2	71.8	69.8	70.6	73.7	72.3
+ RL-EndoR	86.1	79.6	86.9	76.0	73.6	73.9	69.6	77.3	74.5
+ RL-TCER	87.3	82.2	86.2	76.5	75.3	74.8	74.2	76.7	75.2
LongWriter Dataset
LongWriter-8B	76.5	68.8	82.9	57.9	53.9	49.0	52.0	52.9	52.0

Table 2: Effectiveness of TCER for existing finetuned models. Applying our RL methods to finetuned models DeepWriter-8B and LongWriter-8B.

Method	AIME 24	AIME 25	AMC	MATH-500	Minerva	Olympiad	Avg
Qwen2.5-7B-Math	11.2	4.4	30.8	46.0	7.7	14.4	19.1
+ SFT	22.2	22.3	52.8	82.6	40.8	43.7	44.1
+ SFT + RL-EndoR	30.5	25.4	60.9	87.0	42.6	46.8	48.9
+ SFT + RL-TCER	32.4	26.1	62.1	86.4	44.5	49.6	50.2
Llama3.1-8B-Instruct	4.6	0.2	21.2	46.4	20.9	12.4	17.6
+ SFT	9.4	9.5	39.0	68.4	27.2	32.4	30.9

Table 3: Performance comparison on mathematical reasoning benchmarks. TCER outperforms both the SFT baseline and EndoR across different base models.

Figure 3: Performance generalization across different models and datasets. For each model group, the bars represent the performance of the base model (grey), after SFT (light blue), after applying RL-EndoR (dark blue), and after applying our RL-TCER (orange).

Metric	EndoR	TCER
High-quality avg.	-1.11	-0.80
Regular avg.	-1.08	-0.93
High-quality Recall %	20.7%	35.8%

Table 4: Sentence-level analysis of reward signals. Sentences identified as brilliant by Gemini-2.5-Pro, GPT-4o, Claude-Opus4 are evaluated. Analysis conducted on 10,000 outputs with

\geq

5 high-quality sentences per text.

Config	LB	HB-A	HB-B	WB-A	WB-B	WB-C	WB-D	WB-E	WB-F
FULL TCER	86.3	81.1	86.4	72.7	73.1	70.8	67.8	73.4	71.1
w/o $w(y_{i}\|s_{i})\cdot\phi(y_{i}\|s_{i})$	84.5	79.0	84.7	72.6	70.9	70.1	64.6	73.0	68.7
w/o $\log\pi_{s}(y_{i}\|s_{i})$	76.2	70.8	72.0	68.5	68.2	65.6	60.1	69.4	64.1
w/o $w(y_{i}\|s_{i})$	81.5	76.5	79.1	70.6	70.5	71.2	62.7	73.1	65.3
w/o Ref-Augmentation	85.7	78.2	85.6	72.9	72.3	70.6	66.6	73.3	71.2

Table 5: Ablation studies on Qwen2.5-7B-Instruct with the DeepWriting dataset. Each component contributes to the overall performance.

Figure 4: Displays these trajectories as line plots, with EndoR scores shown in blue and TCER scores in orange. High-quality sentences identified by our judge panel are marked with bold red points.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: RL training dynamics on writing tasks. For each dataset, we report the EndoR reward trajectory, the TCER reward trajectory , and a direct entropy comparison between EndoR and TCER .

Panel of Figure 6 — Figure 6: RL training dynamics on mathematical reasoning tasks. We report EndoR and TCER reward trajectories together with entropy comparisons for each model configuration.

Configuration	LB	HB-A	HB-B	WB-A	WB-B	WB-C	WB-D	WB-E	WB-F
$(k=1,\lambda=1)$	83.7	78.5	83.9	70.8	71.2	68.6	65.3	71.5	69.0
$(k=2,\lambda=2)$	85.1	79.8	85.2	71.9	72.4	69.7	66.5	72.6	70.2
$(k=3,\lambda=2)$	86.3	81.1	86.4	72.7	73.1	70.8	67.8	73.4	71.1
$(k=4,\lambda=2)$	85.9	80.7	86.0	72.4	72.8	70.4	67.3	73.0	70.7
$(k=3,\lambda=1)$	85.4	80.2	85.6	72.1	72.5	70.1	67.0	72.8	70.4
$(k=3,\lambda=3)$	85.8	80.6	85.9	72.3	72.7	70.3	67.2	72.9	70.6

Appendix

Table 6: Hyperparameter sensitivity analysis on the DeepWriter dataset with Qwen2.5-7B-Instruct. Results show win rates or benchmark scores across LongBench (LB), HelloBench subsets (HB-A: Open QA, HB-B: Text Generation), and WritingBench domains (WB-A through WB-F representing different professional writing categories). The optimal configuration

(k=3,\lambda=2)

achieves consistent improvements across all metrics.

Figure 8: Reward case study (Model output from SFT).

Figure 9: Reward case study. The table reports EndoR and TCER scores for each line, $\Delta=\text{TCER}-\text{EndoR}$ , and whether the line is selected by the closed-source judge panel.

Figure 10: English reward case study: prompt and model output.

Figure 11: English reward case study: sentence-level EndoR vs. TCER rewards. The table reports $\Delta=\text{TCER}-\text{EndoR}$ and whether each line is selected by the judge panel.

Figure 12: Writing case (SFT). Response generated by the SFT-only model.

Figure 13: Writing case (EndoR). Response generated by the EndoR-trained model under the same prompt.

Figure 14: Writing case (TCER). Response generated by the TCER-trained model under the same prompt.

Figure 15: English writing case (SFT). Response generated by the SFT-only model.

Figure 16: English writing case (EndoR). Response generated by the EndoR-trained model under the same prompt.

Figure 17: English writing case (TCER). Response generated by the TCER-trained model under the same prompt.

Figure 18: Sentence-selection prompt used by individual judge models (GPT-4o, Claude Opus 4, and Gemini 2.5 Pro). Each judge independently selects at least five highlighted sentences from the same output.

Figure 19: Aggregation prompt used by Gemini 2.5 Pro to consolidate highlighted sentences across judges via agreement-based selection.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 433 papers

OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning
Jun 24, 2026Wenxuan Jiang, Zining Fan, Zijian Zhang +6Open-Ended Generation Intrinsic Reward Methods
The Hong Kong Polytechnic University · Peking University · Meituan Longcat Team +2
RARE: Reinforcement Learning with Adaptive Rubric Evolution for Open-Ended Generation
Oct 18, 2026Xueyun Tian, Bingbing Xu, Chuyue Li +9Rubric-Based RL Reward Design for RL
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Qwen Business Unit of Alibaba +1
From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks
Apr 30, 2026Qingyu Ren, Tianjun Pan, Xingzhou Chen +1Reward Modeling RL for Language Models
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University · 2Shanghai Artificial Intelligence Laboratory

Jun 24, 2026J/K move · Enter open · S save

OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning

Wenxuan Jiang, Zining Fan, Zijian Zhang, Xuecheng Wu+5

The Hong Kong Polytechnic University · Peking University · Meituan Longcat Team+2

Open-Ended Generation Intrinsic Reward Methods LLM Alignment Reinforcement Learning RL for Language Model Reasoning Text Generation

Open paper AskPDF arXiv

Reinforcement Learning (RL) has enabled LLMs to excel in objective reasoning tasks such as mathematics and code generation. However, applying RL to open-ended tasks, such as creative writing, remains challenging because LLM-as-a-judge reward models often exhibit stylistic biases and positional inconsistencies, leading to unstable supervision. To address this, we propose OPERA (Objective Perplexity-based Reflective Alignment), which replaces unreliable external judges with intrinsic rewards derived from perplexity dynamics. Specifically, we derive an intrinsic reward signal from perplexity dynamics, quantifying uncertainty reduction at critical reflective states. During the cold-start phase, we introduce a data synthesis method that leverages carefully designed guiding words to generate diverse reasoning traces, along with perplexity-prioritized rollouts that utilize internal log-probabilities to identify logically consistent reasoning branches. This pipeline yields a large-scale dataset comprising 20,000 high-quality reasoning trajectories. Empirical evaluations consistently demonstrate the scalability and efficacy of our approach in alignment for open-ended tasks. Implementing OPERA on Qwen3-8B establishes a new state-of-the-art among open-source models, achieving parity with or surpassing proprietary models like Gemini2.5 and MiniMax-M2.5 in some open-ended tasks. The code is available at https://github.com/pangpang-xuan/OPERA.

Abstract

RL for Language Models

Explore similar work

OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning

RARE: Reinforcement Learning with Adaptive Rubric Evolution for Open-Ended Generation

From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks