R2-Write: Reflection and Revision for Open-Ended Writing with Deep Reasoning

Wanlong Liu; Bo Zhang; Chenliang Li; Shaopeng Lai; Yuning Wu; Xuanyu Lei; Ming Yan

cs.CLApr 3, 2026

R2-Write: Reflection and Revision for Open-Ended Writing with Deep Reasoning

Authors: Wanlong Liu, Bo Zhang, Chenliang Li, Shaopeng Lai, Yuning Wu, Xuanyu Lei, Ming Yan

Organizations: Tongyi Lab, Alibaba Group · Tsinghua University, Beijing, China

Abstract

While deep reasoning with long chain-of-thought has dramatically improved large language models in verifiable domains like mathematics, its effectiveness for open-ended tasks such as writing remains unexplored. In this paper, we conduct a systematic investigation revealing that existing mainstream reasoning models achieve limited gains on open-ended writing tasks. Our further analysis shows that these models lack deep reflection and revision patterns in open-ended writing, resulting in substantially smaller improvements compared to mathematical reasoning tasks. To address this limitation, we introduce R2-Write: an automated framework that synthesizes high-quality thinking trajectories enriched with explicit reflection and revision patterns through iterative writer-judge interaction. To prevent redundant reflections, we design a process reward mechanism that supervises reflection quality during reinforcement learning, improving both performance and token efficiency. Extensive experiments across multiple creative writing and deep-research benchmarks demonstrate significant improvements, validating that explicitly incorporating reflection and revision patterns unlocks deep reasoning capabilities for open-ended writing tasks.

Figures & tables

Benchmark	Qwen3-30B-A3B			Qwen3-8B			DeepSeek
Benchmark	No Thinking	Thinking	$\boldsymbol{\Delta}$ (%)	No Thinking	Thinking	$\boldsymbol{\Delta}$ (%)	V3.1	R1-0528	$\boldsymbol{\Delta}$ (%)
WritingBench	78.2	79.1	+1.2	71.2	71.8	+0.8	78.8	79.0	+0.3
HelloBench	79.8	80.4	+0.7	71.9	72.2	+0.4	77.4	78.0	+0.7
MATH 500	88.2	97.5	+10.5	86.8	96.4	+11.1	92.2	98.8	+7.2
AIME 25	24.6	73.9	+200.4	20.9	67.3	+221.5	43.9	87.5	+99.3

Table 1: Performance of different models on creative writing and mathematical reasoning benchmarks. For Qwen3-30B-A3B, No Thinking corresponds to Qwen3-30B-A3B-Instruct and Thinking to Qwen3-30B-A3B-Thinking . For Qwen3-8B, Thinking and No Thinking use the same base model with different system prompts.

\Delta

denotes the relative percentage change from the left column to the right column.

Figure 1: Thinking pattern analysis. The first row shows the pattern distributions of three reasoning models on WritingBench and MATH500. The second row reports, for each model and task, the proportion of patterns that are judged to be helpful for obtaining the correct (or high-scoring) answer. All pattern annotations are obtained using Claude-4.5-Sonnet ( Anthropic, 2025 ) .

Figure 2: Overview of the R 2 -Write pipeline, which consists of three main parts: query data selection, data creation and RL.

Proprietary LLMs
Method	Training	Writing Bench	Deepresearch Gym			DiscoX
Method	Training	Score( $\uparrow$ )	KPR( $\uparrow$ )	KPC( $\downarrow$ )	Quality( $\uparrow$ )	Acc.( $\uparrow$ )	Fluency( $\uparrow$ )	Appro.( $\uparrow$ )
GPT-5	-	85.87	76.26	2.20	7.98	42.62	12.42	12.40
Gemini2.5-Pro	-	82.15	73.20	2.68	7.40	40.23	11.86	13.26
Claude-4-sonnet	-	81.76	77.10	1.87	7.86	38.65	11.42	11.80
SFT-Based Methods

Table 2: Performance comparison across different training methods on Writing Bench, Deepresearch Gym, and DiscoX datasets.

\uparrow

indicates higher is better,

\downarrow

indicates lower is better.

\text{RL}_{\text{p}}

denotes RL training with process-level supervision.

Figure 3: Token length distribution of thinking trajectories across different methods on Writing Bench.

Method	WritingBench	DeepResearch Gym
R 2 -Write + SFT	0.586	0.541
R 2 -Write + SFT + RL	0.660	0.584
R 2 -Write + SFT + $\text{RL}_{\text{p}}$	0.805	0.729

Table 3: ProcessBench Results on WritingBench and DeepResearch Gym. We use Qwen3-8B as the base model. “RL” denotes the baseline RL setup consistent with the main experiments, and “

\text{RL}_{\text{p}}

” denotes RL with process reward supervision.

Dataset	RA	FLC	QE
WritingBench	10.50%	19.88%	69.62%
DeepResearch Gym	24.17%	34.72%	41.11%

Table 4: Distribution of revision patterns across different datasets. The table shows the percentage of each revision type (RA: Requirement Alignment, FLC: Factual & Logical Correction, QE: Quality Enhancement) identified in the reflection and revision process for WritingBench and DeepResearch Gym datasets.

Method	AIME25	MMLU-Pro
Qwen3-8B + Distill	66.78	59.62
R 2 -Write-SFT	67.84	60.42
Qwen3-8B + Distill + RL	65.50	58.28
R 2 -Write-SFT + $\text{RL}_{\text{p}}$	64.78	59.22

Table 5: Performance comparison on non-writing tasks. R 2 -Write maintains competitive performance on AIME25 and MMLU-Pro, demonstrating that reflection and revision patterns do not compromise the model’s general capabilities.

Pointwise Reward:
Method	Reward $\times$ Algorithm	Score
Qwen3-8B (w/o RL)	–	71.84
+ PPO	Pointwise $\times$ PPO	73.42
+ GRPO	Pointwise $\times$ GRPO	75.20
Pairwise Reward:
+ GRPO	Pairwise $\times$ GRPO	77.87

Table 6: Ablation study on reward design and RL algorithms. All results are evaluated on WritingBench.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Data distribution of our constructed training set, which includes both SFT and RL data. (a) shows the domain distribution for creative writing tasks, and (b) shows the category distribution for report generating tasks.

Rubric Type	GPT-5	Gemini-2.5-Pro	DeepSeek-V3.1	Std.
Query-Specific (0-1)	0.82	0.89	0.81	0.04
General Quality (0-10)	7.35	7.05	7.25	0.15

Appendix

Table 7: Average scores by LLM judges. Query-specific rubrics are on 0-1 scale, general quality rubrics on 0-10 scale.

Task Type	GPT-5	Gemini-2.5-Pro	DeepSeek-V3.1	Avg.
Creative Writing	89.0%	88.0%	91.0%	89.3%
Report Generation	91.0%	90.0%	93.0%	91.3%
Overall	90.0%	89.0%	92.0%	90.3%

Appendix

Table 8: Human-LLM alignment rates across task types.

Pattern Type	Definition	Example
Answer Verification	The reasoning includes explicit or implicit steps that check intermediate computations or final results for correctness.	“Let’s verify this result by …”, “Reviewing: the story is in the first person and within 500 words …”
Backtracking	The reasoning identifies an error or dead end and switches to a different approach or recalculates.	“I forgot to address the second question; I need to rewrite the ending …”, “Wait, I made an error. Let me recalculate …”
Subgoal Setting	The reasoning explicitly introduces intermediate subgoals that decompose the problem into smaller parts.	“First, I’ll calculate …then I’ll …”, “Step 1: Find the area. Step 2: Calculate the perimeter.”
Backward Chaining	The reasoning starts from the target result and works backward to infer the required inputs or steps.	“To get $24$ , I need $24\div 2=12$ …”, “Working backwards from answer …”
Summarization	The reasoning summarizes completed subtasks or current progress and indicates the remaining steps.	“Now we have obtained $x=5$ , next we need to …”, “So far we’ve found …the remaining step is …”

Appendix

Table 9: Definitions and examples of the five reasoning patterns.

Category	Item	Cost
Dataset Construction	Rubrics Generation	$81
Dataset Construction	Pipeline (Judge model)	$52
Evaluation	WritingBench	$1.9
Evaluation	Deepresearch Gym	$2.7
Training (A100)	SFT Training	40 GPU hrs
Training (A100)	RL Training	580 GPU hrs

Appendix

Table 10: Overall Cost Breakdown

Method	Training	HelloBench	DeepresearchBench
Method	Training	Avg. Score( $\uparrow$ )	Overall( $\uparrow$ )	Comp.( $\uparrow$ )	Depth( $\uparrow$ )	Inst.( $\uparrow$ )	Read( $\uparrow$ )
SFT-Based Methods
Qwen3-4B	-	65.86	38.11	36.32	33.41	43.66	42.33
Qwen3-8B	-	71.80	40.52	38.74	36.27	45.85	44.18
Longwriter-9B	SFT	62.00	38.68	36.84	34.45	43.18	40.26
Reverse-Engineering-8B	SFT	74.90	42.42	41.20	39.44	44.18	44.86

Appendix

Table 11: Performance comparison across different training methods on HelloBench and DeepresearchBench datasets.

\uparrow

indicates higher is better.

Figure 5: Effectiveness of reflection pattern usage. We categorize cases where reflection patterns are triggered into three outcomes: Win (R 2 -Write outperforms baseline), Tie (comparable performance), and Lose (baseline outperforms R 2 -Write). DG represents Deepresearch Gym, and WB means Writing Bench. The vast majority of reflection instances lead to performance improvements, demonstrating that the model effectively leverages these patterns rather than applying them indiscriminately.

Figure 6: Evolution of total reward, process reward, and answer reward during RL training (steps 0-400). All three reward components show consistent improvement, indicating that the model learns to optimize both the reasoning process and final answer quality simultaneously.

Figure 7: Comparison of response length dynamics during RL training. Our model (right) shows decreasing response length, indicating more efficient reasoning, while the baseline Qwen3-8B + RL (left) exhibits increasing verbosity.

Panel of Figure 10 — Figure 10: Prompt for generating query-specific evaluation rubrics for language model responses.

Panel of Figure 11 — Figure 11: Prompt for generating general-quality evaluation criteria.

Panel of Figure 12 — Figure 12: Prompt for constructing self-reflection and revision thought process during Data Construction.

Panel of Figure 13 — Figure 13: Prompt for response revision based on self-reflection during Data Construction.

Panel of Figure 14 — Figure 14: The generation prompt for Math tasks in our experiments.

Panel of Figure 15 — Figure 15: The Generation Prompt on Writing tasks (WritingBench and HelloBench) in our experiments.

Panel of Figure 16 — Figure 16: Prompt for Answer Reward in RL training

Panel of Figure 17 — Figure 17: Prompt for Process Reward in RL training (Step 1).

Panel of Figure 18 — Figure 18: Prompt for Process Reward in RL training (Step 2).

Panel of Figure 19 — Figure 19: Prompt for analyzing patterns of thinking trajectories in language model outputs. For writing tasks, we replace the Ground Truth in the prompt with Evaluation Rubrics .

Panel of Figure 20 — Figure 20: Prompt for classifying revision types in writing task reflection. The model analyzes the reasoning content to identify the primary issue: Requirement Alignment (RA), Factual & Logical Correction (FLC), or Quality Enhancement (QE).

Open-Ended Generation

Open-ended generation studies how AI produces coherent, useful, and varied responses when prompts allow many valid answers.

See all 41 papers

OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning
Jun 24, 2026Wenxuan Jiang, Zining Fan, Zijian Zhang +6Open-Ended Generation Intrinsic Reward Methods
The Hong Kong Polytechnic University · Peking University · Meituan Longcat Team +2
IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking
Jun 8, 2026Zechen Sun, Yuyang Sun, Zecheng Tang +6Open-Ended Generation Long-Form Document Generation
Institute of Computer Science and Technology, Soochow University · 2Information Research Center of Military Science, PLA Academy of Military Science
Triviality Corrected Endogenous Reward
Apr 13, 2026Xinda Wang, Zhengxu Hou, Yangshijie Zhang +6RL for Language Models Reward Design for RL
Peking University · Alibaba Group · Lanzhou University

Jun 24, 2026J/K move · Enter open · S save

OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning

Wenxuan Jiang, Zining Fan, Zijian Zhang, Xuecheng Wu+5

The Hong Kong Polytechnic University · Peking University · Meituan Longcat Team+2

Open-Ended Generation Intrinsic Reward Methods LLM Alignment Reinforcement Learning RL for Language Model Reasoning Text Generation

Open paper AskPDF arXiv

Reinforcement Learning (RL) has enabled LLMs to excel in objective reasoning tasks such as mathematics and code generation. However, applying RL to open-ended tasks, such as creative writing, remains challenging because LLM-as-a-judge reward models often exhibit stylistic biases and positional inconsistencies, leading to unstable supervision. To address this, we propose OPERA (Objective Perplexity-based Reflective Alignment), which replaces unreliable external judges with intrinsic rewards derived from perplexity dynamics. Specifically, we derive an intrinsic reward signal from perplexity dynamics, quantifying uncertainty reduction at critical reflective states. During the cold-start phase, we introduce a data synthesis method that leverages carefully designed guiding words to generate diverse reasoning traces, along with perplexity-prioritized rollouts that utilize internal log-probabilities to identify logically consistent reasoning branches. This pipeline yields a large-scale dataset comprising 20,000 high-quality reasoning trajectories. Empirical evaluations consistently demonstrate the scalability and efficacy of our approach in alignment for open-ended tasks. Implementing OPERA on Qwen3-8B establishes a new state-of-the-art among open-source models, achieving parity with or surpassing proprietary models like Gemini2.5 and MiniMax-M2.5 in some open-ended tasks. The code is available at https://github.com/pangpang-xuan/OPERA.

Abstract

Open-Ended Generation

Explore similar work

OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning

IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking

Triviality Corrected Endogenous Reward