While deep reasoning with long chain-of-thought has dramatically improved large language models in verifiable domains like mathematics, its effectiveness for open-ended tasks such as writing remains unexplored. In this paper, we conduct a systematic investigation revealing that existing mainstream reasoning models achieve limited gains on open-ended writing tasks. Our further analysis shows that these models lack deep reflection and revision patterns in open-ended writing, resulting in substantially smaller improvements compared to mathematical reasoning tasks. To address this limitation, we introduce R2-Write: an automated framework that synthesizes high-quality thinking trajectories enriched with explicit reflection and revision patterns through iterative writer-judge interaction. To prevent redundant reflections, we design a process reward mechanism that supervises reflection quality during reinforcement learning, improving both performance and token efficiency. Extensive experiments across multiple creative writing and deep-research benchmarks demonstrate significant improvements, validating that explicitly incorporating reflection and revision patterns unlocks deep reasoning capabilities for open-ended writing tasks.
Figures & tables
Benchmark
Qwen3-30B-A3B
Qwen3-8B
DeepSeek
No Thinking
Thinking
Δ (%)
No Thinking
Thinking
Δ (%)
V3.1
R1-0528
Δ (%)
WritingBench
78.2
79.1
+1.2
71.2
71.8
+0.8
78.8
79.0
+0.3
HelloBench
79.8
80.4
+0.7
71.9
72.2
+0.4
77.4
78.0
+0.7
MATH 500
88.2
97.5
+10.5
86.8
96.4
+11.1
92.2
98.8
+7.2
AIME 25
24.6
73.9
+200.4
20.9
67.3
+221.5
43.9
87.5
+99.3
Table 1: Performance of different models on creative writing and mathematical reasoning benchmarks. For Qwen3-30B-A3B, No Thinking corresponds to Qwen3-30B-A3B-Instruct and Thinking to Qwen3-30B-A3B-Thinking . For Qwen3-8B, Thinking and No Thinking use the same base model with different system prompts. Δ denotes the relative percentage change from the left column to the right column.
Figure 1: Thinking pattern analysis. The first row shows the pattern distributions of three reasoning models on WritingBench and MATH500. The second row reports, for each model and task, the proportion of patterns that are judged to be helpful for obtaining the correct (or high-scoring) answer. All pattern annotations are obtained using Claude-4.5-Sonnet ( Anthropic, 2025 ) .
Figure 2: Overview of the R 2 -Write pipeline, which consists of three main parts: query data selection, data creation and RL.
Method
Training
Writing Bench
Deepresearch Gym
DiscoX
Score( ↑ )
KPR( ↑ )
KPC( ↓ )
Quality( ↑ )
Acc.( ↑ )
Fluency( ↑ )
Appro.( ↑ )
Proprietary LLMs
GPT-5
-
85.87
76.26
2.20
7.98
42.62
12.42
12.40
Gemini2.5-Pro
-
82.15
73.20
2.68
7.40
40.23
11.86
13.26
Claude-4-sonnet
-
81.76
77.10
1.87
7.86
38.65
11.42
11.80
SFT-Based Methods
Table 2: Performance comparison across different training methods on Writing Bench, Deepresearch Gym, and DiscoX datasets. ↑ indicates higher is better, ↓ indicates lower is better. RLp denotes RL training with process-level supervision.
Figure 3: Token length distribution of thinking trajectories across different methods on Writing Bench.
Method
WritingBench
DeepResearch Gym
R 2 -Write + SFT
0.586
0.541
R 2 -Write + SFT + RL
0.660
0.584
R 2 -Write + SFT + RLp
0.805
0.729
Table 3: ProcessBench Results on WritingBench and DeepResearch Gym. We use Qwen3-8B as the base model. “RL” denotes the baseline RL setup consistent with the main experiments, and “ RLp ” denotes RL with process reward supervision.
Dataset
RA
FLC
QE
WritingBench
10.50%
19.88%
69.62%
DeepResearch Gym
24.17%
34.72%
41.11%
Table 4: Distribution of revision patterns across different datasets. The table shows the percentage of each revision type (RA: Requirement Alignment, FLC: Factual & Logical Correction, QE: Quality Enhancement) identified in the reflection and revision process for WritingBench and DeepResearch Gym datasets.
Method
AIME25
MMLU-Pro
Qwen3-8B + Distill
66.78
59.62
R 2 -Write-SFT
67.84
60.42
Qwen3-8B + Distill + RL
65.50
58.28
R 2 -Write-SFT + RLp
64.78
59.22
Table 5: Performance comparison on non-writing tasks. R 2 -Write maintains competitive performance on AIME25 and MMLU-Pro, demonstrating that reflection and revision patterns do not compromise the model’s general capabilities.
Method
Reward × Algorithm
Score
Qwen3-8B (w/o RL)
–
71.84
Pointwise Reward:
+ PPO
Pointwise × PPO
73.42
+ GRPO
Pointwise × GRPO
75.20
Pairwise Reward:
+ GRPO
Pairwise × GRPO
77.87
Table 6: Ablation study on reward design and RL algorithms. All results are evaluated on WritingBench.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Data distribution of our constructed training set, which includes both SFT and RL data. (a) shows the domain distribution for creative writing tasks, and (b) shows the category distribution for report generating tasks.
Rubric Type
GPT-5
Gemini-2.5-Pro
DeepSeek-V3.1
Std.
Query-Specific (0-1)
0.82
0.89
0.81
0.04
General Quality (0-10)
7.35
7.05
7.25
0.15
Appendix
Table 7: Average scores by LLM judges. Query-specific rubrics are on 0-1 scale, general quality rubrics on 0-10 scale.
Task Type
GPT-5
Gemini-2.5-Pro
DeepSeek-V3.1
Avg.
Creative Writing
89.0%
88.0%
91.0%
89.3%
Report Generation
91.0%
90.0%
93.0%
91.3%
Overall
90.0%
89.0%
92.0%
90.3%
Appendix
Table 8: Human-LLM alignment rates across task types.
Pattern Type
Definition
Example
Answer Verification
The reasoning includes explicit or implicit steps that check intermediate computations or final results for correctness.
“Let’s verify this result by …”, “Reviewing: the story is in the first person and within 500 words …”
Backtracking
The reasoning identifies an error or dead end and switches to a different approach or recalculates.
“I forgot to address the second question; I need to rewrite the ending …”, “Wait, I made an error. Let me recalculate …”
Subgoal Setting
The reasoning explicitly introduces intermediate subgoals that decompose the problem into smaller parts.
“First, I’ll calculate …then I’ll …”, “Step 1: Find the area. Step 2: Calculate the perimeter.”
Backward Chaining
The reasoning starts from the target result and works backward to infer the required inputs or steps.
“To get 24 , I need 24÷2=12 …”, “Working backwards from answer …”
Summarization
The reasoning summarizes completed subtasks or current progress and indicates the remaining steps.
“Now we have obtained x=5 , next we need to …”, “So far we’ve found …the remaining step is …”
Appendix
Table 9: Definitions and examples of the five reasoning patterns.
Category
Item
Cost
Dataset Construction
Rubrics Generation
$81
Pipeline (Judge model)
$52
Evaluation
WritingBench
$1.9
Deepresearch Gym
$2.7
Training (A100)
SFT Training
40 GPU hrs
RL Training
580 GPU hrs
Appendix
Table 10: Overall Cost Breakdown
Method
Training
HelloBench
DeepresearchBench
Avg. Score( ↑ )
Overall( ↑ )
Comp.( ↑ )
Depth( ↑ )
Inst.( ↑ )
Read( ↑ )
SFT-Based Methods
Qwen3-4B
-
65.86
38.11
36.32
33.41
43.66
42.33
Qwen3-8B
-
71.80
40.52
38.74
36.27
45.85
44.18
Longwriter-9B
SFT
62.00
38.68
36.84
34.45
43.18
40.26
Reverse-Engineering-8B
SFT
74.90
42.42
41.20
39.44
44.18
44.86
Appendix
Table 11: Performance comparison across different training methods on HelloBench and DeepresearchBench datasets. ↑ indicates higher is better.
Figure 5: Effectiveness of reflection pattern usage. We categorize cases where reflection patterns are triggered into three outcomes: Win (R 2 -Write outperforms baseline), Tie (comparable performance), and Lose (baseline outperforms R 2 -Write). DG represents Deepresearch Gym, and WB means Writing Bench. The vast majority of reflection instances lead to performance improvements, demonstrating that the model effectively leverages these patterns rather than applying them indiscriminately.
Figure 6: Evolution of total reward, process reward, and answer reward during RL training (steps 0-400). All three reward components show consistent improvement, indicating that the model learns to optimize both the reasoning process and final answer quality simultaneously.
Figure 7: Comparison of response length dynamics during RL training. Our model (right) shows decreasing response length, indicating more efficient reasoning, while the baseline Qwen3-8B + RL (left) exhibits increasing verbosity.
Figure 10: Prompt for generating query-specific evaluation rubrics for language model responses.
Figure 11: Prompt for generating general-quality evaluation criteria.
Figure 12: Prompt for constructing self-reflection and revision thought process during Data Construction.
Figure 13: Prompt for response revision based on self-reflection during Data Construction.
Figure 14: The generation prompt for Math tasks in our experiments.
Figure 15: The Generation Prompt on Writing tasks (WritingBench and HelloBench) in our experiments.
Figure 16: Prompt for Answer Reward in RL training
Figure 17: Prompt for Process Reward in RL training (Step 1).
Figure 18: Prompt for Process Reward in RL training (Step 2).
Figure 19: Prompt for analyzing patterns of thinking trajectories in language model outputs. For writing tasks, we replace the Ground Truth in the prompt with Evaluation Rubrics .
Figure 20: Prompt for classifying revision types in writing task reflection. The model analyzes the reasoning content to identify the primary issue: Requirement Alignment (RA), Factual & Logical Correction (FLC), or Quality Enhancement (QE).
Reinforcement Learning (RL) has enabled LLMs to excel in objective reasoning tasks such as mathematics and code generation. However, applying RL to open-ended tasks, such as creative writing, remains challenging because LLM-as-a-judge reward models often exhibit stylistic biases and positional inconsistencies, leading to unstable supervision. To address this, we propose OPERA (Objective Perplexity-based Reflective Alignment), which replaces unreliable external judges with intrinsic rewards derived from perplexity dynamics. Specifically, we derive an intrinsic reward signal from perplexity dynamics, quantifying uncertainty reduction at critical reflective states. During the cold-start phase, we introduce a data synthesis method that leverages carefully designed guiding words to generate diverse reasoning traces, along with perplexity-prioritized rollouts that utilize internal log-probabilities to identify logically consistent reasoning branches. This pipeline yields a large-scale dataset comprising 20,000 high-quality reasoning trajectories. Empirical evaluations consistently demonstrate the scalability and efficacy of our approach in alignment for open-ended tasks. Implementing OPERA on Qwen3-8B establishes a new state-of-the-art among open-source models, achieving parity with or surpassing proprietary models like Gemini2.5 and MiniMax-M2.5 in some open-ended tasks. The code is available at https://github.com/pangpang-xuan/OPERA.
Wenxuan Jiang, Zining Fan, Zijian Zhang +6
The Hong Kong Polytechnic University · Peking University · Meituan Longcat Team +2
Generating coherent and controllable long-form content remains a persistent challenge for Large Language Models (LLMs). While reasoning-enhanced models have demonstrated success in logic-intensive domains, our evaluation reveals that they suffer from a severe length collapse in open-ended writing, where performance degrades sharply as target lengths exceed 2,000 words. We attribute this failure to the limitation of static hierarchical planning, which struggles to provide dynamic guidance over extended contexts. To bridge this gap, we introduce the Interleaved Structural Chain-of-Thought (IS-CoT) framework. Unlike external agentic workflows, IS-CoT embeds a dynamic Plan-Write-Reflect cycle into the generation process, enabling continuous strategy adaptation and global alignment without additional assistance. Based on this framework, we construct a high-quality dataset of interleaved reasoning traces via a multi-teacher pipeline and train IS-Writer-8B. Experiments demonstrate that IS-Writer-8B achieves state-of-the-art performance on challenging long-form benchmarks (e.g., +3.08 vs. DeepSeek-V3.2 on LongBench-Write), exhibiting robust length compliance and coherence competitive with significantly larger proprietary models.
Zechen Sun, Yuyang Sun, Zecheng Tang +6
Institute of Computer Science and Technology, Soochow University · 2Information Research Center of Military Science, PLA Academy of Military Science
Reinforcement learning for open-ended text generation is constrained by the lack of verifiable rewards, necessitating reliance on judge models that require either annotated data or powerful closed-source models. Inspired by recent work on unsupervised reinforcement learning for mathematical reasoning using confidence-based endogenous rewards, we investigate whether this principle can be adapted to open-ended writing tasks. We find that directly applying confidence rewards leads to Triviality Bias: the policy collapses toward high-probability outputs, reducing diversity and meaningful content. We propose TCER (Triviality Corrected Endogenous Reward), which addresses this bias by rewarding the relative information gain between a specialist policy and a generalist reference policy, modulated by a probability-dependent correction mechanism. Across multiple writing benchmarks and model architectures, TCER achieves consistent improvements without external supervision. Furthermore, TCER also transfers effectively to mathematical reasoning, validating the generality of our approach across different generation tasks.
Xinda Wang, Zhengxu Hou, Yangshijie Zhang +6
Peking University · Alibaba Group · Lanzhou University