A fundamental challenge in creative writing lies in reconciling the inherent tension between maintaining global coherence in long-form narratives and preserving local expressiveness in short-form texts. While long-context generation necessitates explicit macroscopic planning, short-form creativity often demands spontaneous, constraint-free expression. Existing alignment paradigms, however, typically employ static reward signals and rely heavily on high-quality supervised data, which is costly and difficult to scale. To address this, we propose UniCreative, a unified reference-free reinforcement learning framework. We first introduce AC-GenRM, an adaptive constraint-aware reward model that dynamically synthesizes query-specific criteria to provide fine-grained preference judgments. Leveraging these signals, we propose ACPO, a policy optimization algorithm that aligns models with human preferences across both content quality and structural paradigms without supervised fine-tuning and ground-truth references. Empirical results demonstrate that AC-GenRM aligns closely with expert evaluations, while ACPO significantly enhances performance across diverse writing tasks. Crucially, our analysis reveals an emergent meta-cognitive ability: the model learns to autonomously differentiate between tasks requiring rigorous planning and those favoring direct generation, validating the effectiveness of our direct alignment approach.
Figures & tables
Figure 1: Examples of UniCreative generations. The long-form task (left) follows a Plan-then-Write procedure, while the short-form task (right) employs direct generation without intermediate planning.
Figure 2: UniCreative Framework. Top: The model adaptively selects Direct Generation or Plan-then-Write modes based on task nature. Bottom: ACPO training. The policy generates responses {o1,…,oG} which are evaluated by AC-GenRM using dynamically synthesized criteria to provide reward signals for optimization.
Category
Model
LitBench
Blessing
Baselines
Claude-Sonnet-3.7
0.731
0.902
Claude-3.5-Haiku
0.675
0.949
GPT-4.1
0.702
0.923
GPT-4.1-Mini
0.630
0.969
DeepSeek-V3
0.700
0.906
DeepSeek-R1
0.710
0.977
Table 1: Reward model evaluation showing agreement rates with proprietary judges on LitBench and Blessing . AC-GenRM is compared against unaligned Base models to isolate the algorithmic contribution.
Models
Avg
D1
D2
D3
D4
D5
D6
R1
C
R2
C
R3
C
Proprietary LLMs
O3-2025-04-16
85.27
84.81
85.20
83.89
85.88
85.82
86.80
85.14
87.45
85.24
90.98
86.31
87.21
Gemini-2.5-pro-preview
83.05
83.21
81.47
83.00
84.52
84.49
82.14
83.58
86.49
83.86
90.45
83.35
83.98
Claude-Sonnet-3.7
78.48
78.24
77.93
76.51
79.37
79.26
80.88
79.43
82.51
78.84
86.14
79.23
80.49
GPT-4o
75.46
74.40
73.42
74.38
77.91
75.86
78.08
76.82
81.57
75.82
85.46
76.13
76.73
o1-Preview
68.57
68.54
67.01
66.57
69.53
70.31
71.41
70.09
75.10
68.49
79.78
70.91
73.81
Table 2: Performance of different LLMs on WritingBench across six domains and three writing requirements. Scores are normalized from a 0-10 range to a 100-point scale. The corresponding domains and requirements are: (D1) Academic & Engineering, (D2) Finance & Business, (D3) Politics & Law, (D4) Literature & Art, (D5) Education, (D6) Advertising & Marketing, (R1) Style, (R2) Format, and (R3) Length. “C” denotes the category-specific scores of the three requirements.
Reference LLMs
Our Models (Ablation Study)
Model Name
Score
Model Name
Score
Gain ( Δ )
DeepSeek-V3-250324
87.2%
Qwen3-1.7B
8.40%
-
DeepSeek-V3.2
90.4%
Qwen3-1.7B-Thinking
64.2%
-
Qwen3-32B
45.2%
Qwen3-1.7B-Thinking + RL
90.0%
+25.8%
Qwen3-32B-Thinking
88.4%
Qwen3-4B
40.8%
-
GPT-4.1
89.4%
Qwen3-4B-Thinking
74.0%
-
Table 3: Performance evaluation on the Blessing short-text dataset. We employ an Adversarial Framework to benchmark standard LLMs (left) against our UniCreative variants (right). Score represents the percentage of excellent ratings. The Gain ( Δ ) column highlights the significant score improvement achieved by our RL method compared to the ‘thinking‘ baseline.
Model Size
Method
Accuracy on Mode Discrimination (5-run Avg. ± SD)
Easy
Hard
Avg.
Gain ( Δ )
Qwen3-1.7B
Thinking
64.2% ± 3.1%
65.1% ± 2.2%
64.7%
-
+ RL (Ours)
57.0% ± 1.1%
56.6% ± 1.4%
56.8%
-7.9%
Qwen3-4B
Thinking
77.1% ± 2.2%
62.4% ± 2.1%
69.8%
-
+ RL (Ours)
95.9% ± 0.7%
82.8% ± 1.7%
89.4%
+19.6%
Qwen3-8B
Thinking
80.5% ± 2.1%
68.5% ± 1.9%
74.5%
-
Table 4: Accuracy on the Mode Discrimination Benchmark across different model scales (5-run avg). Significant gains in 4B and 8B models underscore the efficacy of ACPO in fostering emergent meta-cognitive planning.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Training dynamics of the ACPO algorithm. We visualize key metrics over training steps: (a) Actor Entropy (indicating exploration), (b) KL Divergence from the reference model, (c) Clip Fraction, (d) Policy Gradient Loss, (e) Mean Reward trajectories, and (f) Average Response Length. The curves demonstrate stable convergence and consistent reward maximization.
Query
“Blessings for a friend’s daughter’s wedding.”
Criterion 1
Emotional Resonance : Evaluates the depth and appropriateness of the emotional impact; captures the joy and significance of the occasion.
Criterion 2
Creativity and Uniqueness : Assesses originality and creative flair; avoids clichés and leaves a lasting impression.
Criterion 3
Linguistic Conciseness : Measures clarity and brevity; ensures the message is succinct yet impactful.
Criterion 4
Cultural Appropriateness : Evaluates alignment with cultural norms and wedding-specific contexts.
Criterion 5
Aesthetic Appeal : Assesses the beauty and elegance of the language; rewards poetic and refined phrasing.
Appendix
Table 5: An example of synthesized evaluation criteria for a short-form creative query. AC-GenRM dynamically prioritizes emotional and aesthetic dimensions over structural or logical ones for this genre.
Query (User Input)
Label
Reasoning for Mode Selection
Subset: Easy (Highly intuitive and unambiguous requirements)
How do you say ’Thank you very much’ in Korean?
Short-form
Common phrase translation.
Explain how the human eye processes light and allows us to see images and colors.
Subset: Hard (Characterized by latent complexity or high levels of ambiguity)
Write a ’Recipe’ for disaster.
Short-form
Metaphorical creative writing; follows a recipe format but for an abstract concept.
Analyze the ’Evolution of the Hero’s Journey’ from Gilgamesh to Harry Potter.
Long-form
Comparative literature; requires a structured ’Monomyth’ stage breakdown.
Appendix
Table 6: Detailed examples from the Mode Discrimination Benchmark. The framework tests whether the model can distinguish between tasks requiring a Long-form (Plan-then-Write) pathway versus a Short-form (Direct Generation) pathway.
Training models for open-ended writing requires diverse prompts and sufficiently precise specifications of response quality. Constructing such specifications at scale entails recovering task requirements, consolidating overlapping criteria, and defining observable differences between performance levels. We introduce Writer-R1, a writing training pipeline built around Inductive Rubric Generation (IRG), which integrates these operations into an automated rubric construction procedure. The pipeline expands an open writing corpus and uses the resulting prompt-specific rubrics throughout supervised training and reinforcement learning (RL). Each rubric is concatenated with the user request and also defines the reward criteria, making the requirements used for assessment available to the writer before generation. A compact model, RubricGen-4B, learns to supply rubrics for new requests. Writer-R1 improves WritingBench scores by 0.182 and 0.158 over the 4B and 8B backbones, respectively, with gains also on HelloBench. Training ablations favor IRG over direct rubric generation, and further analyses support rubric transfer and the robustness of writer improvements across evaluators. These results support using inductively mined rubrics as both generation guidance and reward specifications for open-ended writing.
Jihao Zhao, Shuaishuai Zu, Zhiyuan Ji +2
School of Information, Renmin University of China, Beijing, China · School of Information, BRAIN, Renmin University of China, Beijing, China
Large Language Models (LLMs) are prone to generating hallucinated content, which compromises their reliability in knowledge-intensive tasks. To address this challenge without sacrificing creativity, we propose HARPO, a reinforcement learning framework designed to jointly optimize faithfulness and creativity. HARPO incorporates a Hallucination-Aware Generative Reward Model (HA-GRM), trained via verifiable feedback, to assess both faithfulness and writing quality. A Selective Activation Mechanism (SAM) activates writing rewards only for outputs judged hallucination-free by HA-GRM, while a data curriculum progressively shifts training from creative writing to hallucination-centric tasks. On RAGTruth, our Qwen3-4B-based HA-GRM achieves a response-level F1 score of 78.08%, compared with 66.37% for the supervised fine-tuning baseline. Experiments on Qwen2.5 and Qwen3 models from 1.7B to 8B parameters show improvements in both faithful generation and writing quality. On Qwen3-4B, HARPO reduces the HA-GRM-judged hallucination rate on MultiHopRAG from 3.29% to 1.02%, while increasing the Arena-Hard-v2.0 creative-writing score from 16.95% to 27.54%.
Generating coherent and controllable long-form content remains a persistent challenge for Large Language Models (LLMs). While reasoning-enhanced models have demonstrated success in logic-intensive domains, our evaluation reveals that they suffer from a severe length collapse in open-ended writing, where performance degrades sharply as target lengths exceed 2,000 words. We attribute this failure to the limitation of static hierarchical planning, which struggles to provide dynamic guidance over extended contexts. To bridge this gap, we introduce the Interleaved Structural Chain-of-Thought (IS-CoT) framework. Unlike external agentic workflows, IS-CoT embeds a dynamic Plan-Write-Reflect cycle into the generation process, enabling continuous strategy adaptation and global alignment without additional assistance. Based on this framework, we construct a high-quality dataset of interleaved reasoning traces via a multi-teacher pipeline and train IS-Writer-8B. Experiments demonstrate that IS-Writer-8B achieves state-of-the-art performance on challenging long-form benchmarks (e.g., +3.08 vs. DeepSeek-V3.2 on LongBench-Write), exhibiting robust length compliance and coherence competitive with significantly larger proprietary models.
Zechen Sun, Yuyang Sun, Zecheng Tang +6
Institute of Computer Science and Technology, Soochow University · 2Information Research Center of Military Science, PLA Academy of Military Science