UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learning

Xiaolong Wei; Zerun Zhu; Simin Niu; Xingyu Zhang; Peiying Yu; Changxuan Xiao; Yuchen Li; Jicheng Yang

cs.AIApr 7, 2026

UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learning

Authors: Xiaolong Wei, Zerun Zhu, Simin Niu, Xingyu Zhang, Peiying Yu, Changxuan Xiao, Yuchen Li, Jicheng Yang,

Organizations: Beihang University · Baidu Inc. · Renmin University of China

Abstract

A fundamental challenge in creative writing lies in reconciling the inherent tension between maintaining global coherence in long-form narratives and preserving local expressiveness in short-form texts. While long-context generation necessitates explicit macroscopic planning, short-form creativity often demands spontaneous, constraint-free expression. Existing alignment paradigms, however, typically employ static reward signals and rely heavily on high-quality supervised data, which is costly and difficult to scale. To address this, we propose UniCreative, a unified reference-free reinforcement learning framework. We first introduce AC-GenRM, an adaptive constraint-aware reward model that dynamically synthesizes query-specific criteria to provide fine-grained preference judgments. Leveraging these signals, we propose ACPO, a policy optimization algorithm that aligns models with human preferences across both content quality and structural paradigms without supervised fine-tuning and ground-truth references. Empirical results demonstrate that AC-GenRM aligns closely with expert evaluations, while ACPO significantly enhances performance across diverse writing tasks. Crucially, our analysis reveals an emergent meta-cognitive ability: the model learns to autonomously differentiate between tasks requiring rigorous planning and those favoring direct generation, validating the effectiveness of our direct alignment approach.

Figures & tables

Panel of Figure 1 — Figure 1: Examples of UniCreative generations. The long-form task (left) follows a Plan-then-Write procedure, while the short-form task (right) employs direct generation without intermediate planning.

Figure 2: UniCreative Framework. Top: The model adaptively selects Direct Generation or Plan-then-Write modes based on task nature. Bottom: ACPO training. The policy generates responses

\{o_{1},\dots,o_{G}\}

which are evaluated by AC-GenRM using dynamically synthesized criteria to provide reward signals for optimization.

Category	Model	LitBench	Blessing
Baselines	Claude-Sonnet-3.7	0.731	0.902
	Claude-3.5-Haiku	0.675	0.949
	GPT-4.1	0.702	0.923
	GPT-4.1-Mini	0.630	0.969
	DeepSeek-V3	0.700	0.906
	DeepSeek-R1	0.710	0.977

Table 1: Reward model evaluation showing agreement rates with proprietary judges on LitBench and Blessing . AC-GenRM is compared against unaligned Base models to isolate the algorithmic contribution.

Proprietary LLMs
Models	Avg	D1	D2	D3	D4	D5	D6	R1	C	R2	C	R3	C
O3-2025-04-16	85.27	84.81	85.20	83.89	85.88	85.82	86.80	85.14	87.45	85.24	90.98	86.31	87.21
Gemini-2.5-pro-preview	83.05	83.21	81.47	83.00	84.52	84.49	82.14	83.58	86.49	83.86	90.45	83.35	83.98
Claude-Sonnet-3.7	78.48	78.24	77.93	76.51	79.37	79.26	80.88	79.43	82.51	78.84	86.14	79.23	80.49
GPT-4o	75.46	74.40	73.42	74.38	77.91	75.86	78.08	76.82	81.57	75.82	85.46	76.13	76.73
o1-Preview	68.57	68.54	67.01	66.57	69.53	70.31	71.41	70.09	75.10	68.49	79.78	70.91	73.81

Table 2: Performance of different LLMs on WritingBench across six domains and three writing requirements. Scores are normalized from a 0-10 range to a 100-point scale. The corresponding domains and requirements are: (D1) Academic & Engineering, (D2) Finance & Business, (D3) Politics & Law, (D4) Literature & Art, (D5) Education, (D6) Advertising & Marketing, (R1) Style, (R2) Format, and (R3) Length. “C” denotes the category-specific scores of the three requirements.

Reference LLMs		Our Models (Ablation Study)
Model Name	Score	Model Name	Score	Gain ( $\Delta$ )
DeepSeek-V3-250324	87.2%	Qwen3-1.7B	8.40%	-
DeepSeek-V3.2	90.4%	Qwen3-1.7B-Thinking	64.2%	-
Qwen3-32B	45.2%	Qwen3-1.7B-Thinking + RL	90.0%	+25.8%
Qwen3-32B-Thinking	88.4%	Qwen3-4B	40.8%	-
GPT-4.1	89.4%	Qwen3-4B-Thinking	74.0%	-

Table 3: Performance evaluation on the Blessing short-text dataset. We employ an Adversarial Framework to benchmark standard LLMs (left) against our UniCreative variants (right). Score represents the percentage of excellent ratings. The Gain (

\Delta

) column highlights the significant score improvement achieved by our RL method compared to the ‘thinking‘ baseline.

Model Size	Method	Accuracy on Mode Discrimination (5-run Avg. $\pm$ SD)
Model Size	Method	Easy	Hard	Avg.	Gain ( $\Delta$ )
Qwen3-1.7B	Thinking	64.2% $\pm$ 3.1%	65.1% $\pm$ 2.2%	64.7%	-
Qwen3-1.7B	+ RL (Ours)	57.0% $\pm$ 1.1%	56.6% $\pm$ 1.4%	56.8%	-7.9%
Qwen3-4B	Thinking	77.1% $\pm$ 2.2%	62.4% $\pm$ 2.1%	69.8%	-
Qwen3-4B	+ RL (Ours)	95.9% $\pm$ 0.7%	82.8% $\pm$ 1.7%	89.4%	+19.6%
Qwen3-8B	Thinking	80.5% $\pm$ 2.1%	68.5% $\pm$ 1.9%	74.5%	-

Table 4: Accuracy on the Mode Discrimination Benchmark across different model scales (5-run avg). Significant gains in 4B and 8B models underscore the efficacy of ACPO in fostering emergent meta-cognitive planning.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Training dynamics of the ACPO algorithm. We visualize key metrics over training steps: (a) Actor Entropy (indicating exploration), (b) KL Divergence from the reference model, (c) Clip Fraction, (d) Policy Gradient Loss, (e) Mean Reward trajectories, and (f) Average Response Length. The curves demonstrate stable convergence and consistent reward maximization.

Query	“Blessings for a friend’s daughter’s wedding.”
Criterion 1	Emotional Resonance : Evaluates the depth and appropriateness of the emotional impact; captures the joy and significance of the occasion.
Criterion 2	Creativity and Uniqueness : Assesses originality and creative flair; avoids clichés and leaves a lasting impression.
Criterion 3	Linguistic Conciseness : Measures clarity and brevity; ensures the message is succinct yet impactful.
Criterion 4	Cultural Appropriateness : Evaluates alignment with cultural norms and wedding-specific contexts.
Criterion 5	Aesthetic Appeal : Assesses the beauty and elegance of the language; rewards poetic and refined phrasing.

Appendix

Table 5: An example of synthesized evaluation criteria for a short-form creative query. AC-GenRM dynamically prioritizes emotional and aesthetic dimensions over structural or logical ones for this genre.

Query (User Input)	Label	Reasoning for Mode Selection
Subset: Easy (Highly intuitive and unambiguous requirements)
How do you say ’Thank you very much’ in Korean?	Short-form	Common phrase translation.
Explain how the human eye processes light and allows us to see images and colors.	Long-form	Biological explanation; requires logical, scientific steps.
Subset: Hard (Characterized by latent complexity or high levels of ambiguity)
Write a ’Recipe’ for disaster.	Short-form	Metaphorical creative writing; follows a recipe format but for an abstract concept.
Analyze the ’Evolution of the Hero’s Journey’ from Gilgamesh to Harry Potter.	Long-form	Comparative literature; requires a structured ’Monomyth’ stage breakdown.

Appendix

Table 6: Detailed examples from the Mode Discrimination Benchmark. The framework tests whether the model can distinguish between tasks requiring a Long-form (Plan-then-Write) pathway versus a Short-form (Direct Generation) pathway.

Figure 4: Dynamic Criteria Generation Examples.

Figure 5: Over-Determination in Short Texts.

Figure 6: Structural Collapse in Long-form.

Figure 7: Prompt for the Criteria Generator.

Figure 8: Prompt for the AC-GenRM.

Figure 9: Prompt for the Generate.

Text Generation

Text generation studies methods for producing coherent, useful language from prompts, data, and dialogue context.

See all 410 papers

Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing
Mar 16, 2026Jihao Zhao, Shuaishuai Zu, Zhiyuan Ji +2Memory-Augmented RL RL for Language Models
School of Information, Renmin University of China, Beijing, China · School of Information, BRAIN, Renmin University of China, Beijing, China
HARPO: Hallucination-Aware Reinforcement Learning for Faithful and Creative Language Generation
Oct 2, 2026Tiezheng Yu, Yuxin Jiang, Jinpeng Li +4LLM Hallucination Mitigation Text Generation
Huawei Technologies
IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking
Jun 8, 2026Zechen Sun, Yuyang Sun, Zecheng Tang +6Open-Ended Generation Long-Form Document Generation
Institute of Computer Science and Technology, Soochow University · 2Information Research Center of Military Science, PLA Academy of Military Science

Mar 16, 2026J/K move · Enter open · S save

Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing

Jihao Zhao, Shuaishuai Zu, Zhiyuan Ji, Chunlai Zhou+1

School of Information, Renmin University of China, Beijing, China · School of Information, BRAIN, Renmin University of China, Beijing, China

Memory-Augmented RL RL for Language Models Language Model Generation Evaluation Rubric-Based RL Reinforcement Learning

Open paper AskPDF arXiv

Training models for open-ended writing requires diverse prompts and sufficiently precise specifications of response quality. Constructing such specifications at scale entails recovering task requirements, consolidating overlapping criteria, and defining observable differences between performance levels. We introduce Writer-R1, a writing training pipeline built around Inductive Rubric Generation (IRG), which integrates these operations into an automated rubric construction procedure. The pipeline expands an open writing corpus and uses the resulting prompt-specific rubrics throughout supervised training and reinforcement learning (RL). Each rubric is concatenated with the user request and also defines the reward criteria, making the requirements used for assessment available to the writer before generation. A compact model, RubricGen-4B, learns to supply rubrics for new requests. Writer-R1 improves WritingBench scores by 0.182 and 0.158 over the 4B and 8B backbones, respectively, with gains also on HelloBench. Training ablations favor IRG over direct rubric generation, and further analyses support rubric transfer and the robustness of writer improvements across evaluators. These results support using inductively mined rubrics as both generation guidance and reward specifications for open-ended writing.

Abstract

Text Generation

Explore similar work

Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing

HARPO: Hallucination-Aware Reinforcement Learning for Faithful and Creative Language Generation

IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking