Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing

Jihao Zhao; Shuaishuai Zu; Zhiyuan Ji; Chunlai Zhou; Biao Qin

cs.CLMar 16, 2026

Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing

Authors: Jihao Zhao, Shuaishuai Zu, Zhiyuan Ji, Chunlai Zhou, Biao Qin

Organizations: School of Information, Renmin University of China, Beijing, China · School of Information, BRAIN, Renmin University of China, Beijing, China

Abstract

Training models for open-ended writing requires diverse prompts and sufficiently precise specifications of response quality. Constructing such specifications at scale entails recovering task requirements, consolidating overlapping criteria, and defining observable differences between performance levels. We introduce Writer-R1, a writing training pipeline built around Inductive Rubric Generation (IRG), which integrates these operations into an automated rubric construction procedure. The pipeline expands an open writing corpus and uses the resulting prompt-specific rubrics throughout supervised training and reinforcement learning (RL). Each rubric is concatenated with the user request and also defines the reward criteria, making the requirements used for assessment available to the writer before generation. A compact model, RubricGen-4B, learns to supply rubrics for new requests. Writer-R1 improves WritingBench scores by 0.182 and 0.158 over the 4B and 8B backbones, respectively, with gains also on HelloBench. Training ablations favor IRG over direct rubric generation, and further analyses support rubric transfer and the robustness of writer improvements across evaluators. These results support using inductively mined rubrics as both generation guidance and reward specifications for open-ended writing.

Figures & tables

Figure 1: Writer-R1 workflow. IRG rubrics condition generation and define RL rewards; RubricGen-4B supplies inference-time rubrics. $N$ , $K$ , and $G$ count candidate dimensions, retained dimensions, and sampled responses; $S_{i,k}$ scores response $i$ on dimension $k$ .

Mid-Scale General Models
Models	Overall	Domains						Requirements
Models	Overall	D1	D2	D3	D4	D5	D6	R1	C1	R2	C2	R3	C3
Mistral-Large-Instruct	0.764	0.77	0.76	0.78	0.73	0.79	0.76	0.77	0.82	0.77	0.87	0.77	0.79
Llama-3.3-70B-Instruct	0.701	0.70	0.69	0.70	0.68	0.73	0.73	0.71	0.78	0.71	0.82	0.70	0.72
Qwen-2.5-72B-Instruct	0.743	0.77	0.74	0.76	0.69	0.78	0.73	0.75	0.79	0.76	0.86	0.74	0.75
Qwen3-32B	0.788	0.78	0.77	0.79	0.79	0.80	0.81	0.80	0.84	0.79	0.86	0.79	0.80

Table 1: Comparison of writing models on WritingBench. D1 to D6: Academic and Engineering, Finance and Business, Politics and Law, Literature and Art, Education, and Advertising and Marketing. R1 to R3: Style, Format, and Length; C1 to C3: corresponding category scores.

Models	Overall		OEQA		Summ		Chat		TC		HTG
Models	Score	Length	Score	Length	Score	Length	Score	Length	Score	Length	Score	Length
Qwen3-4B	0.641	1,626.61	0.547	1,634.09	0.736	659.04	0.646	2,380.94	0.616	1,992.83	0.662	1,466.15
Qwen3-8B	0.682	1,561.45	0.583	1,671.10	0.743	608.35	0.725	2,364.53	0.649	1,910.64	0.709	1,252.61
WritingBench-7B	0.636	1,012.63	0.553	924.07	0.631	561.09	0.619	1,376.21	0.655	1,174.14	0.722	1,027.65
SuperWriter-7B	0.579	9,497.69	0.478	9,423.32	0.517	9,078.57	0.571	9,563.18	0.627	9,111.82	0.700	10,311.54
LongWriter-Zero-32B	0.713	2,711.94	0.769	2,004.73	0.672	935.03	0.690	3,443.42	0.659	4,935.58	0.776	2,240.93

Table 2: Comparison of response quality and mean word counts on HelloBench. OEQA: open-ended question answering; Summ: summarization; Chat: dialogue; TC: text completion; HTG: heuristic text generation.

Training configuration	Overall	D1	D2	D3	D4	D5	D6
Qwen3-4B	0.681	0.69	0.69	0.69	0.64	0.70	0.68
SFT	0.725	0.77	0.72	0.72	0.66	0.77	0.73
DRG + SFT	0.719	0.78	0.72	0.71	0.65	0.76	0.72
DRG + SFT + RL	0.766	0.74	0.74	0.79	0.78	0.77	0.79
IRG + SFT	0.816	0.85	0.82	0.81	0.77	0.84	0.81
IRG + SFT + RL	0.863	0.87	0.88	0.86	0.84	0.88	0.86

Table 3: WritingBench training ablations with Qwen3-4B; IRG + SFT + RL is Writer-R1-4B. Bold marks column maxima. D1 to D6: Academic and Engineering, Finance and Business, Politics and Law, Literature and Art, Education, and Advertising and Marketing.

Figure 2: Rubric transfer on WritingBench across writers (rows) and rubric sources (columns), with writer parameters fixed. DS-R1: DeepSeek-R1-0528; DS-V3: DeepSeek-V3; DS-V3.1: DeepSeek-V3.1; DS-V3.2: DeepSeek-V3.2-Exp; LongCat: LongCat-Flash-Chat; RGen-4B: RubricGen-4B. N/A: direct answering without feedback; Human: human-curated rubrics.

Figure 3: WritingBench evaluation by LLMs and humans (rows), comparing writers (columns) within rows. Human scores average four expert assessments. DS-V3.2: DeepSeek-V3.2-Exp; Qwen3-235B: Qwen3-235B-A22B; WritingBench: WritingBench-7B; SuperWriter: SuperWriter-7B; LongWriter: LongWriter-Zero-32B.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Setting	Value
SFT	Allocated instances	15K
	Teacher candidates per prompt	5
	Learning rate	$1\times 10^{-5}$
	Epochs	1
	Learning-rate schedule	Cosine
	Warm-up fraction	0.10

Appendix

Table 4: Shared training hyperparameters for Writer-R1-4B and Writer-R1-8B. The prompt and rubric form the writer input in both stages.

Figure 4: Illustration of interdependent writing decisions and their implications for draft assessment. (a) Feedback can inform related decisions about topic interpretation, argument, organization, style, logic, and theme. (b) Defects in early decisions can affect later parts of a response. Explicit criteria provide assessment targets for revision.

Figure 5: Rubric-guided revision with fixed writer parameters. Red: direct answering; blue: holistic LLM feedback; yellow: RubricGen-4B feedback. Radar plots report WritingBench scores; bars count samples requiring zero to four revisions. DeepSeek-R1: DeepSeek-R1-0528. D1 to D6: Academic and Engineering, Finance and Business, Politics and Law, Literature and Art, Education, and Advertising and Marketing. R1 to R3: Style, Format, and Length; C1 to C3: corresponding category scores. Overall: aggregate score.

Target writer	Rubric source	Overall	Domains						Requirements
Target writer	Rubric source	Overall	D1	D2	D3	D4	D5	D6	R1	C1	R2	C2	R3	C3
DeepSeek-V3	Human-curated	0.803	0.815	0.799	0.790	0.806	0.800	0.814	0.809	0.851	0.804	0.868	0.794	0.762
	DeepSeek-R1	0.788	0.791	0.780	0.781	0.793	0.773	0.816	0.800	0.832	0.793	0.850	0.791	0.743
	DeepSeek-R1-0528	0.806	0.814	0.799	0.791	0.808	0.802	0.831	0.817	0.858	0.810	0.866	0.801	0.767
	DeepSeek-V3.2-Exp	0.806	0.809	0.790	0.802	0.817	0.797	0.831	0.817	0.851	0.811	0.868	0.806	0.761
	LongCat-Flash-Chat	0.809	0.809	0.794	0.806	0.815	0.809	0.830	0.817	0.854	0.813	0.875	0.804	0.760

Appendix

Table 5: Writing performance under different rubric sources on WritingBench, with the target writer and evaluation setting fixed within each block. D1 to D6: Academic and Engineering, Finance and Business, Politics and Law, Literature and Art, Education, and Advertising and Marketing. R1 to R3: Style, Format, and Length; C1 to C3: corresponding category scores.

Evaluator	Writing model	Overall	Domains						Requirements
Evaluator	Writing model	Overall	D1	D2	D3	D4	D5	D6	R1	C1	R2	C2	R3	C3
DeepSeek-V3.2-Exp	Qwen3-4B	0.523	0.541	0.517	0.538	0.461	0.561	0.544	0.535	0.589	0.532	0.615	0.498	0.493
	Qwen3-8B	0.573	0.583	0.568	0.570	0.535	0.605	0.598	0.583	0.634	0.579	0.664	0.555	0.558
	WritingBench-7B	0.614	0.641	0.585	0.614	0.573	0.648	0.655	0.630	0.675	0.611	0.669	0.599	0.579
	SuperWriter-7B	0.624	0.628	0.611	0.626	0.594	0.665	0.644	0.637	0.674	0.622	0.675	0.608	0.559
	LongWriter-Zero-32B	0.703	0.710	0.693	0.696	0.682	0.740	0.720	0.704	0.739	0.702	0.744	0.697	0.700

Appendix

Table 6: Domain and requirement performance on WritingBench under different LLM evaluators. D1 to D6: Academic and Engineering, Finance and Business, Politics and Law, Literature and Art, Education, and Advertising and Marketing. R1 to R3: Style, Format, and Length; C1 to C3: corresponding category scores.

Figure 6: Sensitivity of writing performance to revision budget (a) and acceptance threshold (b) on WritingBench for LongCat-Flash-Chat and Kimi-K2. Max Rounds is $R_{\max}$ , including the initial draft; zero indicates direct answering. Passing Score is the normalized threshold $\eta$ .

Figure 7: Sensitivity of Writer-R1-4B performance on WritingBench to sampling temperature (a) and top- $p$ (b).

Memory-Augmented RL

See all 12 papers

From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks
Apr 30, 2026Qingyu Ren, Tianjun Pan, Xingzhou Chen +1Reward Modeling RL for Language Models
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University · 2Shanghai Artificial Intelligence Laboratory
UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learning
Apr 7, 2026Xiaolong Wei, Zerun Zhu, Simin Niu +9Text Generation RL for Language Models
Beihang University · Baidu Inc. · Renmin University of China
Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
Jul 2, 2026Dazhi Fu, Jiuding Yang, Yiwen Guo +1Reward Modeling LLM-as-a-Judge
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · LIGHTSPEED · Independent Researcher

Apr 30, 2026J/K move · Enter open · S save

From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks

Qingyu Ren, Tianjun Pan, Xingzhou Chen, Xuhong Wang

Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University · 2Shanghai Artificial Intelligence Laboratory

Reward Modeling RL for Language Models LLM Evaluation Language Model Generation Evaluation Instruction Following Reward Model Evaluation

Open paper AskPDF arXiv

Large language models have achieved remarkable progress in text generation but still struggle with generative writing tasks. In terms of evaluation, existing benchmarks evaluate writing reward models coarsely and fail to measure performance from the perspective of specific requirements. In terms of training, existing training methods either use LLM-as-a-judge approaches or train coarse-grained reward models, lacking fine-grained requirement-adherence reward modeling. To address these issues, we propose a fine-grained evaluation pipeline WEval for writing reward models and a fine-grained reinforcement learning training framework WRL. The evaluation data of WEval covers multiple task categories and requirement types, enabling systematic evaluation of writing reward models by measuring the correlation between the rankings of the reward model and gold rankings. WRL constructs positive and negative samples by selectively dropping instruction requirements, allowing for more precise reward model training. Experiments show that our models achieve substantial improvements across various writing benchmarks and exhibit strong generalization. The code and data are publicly available at https://github.com/Rainier-rq1/From_Coarse_to_Fine.

Abstract

Memory-Augmented RL

Explore similar work

From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks

UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learning

Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling