Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Qwen Business Unit of Alibaba · University of Illinois at Urbana-Champaign
Verifiable rewards have advanced reinforcement learning in mathematics and code. However, many valuable open-ended tasks share three properties: no automatic verifier of overall quality, multiple requirements to satisfy jointly, and multiple valid responses to each query. Training therefore needs feedback on requirement satisfaction across different valid responses. Rubric-based reinforcement learning provides this feedback by aggregating scores from explicit criteria. As learning progresses unevenly, some criteria become fully satisfied and lose discrimination while others remain informative. Dynamic methods revise shared rubrics or select evolving criteria to restore discrimination, but may also change useful criteria. To address this, we propose Reinforcement Learning with Adaptive Rubric Evolution (RARE). The core idea is to use current policy feedback to guide both which criteria change and what their replacements assess. Sample-level updates replace criteria fully satisfied across current outputs with checks of remaining quality differences, retaining all other criteria and scores. Within this rule, we compare rewarding desirable qualities with penalizing observed flaws. Video script generation exemplifies these properties and supports content production. Its demands for coherent, executable narratives under constraints make it a challenging testbed. Targeting flaws performs best across three video types with a mean gain of 3.82 points over matched static rubrics. Model and human preferences support these gains. Improvements also extend to writing and general generation benchmarks.
Figures & tables
Figure 1: Sample-level RARE replaces saturated criteria while retaining other feedback. Illustrated on script generation, the generator proposes replacements from rollout differences. The judge rescores the replacements on the same responses before the policy update.
Video type
All
Train
Test
2D animation
370 / 2,362
333 / 2,125
37 / 237
3D/stop-motion
302 / 1,951
272 / 1,759
30 / 192
Live action
258 / 1,676
232 / 1,509
26 / 167
Total
930 / 5,989
837 / 5,393
93 / 596
Table 1: Dataset split by source video. Entries show videos / queries.
Model
2D animation
3D/stop-motion
Live action
Avg.
Large-scale reference models
DeepSeek-V4-Pro ( DeepSeek-AI, 2026 )
86.50
85.59
86.10
86.06
Gemini 3.1 Pro Preview ( Google DeepMind, 2026 )
82.12
81.99
78.42
80.84
GPT-5 (2025-08-07) ( OpenAI, 2025 )
89.86
89.39
89.35
89.53
Kimi K2.5 ( Kimi Team, 2026 )
85.46
84.42
82.61
84.16
Qwen3.8-Max ( Qwen Team, 2026 )
91.05
91.15
91.02
91.07
Table 2: RARE improves over matched static baselines. Scores are query averages within each video type. Avg. gives equal weight to the three types. RL results average four seeds. Parentheses show gains over the corresponding static baseline at the same update level.
Figure 2: Replacements restore score differences. Panel (a) shows separation and resaturation. In (b), boxes show interquartile ranges, inner lines medians, and diamonds means. Dashed lines mark full credit. Appendix A.4 details the summaries.
Metric
Negative
Mixed
Positive
Post-switch score ↓
0.82
0.84
0.86
Opens headroom ↑
80.00%
77.90%
68.50%
Restores variance ↑
78.20%
76.40%
67.50%
Mean ∣A^rep∣↑
0.66
0.65
0.57
Later resaturation ↓
24.50%
36.30%
34.10%
Table 3: Negative replacements yield stronger signals. In (a), A^rep uses replacement scores only. In (b), z standardizes each criterion separately. Δ gives relative changes, except percentage points (pp) for the headroom rate.
Figure 3: A replacement exposes differences in causal coherence. Four of eight unchanged scripts show how shifting from element coverage to causal explanation changes scores.
Figure 5: Both judges favor Negative . Dynamic and Static denote Negative and Sample-Static.
Model
BiGGen Bench (1 to 5)
WritingBench (1 to 10)
Base
4.035
8.22
Static
4.152
8.26
RARE ( Negative )
4.259
8.34
Table 4: RARE outperforms Static on both tasks. Scores are benchmark macro averages.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Assessment focus
Query Fulfillment (QF)
Requirement Coverage (Cov.)
Realizes the requested content and outcomes while satisfying explicit formal constraints.
Grounded Elaboration (Elab.)
Develops the query through relevant additions without contradicting its requirements or diluting the main storyline.
Tone and Style Alignment (Style)
Matches the requested genre, mood, rhythm, and visual style.
Narrative Coherence (NC)
Causal and Temporal Logic (Logic)
Maintains plausible causal links and a consistent temporal sequence.
Appendix
Table 5: Twelve fixed metrics assess script quality. Abbreviations match the detailed result tables.
Figure 6: Initial criteria reach saturation throughout dynamic training. Curves show the cumulative fraction first awarding all eight rollouts full credit in each dynamic setting.
Figure 7: A task-level revision improves subsequent training. Curves compare branches trained for 20 steps from a shared checkpoint with fixed original or revised rubrics.
Evaluation panel
Events
New mean
New headroom
New variance
Old full groups
Old mean
Old full rollouts
Trigger batch (eligible cohort)
940
–
73.2%
72.0%
100.0%
1.000
100.0%
End of epoch 1
940
0.906
53.4%
52.6%
56.0%
0.943
86.6%
End of training
940
0.931
43.4%
43.0%
66.3%
0.957
90.4%
Appendix
Table 6: Revised criteria distinguish new scripts while earlier requirements remain largely satisfied. Bold marks larger values across checkpoints.
Style
Checkpoint
Events
New mean
Headroom
Variance
Old full groups
Old mean
Old full rollouts
2D animation
Epoch 1 (step 66)
379
0.887
60.4%
59.4%
48.0%
0.928
83.5%
Final (step 132)
379
0.917
48.3%
47.8%
60.7%
0.946
88.1%
3D and stop-motion
Epoch 1 (step 54)
382
0.926
46.6%
46.1%
63.4%
0.957
89.5%
Final (step 108)
382
0.944
40.1%
39.8%
70.2%
0.968
92.3%
Live action
Epoch 1 (step 47)
179
0.906
53.1%
52.0%
57.0%
0.944
86.7%
Final (step 94)
179
0.931
40.2%
39.7%
69.8%
0.958
90.9%
Appendix
Table 7: Scores rise across all three production styles. Metrics use newly sampled scripts at each checkpoint. Bold marks larger values within each style.
User query Could you write an imaginative fairy tale in which a chicken with exceptionally long legs proves very useful at night ? It should be cute, absurd, and a little moving.
Comparison Base presents the long legs through moonlit footsteps and a reflection. Negative uses them to solve the children’s problem. The chicken presses the toy box with its long leg, returning the toys and Xiaohe’s kite. This action fulfills the requested usefulness at night.
Base
Scene 2, the first appearance, night “As it walks, its toes lift the moonlight, as though writing a little unsigned poem.”
Moonlight covers the grass. The long-legged chicken slowly emerges from behind an old locust tree. Each step stretches its shadow across the ground, like wind moving over a canvas. The ends of its legs glow faintly, as if touched by stars.
Xiaohe peers out from indoors, her voice trembling. “You… you’re not a chicken?”
“Not a chicken, but a bearer of the night.”
Appendix
Table 11: The chicken’s long legs help recover missing toys. Selected scenes from a 2D fairy tale.
User query I would like a beautiful but somewhat cruel nature fable about why people should not casually claim things that do not belong to them .
Comparison Base moves from a gentle encounter to a severed trunk, then has Lin Yuan acknowledge turning the tree into an exhibit. Negative depicts the intervention itself. Lin Ye’s fear of losing the tree motivates binding and hammering its roots, which shake as the soil collapses. The scene links possession to harm through the character’s actions.
Base
Scene 3, a pact beneath the canopy, afternoon “When I reached out to touch it, it reached back to hold my hand.”
The camera advances from Lin Yuan’s perspective. He stands in the forest as sunlight falls through the leaves, the canopy covering him like an umbrella. He slowly reaches out his right hand and touches the trunk with his fingertips. The bark’s patterns are as distinct as growth rings. The tree’s shadow moves toward him across the ground, as if responding.
His breathing slows. Lin Yuan says calmly, “So it does move. It was simply waiting for me, for someone willing to believe in it.”
Scene 4, a man-made fracture, night “They say a saw can save a tree. But what the saw cuts open is its memory.”
Appendix
Table 12: Possessiveness leads to visible harm. Selected scenes from a 2D nature fable.
User query Write a fairy-tale animation without dialogue about a vulnerable creature and something that cannot speak keeping each other company in a dangerous environment .
Comparison Base uses speech and later a glowing warning signal. In Negative , the companion reshapes its body to shelter the beetle from a storm. Physical protection gives companionship a concrete function in the dangerous setting without character dialogue.
Base
Scene 1, meeting on the snowfield, dusk “I had never seen snow that could move. I thought it was merely a silent stone.”
Xiaoxue comes to a stop in the wind, its hind legs slightly bent and its ears twitching. Snow drifts from its nose and falls at its feet like fallen flowers. It looks up and sees a mound of snow shaped like a pillar broken by the wind. Moss covers its surface, and it sways slightly, as though waiting for Xiaoxue to approach.
Xiaoxue, softly, almost inaudibly, says, “Can you… hear the wind too?”
“The wind slips out of the rock crevice like a letter. It says that something is waiting for you.”
Appendix
Table 13: Protective movement conveys companionship. A request for dialogue-free animation.
User query Create a small-animal story from a first-person perspective. It should begin with foraging and survival, then show a brief spark of affection making the animal willing to take risks or even sacrifice itself.
Comparison Both scripts express concern for a vulnerable animal. In Base , the storm prevents Graytail from acting on it. Negative connects an earlier moment of attachment to repeated attempts to reach Darktail despite unstable ice. Frostfur’s return after retreating and its eventual crossing give the requested risk-taking a visible progression.
Base
Scene 5, fleeing the sandstorm, noon “The wind is fire, a blade, the world’s anger.”
A violent wind rises. A sandstorm closes in from every direction, blurring earth and sky. Graytail scrambles about, its claws leaving deep marks in the sand. It looks back several times and sees the rabbit’s burrow covered in sand. The rabbit does not move. It wants to rush back but cannot take a step. The wind has blown it about like a dead leaf.
“I can’t… I can’t let it disappear,” I say, hoarsely and haltingly.
“If I die, it will never have my shadow.”
Appendix
Table 14: Concern motivates a dangerous crossing. A first-person animal story.
User query Give me the opening of a monster short film in a stark, realistic style, preferably starting with a nighttime escape and a fight for survival .
Comparison Base shows an escape followed by a threatening appearance in the mirror. Negative turns the encounter into combat. Smoke briefly forces the monster back, and a later pipe strike makes it stagger before the wound closes. These reactions connect the protagonist’s attempts to survive with consequences for his opponent.
Base
Scene 1, fleeing through an alley, night “I have never seen it, but I know it is watching me. It does not move. It simply waits.”
Li Yuan slides through the narrow passage, his backpack hitting the iron wall with a dull thud. He lowers his head, sweeping his flashlight across the ground and its patterns of broken glass. He hears his heartbeat in time with breathing from a distant doorway. He keeps moving. An old poster on the wall to his right splits, revealing wet darkness behind it.
Li Yuan whispers, almost inaudibly, “It isn’t a living thing… it’s the weather.”
“I run. It follows.”
Appendix
Table 15: The protagonist fights back against the monster. A nighttime pursuit.
User query Could you write a first-person romantic micro-script with a quiet protagonist who likes drawing and writing ? The story should feel cinematic.
Comparison Base moves from a brief glance to the narrator’s claim of being understood, with little response from the other character to support it. Negative shows Su Rui responding to a drawing and later to Lin Yu’s poem. Their relationship develops through these exchanges.
Base
Scene 3, the empty space by the window, day “You sit there longer than the sunset lasts.”
I return to the classroom and sit in my original seat. Sunlight falls on the windowsill. Chen Mo is looking down, gently touching his pencil to the draft paper. The lines are thin, as though tracing a breath. Looking down at the paper, I ask very softly, “Are you drawing the wind? Or those moments when you look up each time I hold my pen?”
I look up and see him raise his eyes, his gaze passing over me. Without a word, he sets his pencil on the corner of the desk and turns away. “I suddenly feel you are waiting rather than drawing. Waiting for me to put down my pen and pick it up again.”
Scene 4, the forgotten desk corner, night “I looked through seven sketchbooks and never found a second drawing. There was only a water-damaged page, with the figure’s hand reaching toward me.”
Appendix
Table 16: The relationship develops through drawings and poetry. A quiet romance.
Condition
Instruction
Negative
Every update must use negative polarity. A response receives the maximum score when the target defect is absent. The score decreases progressively when the defect is partially present, and becomes zero when the defect is pronounced and harms the response quality.
Positive
Every update must use positive polarity. A response receives the maximum score when the target quality is fully realized. The score decreases progressively as the realization becomes weaker, and becomes zero when the quality is absent from the response.
Mixed
Each update may use either positive or negative polarity. If two or more updates are generated, the resulting set must contain both polarities. For either polarity, a higher score must always indicate higher response quality.
Appendix
Table 17: Polarity instructions for the shared prompt. Translated from Chinese.
Rubric-based rewards offer interpretable and fine-grained optimization signals for reinforcement learning in open-ended tasks where verifiable answers are unavailable. However, pre-constructed rubrics remain static throughout training, creating a fundamental mismatch with the evolving policy: fixed criteria gradually lose discriminative power as the model improves, leading to reward saturation and potential hacking. Recent dynamic rubric methods partially address this but rely on external frontier models or ground-truth answers, and update rubrics only at coarse granularity. We propose EvoRubrics, a co-evolutionary RL framework where a Policy LLM and a Rubric Generator jointly improve through adversarial interaction within each training step. As the policy improves under the rubric generator's guidance, the rubric generator adapts its criteria to remain discriminative and informative, enabling evaluation to track the policy in real time and naturally inducing an automatic curriculum. Experiments show that EvoRubrics consistently outperforms static and dynamic rubric baselines across benchmarks. The learned Rubric Generator further generalizes as a transferable reward model. Notably, even a fully self-supervised variant without any external supervision achieves meaningful gains, suggesting that co-evolution between generation and evaluation alone can provide sufficiently rich learning signals. Our code is publicly available at https://anonymous.4open.science/r/EvoRubrics-2155/.
Hongxin Ding, Baixiang Huang, Yue Fang +6
National Engineering Research Center of Software Engineering, Peking University, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education +2
Reliable reward and preference signals are critical for evaluating and optimizing large language models on open-ended tasks. Rubric-based judges offer a transparent way to decompose such judgments into explicit evaluation criteria, but existing annotation-free rubric generators typically rely on a single generic evaluator. As a result, they may overlook important dimensions of human preference, a failure mode we term dimensional blind spots. To address this limitation, we propose Multi-Role Rubric Generation (MRRG), a training-free and reference-free framework that elicits evaluation criteria from multiple complementary roles and consolidates them into an auditable rubric-based scorer. This scorer can be used both to validate pairwise preferences and to provide rewards for GRPO-style Reinforcement Learning with Verifiable Rewards (RLVR). Experiments on preference validation benchmarks show that MRRG consistently outperforms single-role rubric generation baselines across multiple backbone models. Further RLVR experiments demonstrate that MRRG yields a stronger reward signal for improving open-ended generation.
Dazhi Fu, Jiuding Yang, Yiwen Guo +1
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · LIGHTSPEED · Independent Researcher
Reinforcement Learning (RL) has advanced Large Language Models (LLMs) in verifiable domains, while open-ended generation remains challenging due to the absence of definitive rewards. Rubric-based RL provides explicit evaluation criteria, but learning to construct these criteria remains challenging when final-answer correctness is not verifiable. We propose EvoRubric, a co-evolutionary RL framework that combines criterion-validity feedback, response discrimination, and peer agreement to learn rubrics for open-ended generation. A shared policy acts as both a Reasoner and a Rubric Generator, using its current responses and historical rubrics to discover new evaluation dimensions. To combine adaptive rubric discovery with a stable validity check, a frozen copy of the initial policy serves as the Meta-Verifier, while a frozen Grader scores responses against the retained criteria. Discriminative feedback, Leave-One-Out peer consensus, and a persistent memory pool transform this feedback into complementary rewards that jointly optimize both policy roles, closing the loop between response improvement and rubric discovery. EvoRubric improves over matched static and external evolving-rubric baselines across five benchmarks in Medical, Writing, and Science. Across three training seeds, it achieves five-benchmark averages of 56.28 at 8B and 61.13 at 14B, exceeding the strongest matched baselines by 3.09 and 2.06 points, respectively. Human audits assess criterion validity and response quality, and experiments with expert-initialized rubrics demonstrate compatibility with human priors.