cs.CLOct 18, 2026
SaveRARE: Reinforcement Learning with Adaptive Rubric Evolution for Open-Ended Generation
Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Qwen Business Unit of Alibaba · University of Illinois at Urbana-Champaign
Abstract
Verifiable rewards have advanced reinforcement learning in mathematics and code. However, many valuable open-ended tasks share three properties: no automatic verifier of overall quality, multiple requirements to satisfy jointly, and multiple valid responses to each query. Training therefore needs feedback on requirement satisfaction across different valid responses. Rubric-based reinforcement learning provides this feedback by aggregating scores from explicit criteria. As learning progresses unevenly, some criteria become fully satisfied and lose discrimination while others remain informative. Dynamic methods revise shared rubrics or select evolving criteria to restore discrimination, but may also change useful criteria. To address this, we propose Reinforcement Learning with Adaptive Rubric Evolution (RARE). The core idea is to use current policy feedback to guide both which criteria change and what their replacements assess. Sample-level updates replace criteria fully satisfied across current outputs with checks of remaining quality differences, retaining all other criteria and scores. Within this rule, we compare rewarding desirable qualities with penalizing observed flaws. Video script generation exemplifies these properties and supports content production. Its demands for coherent, executable narratives under constraints make it a challenging testbed. Targeting flaws performs best across three video types with a mean gain of 3.82 points over matched static rubrics. Model and human preferences support these gains. Improvements also extend to writing and general generation benchmarks.
Figures & tables
Figure 1: Sample-level RARE replaces saturated criteria while retaining other feedback. Illustrated on script generation, the generator proposes replacements from rollout differences. The judge rescores the replacements on the same responses before the policy update.
| Video type | All | Train | Test |
|---|---|---|---|
| 2D animation | 370 / 2,362 | 333 / 2,125 | 37 / 237 |
| 3D/stop-motion | 302 / 1,951 | 272 / 1,759 | 30 / 192 |
| Live action | 258 / 1,676 | 232 / 1,509 | 26 / 167 |
| Total | 930 / 5,989 | 837 / 5,393 | 93 / 596 |
Table 1: Dataset split by source video. Entries show videos / queries.
| Model | 2D animation | 3D/stop-motion | Live action | Avg. |
|---|---|---|---|---|
| Large-scale reference models | ||||
| DeepSeek-V4-Pro ( DeepSeek-AI, 2026 ) | 86.50 | 85.59 | 86.10 | 86.06 |
| Gemini 3.1 Pro Preview ( Google DeepMind, 2026 ) | 82.12 | 81.99 | 78.42 | 80.84 |
| GPT-5 (2025-08-07) ( OpenAI, 2025 ) | 89.86 | 89.39 | 89.35 | 89.53 |
| Kimi K2.5 ( Kimi Team, 2026 ) | 85.46 | 84.42 | 82.61 | 84.16 |
| Qwen3.8-Max ( Qwen Team, 2026 ) | 91.05 | 91.15 | 91.02 | 91.07 |
Table 2: RARE improves over matched static baselines. Scores are query averages within each video type. Avg. gives equal weight to the three types. RL results average four seeds. Parentheses show gains over the corresponding static baseline at the same update level.
Figure 2: Replacements restore score differences. Panel (a) shows separation and resaturation. In (b), boxes show interquartile ranges, inner lines medians, and diamonds means. Dashed lines mark full credit. Appendix A.4 details the summaries.
| Metric | Negative | Mixed | Positive |
|---|---|---|---|
| Post-switch score | 0.82 | 0.84 | 0.86 |
| Opens headroom | 80.00% | 77.90% | 68.50% |
| Restores variance | 78.20% | 76.40% | 67.50% |
| Mean | 0.66 | 0.65 | 0.57 |
| Later resaturation | 24.50% | 36.30% | 34.10% |
Table 3: Negative replacements yield stronger signals. In (a), uses replacement scores only. In (b), standardizes each criterion separately. gives relative changes, except percentage points (pp) for the headroom rate.
Figure 3: A replacement exposes differences in causal coherence. Four of eight unchanged scripts show how shifting from element coverage to causal explanation changes scores.
Figure 4: Rubric revisions accompany quality gains. Panel (a) shows smoothed updates and evaluation scores. Panel (b) maps revision categories to evaluation dimensions.
Figure 5: Both judges favor Negative . Dynamic and Static denote Negative and Sample-Static.
| Model | BiGGen Bench (1 to 5) | WritingBench (1 to 10) |
|---|---|---|
| Base | 4.035 | 8.22 |
| Static | 4.152 | 8.26 |
| RARE ( Negative ) | 4.259 | 8.34 |
Table 4: RARE outperforms Static on both tasks. Scores are benchmark macro averages.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | Assessment focus |
|---|---|
| Query Fulfillment (QF) | |
| Requirement Coverage (Cov.) | Realizes the requested content and outcomes while satisfying explicit formal constraints. |
| Grounded Elaboration (Elab.) | Develops the query through relevant additions without contradicting its requirements or diluting the main storyline. |
| Tone and Style Alignment (Style) | Matches the requested genre, mood, rhythm, and visual style. |
| Narrative Coherence (NC) | |
| Causal and Temporal Logic (Logic) | Maintains plausible causal links and a consistent temporal sequence. |
Table 5: Twelve fixed metrics assess script quality. Abbreviations match the detailed result tables.
Figure 6: Initial criteria reach saturation throughout dynamic training. Curves show the cumulative fraction first awarding all eight rollouts full credit in each dynamic setting.
Figure 7: A task-level revision improves subsequent training. Curves compare branches trained for 20 steps from a shared checkpoint with fixed original or revised rubrics.
| Evaluation panel | Events | New mean | New headroom | New variance | Old full groups | Old mean | Old full rollouts |
|---|---|---|---|---|---|---|---|
| Trigger batch (eligible cohort) | 940 | – | 73.2% | 72.0% | 100.0% | 1.000 | 100.0% |
| End of epoch 1 | 940 | 0.906 | 53.4% | 52.6% | 56.0% | 0.943 | 86.6% |
| End of training | 940 | 0.931 | 43.4% | 43.0% | 66.3% | 0.957 | 90.4% |
Table 6: Revised criteria distinguish new scripts while earlier requirements remain largely satisfied. Bold marks larger values across checkpoints.
| Style | Checkpoint | Events | New mean | Headroom | Variance | Old full groups | Old mean | Old full rollouts |
|---|---|---|---|---|---|---|---|---|
| 2D animation | Epoch 1 (step 66) | 379 | 0.887 | 60.4% | 59.4% | 48.0% | 0.928 | 83.5% |
| Final (step 132) | 379 | 0.917 | 48.3% | 47.8% | 60.7% | 0.946 | 88.1% | |
| 3D and stop-motion | Epoch 1 (step 54) | 382 | 0.926 | 46.6% | 46.1% | 63.4% | 0.957 | 89.5% |
| Final (step 108) | 382 | 0.944 | 40.1% | 39.8% | 70.2% | 0.968 | 92.3% | |
| Live action | Epoch 1 (step 47) | 179 | 0.906 | 53.1% | 52.0% | 57.0% | 0.944 | 86.7% |
| Final (step 94) | 179 | 0.931 | 40.2% | 39.7% | 69.8% | 0.958 | 90.9% |
Table 7: Scores rise across all three production styles. Metrics use newly sampled scripts at each checkpoint. Bold marks larger values within each style.
| User query Could you write an imaginative fairy tale in which a chicken with exceptionally long legs proves very useful at night ? It should be cute, absurd, and a little moving. |
|---|
| Comparison Base presents the long legs through moonlit footsteps and a reflection. Negative uses them to solve the children’s problem. The chicken presses the toy box with its long leg, returning the toys and Xiaohe’s kite. This action fulfills the requested usefulness at night. |
| Base |
| Scene 2, the first appearance, night “As it walks, its toes lift the moonlight, as though writing a little unsigned poem.” |
| Moonlight covers the grass. The long-legged chicken slowly emerges from behind an old locust tree. Each step stretches its shadow across the ground, like wind moving over a canvas. The ends of its legs glow faintly, as if touched by stars. |
| Xiaohe peers out from indoors, her voice trembling. “You… you’re not a chicken?” |
| “Not a chicken, but a bearer of the night.” |
Table 11: The chicken’s long legs help recover missing toys. Selected scenes from a 2D fairy tale.
| User query I would like a beautiful but somewhat cruel nature fable about why people should not casually claim things that do not belong to them . |
|---|
| Comparison Base moves from a gentle encounter to a severed trunk, then has Lin Yuan acknowledge turning the tree into an exhibit. Negative depicts the intervention itself. Lin Ye’s fear of losing the tree motivates binding and hammering its roots, which shake as the soil collapses. The scene links possession to harm through the character’s actions. |
| Base |
| Scene 3, a pact beneath the canopy, afternoon “When I reached out to touch it, it reached back to hold my hand.” |
| The camera advances from Lin Yuan’s perspective. He stands in the forest as sunlight falls through the leaves, the canopy covering him like an umbrella. He slowly reaches out his right hand and touches the trunk with his fingertips. The bark’s patterns are as distinct as growth rings. The tree’s shadow moves toward him across the ground, as if responding. |
| His breathing slows. Lin Yuan says calmly, “So it does move. It was simply waiting for me, for someone willing to believe in it.” |
| Scene 4, a man-made fracture, night “They say a saw can save a tree. But what the saw cuts open is its memory.” |
Table 12: Possessiveness leads to visible harm. Selected scenes from a 2D nature fable.
| User query Write a fairy-tale animation without dialogue about a vulnerable creature and something that cannot speak keeping each other company in a dangerous environment . |
|---|
| Comparison Base uses speech and later a glowing warning signal. In Negative , the companion reshapes its body to shelter the beetle from a storm. Physical protection gives companionship a concrete function in the dangerous setting without character dialogue. |
| Base |
| Scene 1, meeting on the snowfield, dusk “I had never seen snow that could move. I thought it was merely a silent stone.” |
| Xiaoxue comes to a stop in the wind, its hind legs slightly bent and its ears twitching. Snow drifts from its nose and falls at its feet like fallen flowers. It looks up and sees a mound of snow shaped like a pillar broken by the wind. Moss covers its surface, and it sways slightly, as though waiting for Xiaoxue to approach. |
| Xiaoxue, softly, almost inaudibly, says, “Can you… hear the wind too?” |
| “The wind slips out of the rock crevice like a letter. It says that something is waiting for you.” |
Table 13: Protective movement conveys companionship. A request for dialogue-free animation.
| User query Create a small-animal story from a first-person perspective. It should begin with foraging and survival, then show a brief spark of affection making the animal willing to take risks or even sacrifice itself. |
|---|
| Comparison Both scripts express concern for a vulnerable animal. In Base , the storm prevents Graytail from acting on it. Negative connects an earlier moment of attachment to repeated attempts to reach Darktail despite unstable ice. Frostfur’s return after retreating and its eventual crossing give the requested risk-taking a visible progression. |
| Base |
| Scene 5, fleeing the sandstorm, noon “The wind is fire, a blade, the world’s anger.” |
| A violent wind rises. A sandstorm closes in from every direction, blurring earth and sky. Graytail scrambles about, its claws leaving deep marks in the sand. It looks back several times and sees the rabbit’s burrow covered in sand. The rabbit does not move. It wants to rush back but cannot take a step. The wind has blown it about like a dead leaf. |
| “I can’t… I can’t let it disappear,” I say, hoarsely and haltingly. |
| “If I die, it will never have my shadow.” |
Table 14: Concern motivates a dangerous crossing. A first-person animal story.
| User query Give me the opening of a monster short film in a stark, realistic style, preferably starting with a nighttime escape and a fight for survival . |
|---|
| Comparison Base shows an escape followed by a threatening appearance in the mirror. Negative turns the encounter into combat. Smoke briefly forces the monster back, and a later pipe strike makes it stagger before the wound closes. These reactions connect the protagonist’s attempts to survive with consequences for his opponent. |
| Base |
| Scene 1, fleeing through an alley, night “I have never seen it, but I know it is watching me. It does not move. It simply waits.” |
| Li Yuan slides through the narrow passage, his backpack hitting the iron wall with a dull thud. He lowers his head, sweeping his flashlight across the ground and its patterns of broken glass. He hears his heartbeat in time with breathing from a distant doorway. He keeps moving. An old poster on the wall to his right splits, revealing wet darkness behind it. |
| Li Yuan whispers, almost inaudibly, “It isn’t a living thing… it’s the weather.” |
| “I run. It follows.” |
Table 15: The protagonist fights back against the monster. A nighttime pursuit.
| User query Could you write a first-person romantic micro-script with a quiet protagonist who likes drawing and writing ? The story should feel cinematic. |
|---|
| Comparison Base moves from a brief glance to the narrator’s claim of being understood, with little response from the other character to support it. Negative shows Su Rui responding to a drawing and later to Lin Yu’s poem. Their relationship develops through these exchanges. |
| Base |
| Scene 3, the empty space by the window, day “You sit there longer than the sunset lasts.” |
| I return to the classroom and sit in my original seat. Sunlight falls on the windowsill. Chen Mo is looking down, gently touching his pencil to the draft paper. The lines are thin, as though tracing a breath. Looking down at the paper, I ask very softly, “Are you drawing the wind? Or those moments when you look up each time I hold my pen?” |
| I look up and see him raise his eyes, his gaze passing over me. Without a word, he sets his pencil on the corner of the desk and turns away. “I suddenly feel you are waiting rather than drawing. Waiting for me to put down my pen and pick it up again.” |
| Scene 4, the forgotten desk corner, night “I looked through seven sketchbooks and never found a second drawing. There was only a water-damaged page, with the figure’s hand reaching toward me.” |
Table 16: The relationship develops through drawings and poetry. A quiet romance.
| Condition | Instruction |
|---|---|
| Negative | Every update must use negative polarity. A response receives the maximum score when the target defect is absent. The score decreases progressively when the defect is partially present, and becomes zero when the defect is pronounced and harms the response quality. |
| Positive | Every update must use positive polarity. A response receives the maximum score when the target quality is fully realized. The score decreases progressively as the realization becomes weaker, and becomes zero when the quality is absent from the response. |
| Mixed | Each update may use either positive or negative polarity. If two or more updates are generated, the resulting set must contain both polarities. For either polarity, a higher score must always indicate higher response quality. |
Table 17: Polarity instructions for the shared prompt. Translated from Chinese.
More in this topic
Rubric-Based RL
Rubric-based reinforcement learning uses explicit, often task-specific criteria to reward model behavior beyond simple outcome checks.
See all 104 papers