Rubric-Guided Generation

Follow this topic Ask about this topic

Rubric-guided generation studies how to turn a request into explicit criteria, such as coverage, factual support, or style, and use them to assess outputs and guide models toward better ones. Methods include preference- and expert-grounded rubric induction, retrieval and consensus, and adaptive or adversarial refinement; criteria can then serve as rewards for reinforcement learning, step-level feedback, or token-level distillation. Applications range from research reports and medical answers to reasoning, essays, code, visual design, and music. Rubrics can provide finer learning signals than scalar rewards, but automatically generated criteria may still miss expert standards, be misweighted, or be gamed, making reliability, calibration, and efficient validation central challenges.

Latest papers

42 papers7 papers in the last four weeks, up 75% on the four weeks before. 0.0% of all new papers.

When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation
May 19, 2026Jinlong Liu, Mohammed Bahja, Mark LeeSupervised Fine-Tuning Creativity Assessment
School of Computer Science, University of Birmingham, United Kingdom
Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
May 19, 2026Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5Reward Modeling Rubric-Based RL
Scale AI · Persona
Step-wise Rubric Rewards for LLM Reasoning
May 17, 2026Weichu Xie, Haozhe Zhao, Wenpu Liu +15Reward Modeling Credit Assignment in RL
Peking University · JD Explore Academy · Shanghai Jiao Tong University +2
RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards
May 11, 2026Gaotang Li, Bhavana Dalvi Mishra, Zifeng Wang +9Reinforcement Learning Rubric-Based RL
University of Illinois Urbana-Champaign · Google Cloud AI Research
Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance
May 8, 2026Jiachen Yu, Zhihao Xu, Junjie Wang +1Instruction Following Rubric-Based Evaluation
1Tsinghua University · 2Renmin University of China
RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation
Mar 21, 2026Kaustubh D. Dhole, Eugene AgichteinAutomatic Rubric Generation LLM Evaluation
Department of Computer Science, Emory University, Atlanta, GA, USA
Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models
Mar 17, 2026Weijie Qiu, Dai Guan, Junxin Wang +6Multimodal Reward Modeling VLM Evaluation
Qwen Large Model Application Team, Alibaba · Institute of Automation, Chinese Academy of Sciences
CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling
Mar 9, 2026Dengcan Liu, Fengkai Yang, Xiaohan Wang +7Preference Modeling Reward Modeling
University of Science and Technology of China · Peking University · Meituan +1
Using Learning Progressions to Guide AI Feedback for Science Learning
Mar 3, 2026Xin Xia, Nejla Yuruk, Yun Wang +1Educational Feedback Generation Generative AI in Education
University of Georgia · University of Georgia, Athens GA 30605, USA · Gazi University +1
RubricBench: Aligning Model-Generated Rubrics with Human Standards
Mar 2, 2026Qiyuan Zhang, Junyi Zhou, Yufei Wang +8LLM Evaluation Rubric-Based Evaluation
City University of Hong Kong · Tencent Hunyuan · McGill - Mila & Quebec AI Institute +2
SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probing
Feb 24, 2026Yifei Xu, Guilherme Potje, Shivam Shandilya +9Rubric-Based RL Reward Design for RL
Microsoft · University of California, Los Angeles
Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation
Feb 3, 2026Changze Lv, Jie Zhou, Wentao Zhao +14Rubric-Guided Generation Deep Research Agents
Tencent · Fudan University · Tsinghua University

May 19, 2026J/K move · Enter open · S save

When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation

Jinlong Liu, Mohammed Bahja, Mark Lee

School of Computer Science, University of Birmingham, United Kingdom

Supervised Fine-Tuning Creativity Assessment LLM Evaluation Rubric-Guided Generation Text Generation Generative AI Evaluation

Open paper AskPDF arXiv

Automatic evaluation of long-form literary writing remains challenging, particularly for creativity-related dimensions such as originality and flexibility. We address the lack of large-scale supervision for structured Torrance Test of Creative Writing (TTCW)-based evaluation by constructing a dataset of 263{,}911 long-form stories, each paired with scalar scores and meta-synthesised review comments across 14 TTCW-based dimensions. Using this dataset, we study whether reviewer-style reasoning traces improve structured literary review generation across Qwen3-4B, Qwen3-8B, and Nemotron-8B. Across all three models, no-trace supervision yields higher score imitation fidelity and near-complete report generation, whereas trace supervision reduces completion to

85.9

90.9\%

. Failure analysis shows that

88.6\%

of failed reasoning generations contain semantic repetition loops inside the trace. These loops concentrate in particular story--criterion interactions and progressively harden after onset, while comparable repetition is rare in the supervision data. Our results suggest that, for long-form tasks requiring strictly structured outputs, explicit reasoning traces can introduce generation instability that outweighs their overall benefit.