Rubric-Based Evaluation

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 64% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 136

All topics

RAVEN-Eval: Rubric-Guided Automatic Evaluation for AI Video Generation Models Based on LMM Preference Judgement
Aug 10, 2026Ziheng Jia, Jiaying Qian, Zicheng Zhang +3LLM-as-a-Judge Rubric-Based Evaluation
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing
Aug 6, 2026Ziyun Zeng, Zixuan Wang, Yongsheng Yu +2Video Editing Benchmarks Video Editing
University of Rochester
PRISM: Priority-aware Rubric Internalization via Structured Multimodal Data Synthesis
Aug 5, 2026Xiaomin He, Dongling Xiao, Jiahao Xie +4Multimodal Generation Rubric-Based Evaluation
ByteDance · Peking University
AudioScape-TTA: A Structured Soundscape Benchmark for Fine-Grained Text-to-Audio Evaluation
Aug 5, 2026Jinting Wang, Yuguang Yang, Shengyu Li +4Text-to-Audio Generation Rubric-Based Evaluation
1The Hong Kong University of Science and Technology (Guangzhou) · 2Tencent · 3Zhejiang University
FinReportBench: Measuring and Improving Institution-Grade Financial Report Generation
Aug 5, 2026Yinghao Tang, Tan Zhenwei, Yiyao Wang +4Financial Services LLM Evaluation
State Key Laboratory of CAD&CG, Zhejiang University · Ant Group
FinProBench: Evaluating Financial AI Agents with Role-Grounded Rubrics Derived from Professional Deliverables
Aug 4, 2026Ben Wang, Kang Zhou, Lifan Guo +2AI Agent Evaluation AI Agent Benchmarks
Qwen DianJin Team, Alibaba Cloud Computing
Rubrics as Privileged Information for Open-Ended Generation
Aug 3, 2026Deepika Bablani, Ajay Gupta, Wanming ChenOpen-Ended Generation Rubric-Based Evaluation
Apple
RADAR: Rubric-Aware Dependency and Redundancy Analysis for LLM-as-Judge Evaluation
Aug 3, 2026Divyansh Singh, Reza Davari, Afra MashhadiLLM-as-a-Judge LLM Auditing
Microsoft · University of Florida · University of Washington
Comparative Validation of GPT-4o-mini and Teacher Mean Scores for Automated Scoring of Music Analysis Responses: Single-Pass Deployment, Repeatability, and Strategy-Specific Bias
Aug 3, 2026Baicheng Lin, Lingxi Jin, Kyung-Seok MinLLM-as-a-Judge Educational Technology
Department of Education, Sejong University, South Korea · Department of Educational Technology, Ewha Womans University, South Korea
AI-Based Thesis Assessment: An Empirical Study of Human Evaluation Priorities and Their Impact on Automated Assessment
Aug 1, 2026Garv Vikram Gursahaney, Baskhad Idrisov, Thorsten Fröhlich +1Rubric-Based Evaluation Automated Grading
IU International University of Applied Sciences, Germany.
CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
Jul 31, 2026Mengting Chen, Yanshu Sun, Wanting Liang +5Open-Ended Generation LLM Evaluation
FinStep · StepFun
TRACE Bench: Task-driven Roleplay Agentic Checklist Evaluation
Jul 31, 2026Jiahui Zhang, Ziwei Zhang, Yipeng Wang +7Large Language Model-Based Role-Play Simulation Agent Evaluation
Kuaishou GameMind Lab
An Instrument to Evaluate Governance Proposals: AI Policy Analysis at Scale
Jul 30, 2026Paulo Carvao, Claudio Mayrink Verdun, Isabel Adler +1LLM Evaluation Policy Evaluation
Mossavar-Rahmani Center for Business and Government · Harvard Kennedy School · Harvard Kennedy School of Government +2
Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation
Jul 30, 2026Weining ZhangLLM Evaluation LLM-as-a-Judge
Cheung Kong Graduate School of Business
Capital Markets LLM Reliability Score (CM-LRS): From Plausible to Bankable
Jul 23, 2026Prerit AhujaFinancial Services LLM Evaluation
Independent Researcher · London-based
Co-Evolving LLM Evaluators and Policies via DynamicRubric
Jul 22, 2026Beining Wang, Weihang Su, Hongtao Tian +8RL for Language Models LLM Evaluation
Department of Computer Science and Technology, Tsinghua University · 2WeChat, Tencent
Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking
Jul 22, 2026Kailin Jiang, Lei Liu, Jian Xi +7Learning to Rank Rubric-Based Evaluation
University of Science and Technology of China · Yuanbao Team, Tencent · University of Chinese Academy of Sciences +1
Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness
Jul 21, 2026Xilun Chen, Zhaleh Feizollahi, Ross Goodwin +5Open-Ended Generation LLM-as-a-Judge
Meta AI
Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards
Jul 21, 2026Xuefeng Jin, Jiashuo Zhang, Teng Cao +1Automated Essay Scoring Rubric-Based RL
Alibaba Cloud Computing, Alibaba Group
Assessment in Team Problem-Solving Exercises in Computing Education
Jul 21, 2026Valdemar Švábenský, Jan Vykopal, Sukrit Leelaluk +3Educational Assessment Rubric-Based Evaluation
Faculty of Informatics Masaryk University Brno, Czech Republic · Center for ICT Infrastructure Yamaguchi University Yamaguchi, Japan · Graduate School of Information Science and Electrical Engineering, Kyushu University Fukuoka, Japan
EduPanel: A Three-Agent LLM Judge for Teaching Videos -- Reliability, Complementarity, and Human Trust Calibration
Jul 20, 2026Jia-Kai Dong, Yi-Cheng Lin, Hung-yi LeeLLM-as-a-Judge Educational Assessment
National Taiwan University · NTU Artificial Intelligence Center of Research Excellence
AgentFAIR: A Multi-Agent Collaborative Framework for FAIRness Evaluation of Geospatial Datasets
Jul 17, 2026Ming Chen, Pranav PaiMulti-Agent Systems Rubric-Based Evaluation
The University of Melbourne Melbourne, Victoria, Australia
Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence
Jul 16, 2026Haocheng Yang, Licheng Pan, Yuan Lu +7Rubric-Based Evaluation
School of Computing, National University of Singapore · Xiaohongshu Inc. · School of Cyber Science and Technology, Zhejiang University +3
Project Kaleidoscope: Contextual, Human-Aligned Evaluation for Real-World AI Applications
Jul 16, 2026Leanne Tan, Rohan Jaggi, Shaun Khoo +1Human-in-the-Loop Evaluation Rubric-Based Evaluation
GovTech, Singapore · National University of Singapore · University of British Columbia
When Rubrics Change: Cross-Rubric Generalization for Critical Thinking Essay Scoring
Jul 15, 2026Nischal Ashok Kumar, Payu Wittawatolarn, Sana Kang +7Educational Assessment Automated Essay Scoring
University of Massachusetts Amherst, United States · University of Minnesota, United States · Brighter Research, United States +3
Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction
Jul 14, 2026Hanhua Hong, Yizhi Li, Jiaoyan Chen +4LLM Evaluation Rubric-Based Evaluation
The University of Manchester · IQuest Research · University of Information Technology, VNU +2
Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents
Jul 14, 2026Xing Zhang, Guanghui Wang, Yanwei Cui +4LLM Agent Evaluation LLM Agent Self-Improvement
1AWS Generative AI Innovation Center · 2HSBC Holdings Plc., HSBC Technology Center, China
FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality
Jul 14, 2026Beidi Luan, Rui Sun, Sinuo Wang +5LLM-as-a-Judge Rubric-Based Evaluation
1StepFun · University of Adelaide · 2FinStep +1
Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution
Jul 9, 2026Ethan Leung, Elias Lumer, Corey Feld +3LLM-as-a-Judge Citation Verification
Commercial Technology and Innovation Office, PricewaterhouseCoopers, U.S.
Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach
Jul 2, 2026Manuel Alonso-Carracedo, Ruben Fernandez-Boullon, Pedro Celard +2LLM-as-a-Judge Computer Science Education
aUniversidade de Vigo, Department of Computer Science, ESEI-Higher School of Computer Engineering, Edificio Politécnico, Campus Universitario As Lagoas s/n, Ourense, 32004, Spain · bIFCAE-Institute for Research in Physics, Computing and Aerospace Science, Universidade de Vigo, Ourense, 32004, Spain

Aug 10, 2026, cs.AIJ/K move · Enter open · S save

RAVEN-Eval: Rubric-Guided Automatic Evaluation for AI Video Generation Models Based on LMM Preference Judgement

Ziheng Jia, Jiaying Qian, Zicheng Zhang, Xiaorong Zhu+2

Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory

LLM-as-a-Judge Rubric-Based Evaluation Automated Evaluation Video Generation Evaluation

Open paper AskPDF arXiv

AI video generation has advanced rapidly and entered widespread commercial use. As a result, quality differences among videos produced by state-of-the-art AI video generation models~(AIVGMs) have become increasingly difficult to discern using conventional evaluation criteria, such as visual fidelity and semantic instruction following. Meanwhile, human evaluation now requires more expertise and sustained attention, substantially increasing annotation costs. This calls for automated evaluation that can reliably distinguish fine-grained differences among advanced AIVGMs with minimal human intervention. To address this challenge, we present RAVEN-Eval, a rubric-guided automated evaluation framework for AIVGMs, built primarily on the LMM-as-a-judge paradigm. Through an automatic task curation and quality-filtering pipeline, RAVEN-Eval curates 150 text-to-video~(T2V) tasks and 100 image-to-video~(I2V) tasks, and systematically collects more than 4,500 AIGVs. At its core, RAVEN-Eval adopts rubric-guided automated LMM preference judgement, in which LMM judges conduct pairwise comparisons according to task-specific rubrics. It further introduces an anchor-based model insertion approach to reduce the evaluation cost of incorporating new models. Finally, we evaluate 20 high-performance AIVGMs, as well as the judging capabilities of 13 LMM judges, and establish the RAVEN-Eval Leaderboards. Overall, RAVEN-Eval paves a scalable path for automatic and trustworthy evaluation of rapidly evolving AIVGMs.

Momentum

Related topics

Latest papers 136