Benchmark Design

Benchmark design studies how to turn a capability or real-world job into tasks, datasets, scoring rules, and protocols that support fair, reproducible comparisons. Common approaches include expert-curated and executable tests, domain-specific suites, hidden or refreshed items, controlled splits and stress tests, and multi-metric or rubric-based scoring. Researchers also audit validity, leakage, shortcuts, uncertainty, saturation, and ranking sensitivity, while increasingly evaluating full agent workflows under realistic tools, budgets, and deployment constraints. The central tension is making evaluations realistic and current without sacrificing transparency, reproducibility, and comparability; a high score alone rarely establishes general capability or readiness for use.