Benchmark Prediction with Sparse and Unpaired Cached Responses
Organizations: Helivan · Hugging Face · Jataware · Johns Hopkins University
Abstract
Evaluating a generative model on all queries from all available benchmark tasks is prohibitively expensive. As such, models are often evaluated on different subsets of tasks and/or different subsets of queries from a given task. The resulting collection of responses and the corresponding (model, task) score matrix are sparse and unpaired. In this paper, we introduce the Product Kernel Perspective Space (PKPS), a collection of low-dimensional representations of models based on their embedded responses. PKPS generalizes existing methods for representing black-box generative models by comparing responses to similar -- rather than just identical -- queries and is therefore able to utilize sparse and unpaired information. Theoretically, we show that benchmark prediction using PKPS is query-efficient relative to methods that only use information from dense and paired portions of the evaluation data. Empirically, we show that PKPS-based methods outperform baselines on benchmark prediction across two large benchmark suites.
Figures & tables
| HELM suite ( tasks, models) | EEE suite ( tasks, models) | |||||||||
| Query-eff. | Completion | Query-eff. | Completion | |||||||
| Method | ( ) | ( ) | ( ) | ( ) | (sp.) | ( ) | ( ) | ( ) | ( ) | (sp.) |
| Sample score | — | — | — | — | — | — | ||||
| IRT | — | — | — | — | — | — | ||||
| LRMC | — | — | — | — | ||||||
| DKPS | — | — | — | — | — | — | ||||
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Task | Queries | Dataset | Task | Queries |
|---|---|---|---|---|---|
| MATH | algebra | 135 | WMT-14 | cs–en | 873 |
| MATH | counting & probability | 39 | WMT-14 | de–en | 776 |
| MATH | geometry | 38 | WMT-14 | fr–en | 754 |
| MATH | intermediate algebra | 52 | WMT-14 | hi–en | 387 |
| MATH | number theory | 30 | WMT-14 | ru–en | 613 |
| MATH | prealgebra | 86 | MedQA | med_qa | 1000 |
| Benchmark | Task | Items | Benchmark | Task | Items |
|---|---|---|---|---|---|
| MATH-MC | level 1 | 430 | JudgeBench | coding | 42 |
| MATH-MC | level 2 | 882 | JudgeBench | knowledge | 154 |
| MATH-MC | level 3 | 1115 | JudgeBench | math | 56 |
| MATH-MC | level 4 | 1199 | JudgeBench | reasoning | 98 |
| MATH-MC | level 5 | 1288 | RewardBench 2 | factuality | 475 |
| GSM-MC | gsm_mc | 1319 | RewardBench 2 | focus | 495 |
Generative AI Evaluation
Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.
See all 335 papers