cs.LGNov 2, 2026

Benchmark Prediction with Sparse and Unpaired Cached Responses

Authors: Hayden Helm, Avijit Ghosh, Ben Johnson, Carey Priebe

Organizations: Helivan · Hugging Face · Jataware · Johns Hopkins University

Abstract

Evaluating a generative model on all queries from all available benchmark tasks is prohibitively expensive. As such, models are often evaluated on different subsets of tasks and/or different subsets of queries from a given task. The resulting collection of responses and the corresponding (model, task) score matrix are sparse and unpaired. In this paper, we introduce the Product Kernel Perspective Space (PKPS), a collection of low-dimensional representations of models based on their embedded responses. PKPS generalizes existing methods for representing black-box generative models by comparing responses to similar -- rather than just identical -- queries and is therefore able to utilize sparse and unpaired information. Theoretically, we show that benchmark prediction using PKPS is query-efficient relative to methods that only use information from dense and paired portions of the evaluation data. Empirically, we show that PKPS-based methods outperform baselines on benchmark prediction across two large benchmark suites.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Query-efficient model evaluation using cached responses

    May 8, 2026Hayden Helm, Ben Johnson, Carey PriebeKernel Methods

  2. Efficient Benchmarking Is Just Feature Selection and Multiple Regression

    May 25, 2026Sam Bowyer, Acyr Locatelli, Kris CaoLLM EvaluationFeature Selection

  3. You Don't Need to Run Every Eval

    Jun 22, 2026Yuchen Zeng, Dimitris PapailiopoulosLLM EvaluationRobust Matrix Completion