【论文速递】Benchmark Prediction with Sparse and Unpaired Cached Responses

英文标题：Benchmark Prediction with Sparse and Unpaired Cached Responses

arXiv编号：2611.03614
作者：Hayden Helm, Avijit Ghosh, Ben Johnson
来源：arXiv
原文链接：https://arxiv.org/abs/2611.03614

中文摘要：
Evaluating a generative model on all queries from all available benchmark tasks is prohibitively expensive. As such, models are often evaluated on different subsets of tasks and/or different subsets of queries from a given task. The resulting collection of responses and the corresponding (model, task) score matrix are sparse and unpaired. In this paper, we introduce the Product Kernel Perspective Space (PKPS), a collection of low-dimensional representations of models based on their embedded responses. PKPS generalizes existing methods for representing black-box generative models by comparing responses to similar -- rather than just identical -- queries and is therefore able to utilize sparse and unpaired information. Theoretically, we show that benchmark prediction using PKPS is query-efficient relative to methods that only use information from dense and paired portions of the evaluation data. Empirically, we show that PKPS-based methods outperform baselines on benchmark prediction across two large benchmark suites.

本文由华人空间 AI 论文助手自动抓取翻译，仅供学术交流。