cs.AIMar 2, 2026
SaveRich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization
Organizations: University of Michigan · Google DeepMind
Abstract
Moving beyond evaluations that collapse performance across heterogeneous prompts toward fine-grained evaluation at the prompt level, or within relatively homogeneous subsets, is necessary to diagnose generative models' strengths and weaknesses. Such fine-grained evaluations, however, suffer from a data bottleneck: human gold-standard labels are too costly at this scale, while automated ratings are often misaligned with human judgment. To resolve this challenge, we propose a novel statistical model based on tensor factorization that merges cheap autorater data with a limited set of human gold-standard labels. Specifically, our approach uses autorater scores to pretrain latent representations of prompts and generative models, and then aligns those pretrained representations to human preferences using a small calibration set. This sample-efficient methodology is robust to autorater quality, more accurately predicts human preferences on a per-prompt basis than standard baselines, and provides tight confidence intervals for key statistical parameters of interest. We also showcase the practical utility of our method by constructing granular leaderboards based on prompt qualities and by estimating model performance solely from autorater scores, eliminating the need for additional human annotations.
Figures & tables
Figure 1: Test cross-entropy loss comparison between our proposed method (default and fine-tuned) and baseline approaches (Constant, Prompt-specific, and P2L) across three benchmarks: Gecko (left), BigGen Bench (center), and LMArena (right). Our methods consistently achieve lower losses for different human annotation budgets, demonstrating the benefits of prompt-specific modeling and auxiliary autorater data.
Figure 2: Category cohesion rankings for Gecko (left) and BigGen Bench (right). Categories are ordered from most to least cohesive based on the metric described in § 3.3 .
Figure 3: Model rankings with simultaneous confidence intervals for two Gecko categories: “Lang/Compositional” (left) and “Additive” (right), estimated using only of human annotations. The plots reveal performance discrepancies across different skills; for instance, Imagen matches SDXL in compositional tasks but underperforms in additive tasks.
Figure 4: Model rankings with simultaneous confidence intervals for two BigGen Bench tasks: “Multi-step” (left) and “Interplanetary Diplomacy” (right), estimated using a sample of human annotations.
Figure 5: Fine-grained comparison of model capabilities using of human annotations. Top: Difference in estimated scores between Imagen and Muse on Gecko prompts, colored by category. Imagen excels in text rendering, while Muse shows advantages in object counting. Bottom: Difference between LLaMa-2-13b and GPT-3.5-Turbo on BigGen Bench prompts. GPT-3.5-Turbo demonstrates a significant advantage in reasoning-related prompts.
Figure 6: In LMArena, we estimate that LLaMa-3.3-70b is capable of beating Gemini-2.5-Pro in and tying other of the times.
Figure 7: Validation of predictive power on held-out models. The plots compare the ground truth average scores (or win-rate differences) against the values predicted by our method for a model whose human annotations were entirely withheld during training. Results are shown for Gecko (left), BigGen Bench (center), and LMArena (right), indicating strong correlation and preservation of pairwise rankings.
Figure 8: Conditional distributions of prompt length for SD1.5 on the Gecko benchmark, comparing densities when it achieves its best versus worst rank. The shift shows SD1.5 performs relatively better on shorter prompts.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Task | p-val |
|---|---|---|
| Gecko - Cohesive | lang/compositional:compositional-3E-2A | 0.000 |
| rel/action:action_rev/3 | 0.001 | |
| rel/scale:same_size | 0.063 | |
| att/count:additive_3 | 0.001 | |
| att/color:abstract | 0.028 | |
| Gecko - Non-Cohesive | ne/landmarks:landmarks | 0.194 |
Table 1: P-values for Gecko and BGB across Cohesive and Non-Cohesive tasks
| Category | Task | # prompts |
|---|---|---|
| Gecko - Cohesive | lang/compositional:compositional-3E-2A | 10 |
| rel/action:action_rev/3 | 8 | |
| rel/scale:same_size | 6 | |
| att/count:additive_3 | 11 | |
| att/color:abstract | 8 | |
| Gecko - Non-Cohesive | ne/landmarks:landmarks | 100 |
Table 2: Number of Prompts for Gecko and BGB across Cohesive and Non-Cohesive tasks
| Variable | Coef. | Std.Err. | t | P |
|---|---|---|---|---|
| const | -0.7713 | 0.037 | -20.818 | 0.000 |
| category_tag_criteria_v0_1_specificity | 0.1631 | 0.040 | 4.103 | 0.000 |
| category_tag_criteria_v0_1_domain_knowledge | -0.1058 | 0.044 | -2.427 | 0.015 |
| category_tag_criteria_v0_1_complexity | -0.0100 | 0.035 | -0.283 | 0.777 |
| category_tag_criteria_v0_1_technical_accuracy | 0.0032 | 0.039 | 0.082 | 0.935 |
| category_tag_criteria_v0_1_real_world | 0.0001 | 0.039 | 0.003 | 0.997 |
Table 3: Explains the gap between LLaMa-3.3-70b and Gemini-2.5-Pro (OLS)
Figure 9: First stage losses.
Figure 10: Varying the rank and assessing impact on predictive performance. We keep of the dataset for training and the rest for testing.
Figure 11: We compare our approaches with the constant baseline and a logistic regression model trained on the outputs of individual autoraters. When the autorater evaluation template matches the human template, we simply average the realizations for the of interest and use this average as a feature for the logistic model. Conversely, when autoraters use single-sided templates while humans perform side-by-side comparisons, we average the realizations of for each of the two models being evaluated and take the difference between these averages to obtain the features for the logistic regression.
Figure 12: When varying the fraction of autoraters used, we observe that the highest gains occur at the lowest fractions. This suggests that the collective signal from autoraters may begin to saturate if sufficient diversity is not provided. In this specific experiment, we fix the fraction and sample multiple combinations of judges to satisfy that proportion; the error bars represent the variability across these combinations and different data splits. We allocate of the dataset for training and use the remainder for testing.
Figure 13: This is a version Figure 3 with more categories.
Figure 14: This is a version Figure 4 with more categories.
Figure 15: This plot illustrates how the estimates improve when we scale the number of human labels, in contrast to Figure 3 , which utilizes only . Overall, we observe that the model ordering is maintained, though the confidence intervals are evidently narrower. In practice, a practitioner could calibrate the required number of human labels by running a preliminary study to estimate the sample size needed to achieve a specific level of precision (knowing that the interval length is proportional to , where is the number of human annotations); this type of approach is commonly employed in Monte Carlo simulation for integral estimation.
Figure 16: This plot illustrates how the estimates improve when we scale the number of human labels, in contrast to Figure 4 . As for Gecko, we observe that the model ordering is roughly maintained, though the confidence intervals are evidently narrower.
Figure 17: Fine-grained comparison of model capabilities using of human annotations. Top: Difference in estimated scores between Imagen and Muse on Gecko prompts, colored by category. Imagen excels in text rendering, while Muse shows advantages in object counting. Bottom: Difference between LLaMa-2-13b and GPT-3.5-Turbo on BigGen Bench prompts. GPT-3.5-Turbo demonstrates a significant advantage in reasoning-related prompts.
More in this topic
Generative AI Evaluation
Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.
See all 309 papers