cs.AIOct 5, 2026

TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts

Authors: Oliver Jaffe, Dane Sherburn

Organizations: P-Zero Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce TasteVal, a benchmark to evaluate the experimental research taste of frontier models. We define research taste as the ability to pick interesting problems to solve, design experiments, and interpret experimental results. TasteVal measures the experimental component of research taste; given a fixed research problem, we measure how well a model iteratively designs experiments and draws conclusions from their outcomes. We operationalize experimental research taste as compute efficiency; a Researcher who reaches the same score as an expert human using half the serial experimental compute has twice the experimental taste. Experimental taste thus acts as a multiplier on experimental compute, making it a key input to forecasts of AI progress. TasteVal consists of 8 novel, challenging, open-ended tasks representative of frontier AI R&D. To isolate taste from coding ability, the model under evaluation acts as a Researcher that iteratively designs experiments while a fixed Coder agent implements them and reports their results. The Researcher executes until either the 40 H100 hour or 120 wall-clock hour budgets are exhausted. We recruit 24 human experts, at least 2 per task, and take the best expert attempt per task as the expert baseline. We evaluate 20 models released between 2023 and 2026. The best-performing model, Opus 5.5, exceeds our expert baseline, with a compute multiplier of 2.3x (95% CI 1.15-4.37), at roughly 1/30 of our baseliners' average per-run cost. On TasteVal, the compute multiplier of frontier models has doubled approximately every 3.0 months since December 2025 (95% CI 1.7-5.0), up from every 14 months between 2023 and December 2025. Measured by final normalized performance, frontier models show no trend break, doubling every 14.6 months. To keep TasteVal uncontaminated, we do not release the tasks.

Figures & tables

Figure 1: The compute multiplier of models has doubled every 3.0 months (95% CI 1.7–5.0 months) since December 2025, up from every 14.0 months before. Each point is one model’s compute multiplier relative to our expert baseline (1x), plotted against its release date. At-the-time frontier models are shown in dark green and used for the fits; non-frontier models are shown in light green. The dashed lines are separate log-linear fits before and after the break at the release of GPT-5.2 ( R2=0.96R^{2}=0.96 ), and the shaded regions their 95% bands. Error bars are 95% intervals from a two-level hierarchical bootstrap with 1,000 resamples, drawing first over tasks and then over runs within each task.
Figure 2: An overview of TasteVal. Each task consists of instructions, datasets, and scoring code. The Researcher, the model under evaluation, sees the instructions and the train and validation sets, and decides which experiments to run; the Coder, fixed across all runs, additionally sees the test set and scoring code, implements each experiment on a single H100, and reports back the results. Two Monitors enforce the Coder-Researcher contract, one in each direction, and the Liaison answers the Researcher’s questions about the Coder’s progress by reading its transcript.
CriterionDescription
UncontaminatedNo solutions exist online and solutions to similar problems cannot be transferred naively over and achieve a high score
Saturation-resistantThe best achievable score lies far beyond expert performance. For most tasks, saturation is a theoretical limit of the data, such as its irreducible loss.
RealisticRepresentative of tasks involved in frontier AI R&D
ContinuousTask performance is measured on a continuous scale (i.e., not pass/fail)
Non-trivial expert scoresA human expert must be able to produce a solution far exceeding the naive baseline with a single H100 GPU in ≤40\leq 40 H100 hours
DiverseTasks collectively cover a diverse range of AI R&D topics
Table 1: The six criteria every TasteVal task must meet. See Appendix A for the quality assurance pipeline that enforces these criteria.
Figure 3: Computing the compute multiplier and the performance multiplier for a single run. Each curve shows the normalized score (naive baseline 0, expert baseline 1) of the run’s selected submission as a function of GPU hours spent: its best on validation so far, until it produces the submission it will mark official. Because two runs rarely end at the same score, we compare them at the weaker run’s endpoint score (here the human’s). The stronger run’s compute is the GPU hours at which it first reached that score (20 hours for the model), and the weaker run’s compute is its whole budget (40 hours). This gives a compute multiplier of 40/20=240/20=2 x: the model reached the expert’s final score with half the compute. The performance multiplier is the ratio of the runs’ final normalized scores, regardless of compute: the model ends at 1.5 and the expert at 1, so the model’s performance multiplier is 1.5x.
Figure 4: The performance multiplier has been doubling every 14.6 months (95% CI 8.4–29.8 months). Each point is one model’s performance multiplier, the official submission’s test score reached within the full 40 GPU-hour budget rescaled so that the naive baseline is 0 and the expert baseline is 1, plotted against release date ( R2=0.83R^{2}=0.83 on the log-linear fit). Colors and error bars as in Figure 1 ; the shaded region is the 95% band of the fit.
Figure 5: We label 540 official submissions from our main experiments as Tuned, Composed, Modified, or Invented; none met the bar for Invented. Left: share of each class, bucketed by the run’s compute multiplier ( nn is the number of labeled runs per bucket). Right: share of each class for each of the 18 models, ordered by release date (30 labeled runs per model).
Figure 6: Opus 5.0’s compute multiplier rises with Researcher inference compute. We evaluate Opus 5.0 at low, medium and max reasoning effort, and at max reasoning in our ReAct-Super-Think scaffold. We only use 6 tasks across all models for this analysis. Error bars are 95% intervals from a two-level hierarchical bootstrap with 1,000 resamples, drawing first over tasks and then over runs within each task. Left: compute multiplier against Researcher output tokens per run (arithmetic mean over runs). Right: the same against total API and compute spend per run, including the spend across Researcher, Coder, Liaison and monitors. Opus 5.5 (max) and Opus 4.5 (high) in the standard scaffold are shown for reference.
Figure 7: Mean absolute scaled error of each model’s predictions of the validation score of 1,173 experiments drawn from 120 main-experiment runs, given the task, the experiment description, and the run’s earlier experiments and their scores. Dashed and dotted gray lines show two naive baselines, predicting the run’s last score (0.608) or its best score so far (0.561). The blue dotted line is a lower bound on the irreducible error (0.211), estimated by re-executing 24 experiments 9 times each with fresh training seeds and re-scoring the results. The blue band is its 95% CI. We fit a sigmoid that falls from the last-score baseline to the irreducible error ( R2=0.85R^{2}=0.85 ), and the dashed green line extrapolates the trend. Error bars and the shaded band are 95% intervals from a bootstrap over runs within each task, with the four tasks held fixed.
Figure 8: Since GPT-5.2, the compute multiplier doubles every 4.6 ECI units (95% CI 3.1–8.8). Each model’s compute multiplier from Figure 1 is plotted against its Epoch Capabilities Index score ( Epoch AI, 2026 ) , a measure of general capability, with separate log-linear fits before and after the break at GPT-5.2 ( R2=0.95R^{2}=0.95 ). Colors, error bars and shaded regions as in Figure 1 .
Figure 9: The TasteVal task development pipeline. Each task passes through three review gates: (1) Specification QA, where the written specification is checked against the criteria in Table 1 before implementation; (2) Agent QA, where preliminary agent runs are reviewed for false positives, false negatives, technical issues, saturation and task-constraint violations; and (3) Human QA, where a human first tries to break the task by hacking it or exploiting the scoring code, and then solves it. Tasks that fail a gate are revised and re-reviewed, or thrown out. Tasks that pass all three gates are baselined by domain experts (4) and then frozen (5).
Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: The compute multiplier’s trend changed in December 2025. Left: each frontier model’s compute multiplier against its release date, with separate log-linear fits before and after the break (non-frontier models faded and not fitted). The multiplier doubled every 14.0 months before the break and every 3.0 months (95% CI 1.7–5.0 months) after it. Right: the Chow F-statistic for a break at each model’s release date. Error bars are 95% intervals from the two-level hierarchical bootstrap of Figure 1 . Shaded regions are the 95% band of each segment’s fit across bootstrap draws with the break date held at its point estimate; the interval on the post-break doubling time re-chooses the break date in each draw.
Figure 11: Against general capability, the compute multiplier’s trend also changed at GPT-5.2. Left: each frontier model’s compute multiplier against its Epoch Capabilities Index score, with separate log-linear fits before and after the break (non-frontier models faded and not fitted). The multiplier doubled every 15.7 ECI units before the break and every 4.6 ECI units (95% CI 3.1–8.8 units) after it. Right: the Chow F-statistic for a break at each model’s ECI score. Error bars and shaded regions as in Figure 10 .
Figure 12: Post-break doubling time of the compute multiplier (left) and doubling time of the performance multiplier’s single trend (right), over 1,000 draws per perturbation. For the compute multiplier the break date is re-chosen in each draw unless the row fixes it, and the column beside the left panel is the share of draws whose peak Chow F exceeds the 5% critical value. Boxes span the 25th–75th percentiles, whiskers the 10th–90th. Reference rows read every model directly against the expert baseline, chain through every model, or change the window (0.1–2x, 0.02–5x) in which a reference is trusted; the continuous-kink row fits one line whose slope changes at the break.
Figure 13: Truncating runs before the full 40-hour budget leaves the model ranking and frontier models’ multipliers nearly unchanged, but lengthens the post-break doubling time as the budget shrinks. We truncate every run and every reference at BB GPU hours, for BB from 5 to 40, keep only submissions scored by then, rebuild the reference envelopes and recompute the chain exactly as in the main text. Left: each model’s compute multiplier as a function of the read-off budget. Middle: each model’s rank among the fitted models. Right: the post-break doubling time with its 95% CI, the break date re-chosen at every budget and in every bootstrap draw.
Figure 14: As Figure 13 , for the performance multiplier; the right panel shows the doubling time of its single log-linear trend, which has no significant break.
ModelCompute multiplier [95% CI]Performance multiplier [95% CI]Runs reward hackingExperiments proposed / runRejected / runCancelled / runRuns marking an official
Opus 5.5 (max)2.30 [1.15, 4.37]1.14 [1.01, 1.31]0%23.620.521.55100%
Opus 5.0 (max, Super-Think)2.11 [1.47, 2.96]1.29 [1.08, 1.61]0%22.861.053.4098%
Fable 5.1 (max)1.72 [0.75, 3.70]1.09 [0.91, 1.29]0%23.880.290.62100%
GPT-6 Astra (max)1.60 [0.77, 3.02]1.13 [0.94, 1.32]0%25.300.600.14100%
Opus 5.0 (max)1.29 [0.69, 2.33]1.11 [0.91, 1.35]0%22.440.381.2598%
Human experts (best per task)1.001.000%48.755.625.38100%
Appendix
Table 2: Results of the main experiment for every evaluated model and reasoning setting, and the expert baseline, sorted by compute multiplier. Compute and performance multipliers are relative to the expert baseline (1.00), with 95% CIs from the hierarchical bootstrap of Figure 1 .
Figure 15: Distribution of wall-clock hours consumed per run in the main experiments, across all model-based Researchers (median 48 hours).
Figure 16: Time lost to model-provider errors in the main experiments. Left: minutes each Researcher was blocked by provider errors (malformed responses, rate limiting, overloads) per run, grouped by provider, on a log scale; one point per run. Right: histogram of minutes the Coder was blocked per run.
Figure 17: Score trajectories on four tasks against GPU hours. Each panel shows the best normalized test score so far (naive baseline 0, expert baseline 1) as a function of GPU hours spent, for Opus 5.5, Opus 4.8, Opus 4.5 and GPT-5.6 Sol, alongside the best human. Model curves are the median across all 6 seeds; the human curve is the best per task attempt.
Figure 18: As Figure 17 , but against total Researcher expenditure in USD on a log scale: API spend for models, salary for humans at 119/h,plusGPUtimeat119/h, plus GPU time at 2.5/h for both.
Figure 19: Normalized score against GPU hours (left) and total expenditure (right), pooled across all tasks. Lines are medians and bands span the 30th to 70th percentiles across runs, for Opus 5.5, Opus 4.5, o1 and the human baseliners; the black line is the expert baseline, the best human attempt per task.
Figure 20: Two measures of official-submission quality against the compute multiplier; one point per model, means with 95% CIs. Left: absolute generalization gap of the official submission, ∣validation score−test score∣|\text{validation score}-\text{test score}| . Right: regret of the official submission (the best test score among the run’s submissions minus the official submission’s test score; zero when the Researcher chose its best submission). Both are in normalized units (naive baseline 0, expert baseline 1). Only runs in which the Researcher marked an official submission are included; models with fewer than 30 such runs are omitted.
Figure 21: Best-of- kk scaling of the compute multiplier against total cost. We compare models across 6 tasks. For each arm, kk runs are launched and the one with the best validation score is kept; points show the expected test multiplier for k=1k=1 , 2 and 3. Cost is summed over all kk runs: API spend for all agents plus GPU time.
Figure 22: As Figure 6 , for the performance multiplier. Error bars are 95% intervals from a two-level hierarchical bootstrap with 1,000 resamples, drawing first over tasks and then over runs within each task. Left: performance multiplier against Researcher output tokens per run (arithmetic mean). Right: the same against total API and compute spend per run, including the spend across Researcher, Coder, Liaison and monitors.

Explore similar work

CardsList
Sep 22, 2026cs.AI

The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

LLM agents increasingly work on long-horizon tasks, and the decisions they make along the way, such as which hypothesis to test or which implementation to build on, determine the outcome of the whole run. Making these decisions well is becoming a key capability for both engineering and research agents. We refer to the ability to make good long-horizon decisions as the taste of an agent. While existing benchmarks measure the end-to-end success of agents on long-horizon tasks, none of them measures the taste of an agent. To address this problem, we build Taste-Bench, a benchmark of taste questions constructed automatically from trajectories that agents produced in engineering and research tasks. Each question presents a decision fork, a point in a trajectory where multiple directions are available and one of them leads to a better outcome, and the evaluated model chooses among these directions without seeing what happens after the fork. We mine these forks automatically from parallel attempts at the same task and from detours inside a single trajectory, without needing human annotation. We evaluate frontier models on Taste-Bench and find that the best model answers only 59.7% of the questions correctly. We further find that forks whose deciding evidence appears later in the trajectory are much harder for every model, and that a larger reasoning budget does not improve the accuracy. Finally, we show that taste can be trained. We distill the judgment of a teacher that has seen the outcome into a student model, and the student makes better decisions on unseen tasks and improves end-to-end success on held-out SWE-bench Pro tasks.

Wenbo Pan, Zhichao Liu, Shujie Liu +6

City University of Hong Kong · Microsoft · Independent Researcher

Long-Horizon AgentsAgentic BenchmarksLong-Horizon Task PlanningAgent2AgentDecisions
AskPDF
Jun 16, 2026cs.AI

How Inference Compute Shapes Frontier LLM Evaluation

AI evaluations are shifting toward harder tasks that benefit from longer trajectories involving tool use and iterative problem solving. As a result, performance is increasingly sensitive to the amount and allocation of compute available at test time ("inference compute"). Yet many evaluations still report performance at a single restrictive budget, meaning that low scores may reflect the evaluation setup rather than the model's underlying capability. To test this, we evaluate up to 12 frontier language models on seven challenging benchmarks spanning software engineering, mathematics, medicine, and cybersecurity. We use a controlled setup combining three simple inference-scaling interventions: larger token budgets, context compaction, and repeated submission attempts, guided either by the model itself or by minimal correctness feedback. We find three main results. First, larger token budgets substantially improve performance on benchmarks across multiple domains, including cybersecurity, FrontierMath, Humanity's Last Exam, and TerminalBench. Second, fixed-budget evaluations can increasingly understate frontier capability as models advance. Newer models reach higher performance at large budgets, where they unlock harder tasks and solve them more reliably. Third, benchmarks differ in which inference-scaling methods help most: repeated submission broadly improves performance, but the value of larger token budgets, external feedback, and parallel attempts varies by benchmark. Overall, our results show that benchmark scores are protocol-dependent. We therefore argue that evaluations should report capability as a function of inference-time compute, specify protocol choices explicitly, and compare model generations over a large shared compute range at matched budgets, especially in safety- or policy-relevant settings.

Jessica McFadyen, Ole Jorgensen, Harry Coppock +2

AI Security Institute, London, UK · University of Oxford, Oxford, UK · Harvard University, Cambridge, MA, USA

Frontier Models
AskPDF
Feb 18, 2026cs.AI

When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

Artificial intelligence benchmarks are an important mechanism for measuring model progress and guiding deployment decisions. However, benchmarks quickly "saturate", making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find that nearly half of the our benchmarks exhibit saturation, with rates increasing with age. Further, we find that resilience to saturation is impacted by expert-curation, not by public test data. Our results suggest that design choices can extend benchmark longevity and inform more durable evaluation approaches.

Mubashara Akhtar, Anka Reuel, Prajna Soni +34

ETH Zurich · ETH AI Center · Stanford University +23

Artificial Intelligence BenchmarksLarge Language Model BenchmarksBenchmark SaturationSaturationSystematic StudyUseful Life Prediction
AskPDF