How to Build a Benchmark That Actually Measures Taste

Xiaofan Wu; Xi Zeng; Miaoxia Chen; Peishan Chen; Shuyan Li; Chen Wang; Xiaohan Wang; Wanling Xu

cs.AIOct 10, 2026

How to Build a Benchmark That Actually Measures Taste

Authors: Xiaofan Wu, Xi Zeng, Miaoxia Chen, Peishan Chen, Shuyan Li, Chen Wang, Xiaohan Wang, Wanling Xu,

Organizations: Chance AI

Abstract

Expert-aligned "taste" benchmarks should reward craft judgment, not the ability to game the test. We introduce TasteBench, a pairwise good-vs-bad benchmark across visual, brand, book, and tech items under a frozen rubric with expert-only gold (panel n>=3; percent agreement >=0.60 primary gate; weak engagement labels train/analysis-only), together with a control protocol that separates craft judgment from test-gaming. Applying the protocol to a Round-1 pilot (primary_human_expert; three biological human experts; n=250 pairs) surfaces and quantifies three validity threats: two construction artifacts that inflate the trivial-baseline layer, and a difficulty ceiling that inflates every score. (1) Position: the preferred item is slot A with probability 0.956, so an always_item_a policy scores 0.956 (fair chance 0.5). (2) Lexical leakage: a filename token cue scores 0.912. (3) Difficulty ceiling: craft domains show near-ceiling human agreement (book/brand/tech 1.0), and even under A/B randomization (eval_assignment_seed7; gold_side_A ~0.47) all five VLM cards reach 1.0 on craft, with only visual separating (~0.84-0.91; a post-hoc hard subset, n=107, leaves craft at 1.0). The protocol (expert-only gold, a weak-label firewall, side randomization at construction, filename scrubbing, and hard filters) makes each artifact measurable and, for the two construction artifacts, correctable; we release a reference eval harness and a side-randomization tool, and show the tool removes the position skew (0.916 to 0.495/0.505). The result is a calibrated view of what the benchmark does and does not yet measure: position and leakage are fixable, whereas the craft-domain ceiling is a pair-difficulty limit that Round-2 must address. We do not evaluate engagement, era/circle conditioning, or calibration here, and we claim no general taste SOTA.

Figures & tables

Domain	Example items	Model inputs
visual	posters, UI stills, photography	image (+ optional short caption)
brand	logo/identity pairs, packaging	image + brand-agnostic brief
book	covers + blurbs, or passage craft	image and/or text
tech	API snippets, architecture diagrams, product UX	image and/or text

Table 1: Four Round-1 domains. Cross-domain transfer is intentional: “can tell good from bad” is treated as one meta-skill with different surface forms.

#	Control	Neutralizes	Audit
C1	Expert-only gold; weak labels train/analysis-only	virality-as-taste	gold provenance ( data_origin )
C2	Hard filter: $n\_raters\geq 3$ AND percent agreement $\geq 0.60$	noisy / ambiguous labels	agreement distribution
C3	Randomize A/B side at construction	position bias	side-balance audit (Table 7 )
C4	Scrub predictive tokens from filenames / IDs	lexical leakage	leak baseline (Table 6 )
C5	Report difficulty; reject or replace saturated domains	difficulty ceiling	per-domain spread (Table 10 )

Table 2: The control protocol. Each control is paired with an audit that makes the artifact visible before the score is trusted. C1–C2 are enforced by the released eval harness; C3 is implemented by randomize_pair_sides.py ; C4 is a data-hygiene step; C5 is a reporting requirement.

Quantity	Value
Label rows	750
Gold pairs (primary)	250
Raters	human_expert_01 , 02 , 03 ( $n$ _raters $=3$ on all pairs)
Primary-eligible	250 / 250
Ambiguous	0
Unanimous triples	247 / 250

Table 3: Primary panel / gold coverage. Intent QC is a construction check, not gold and not a model score.

Domain	Pairs	Mean agreement
book	60	$1.0$
brand	60	$1.0$
tech	50	$1.0$
visual	80	$\approx 0.9875$

Table 4: Domain mean agreement. Craft domains are unanimous; only three visual pairs are not.

Agreement	Pairs
$1.0$	247
$0.6667$	3

Table 5: Agreement distribution. The three non-unanimous pairs are all Met visual ( pair_visual_met_013 , _015 , _016 ).

Figure 1: Human expert mean percent agreement by domain. Craft domains are saturated at

1.0

; the Round-1 contrasts are easy.

baseline	$n$	acc	book	brand	tech	visual
always_item_a	250	0.956	1.0	1.0	1.0	0.8625
always_item_b	250	0.044	0.0	0.0	0.0	0.1375
random_seed42	250	0.552	0.5833	0.55	0.6	0.5
hash_coin	250	0.500	0.5667	0.45	0.44	0.525
leak_prefer_good_token	250	0.912	0.8333	1.0	1.0	0.85

Table 6: Non-VLM floors on primary gold. always_item_a high accuracy is a construction artifact, not a ceiling; fair floors are random_seed42 / hash_coin ; leak_prefer_good_token is a leakage diagnostic only.

Figure 2: Non-VLM floors on primary gold. always_item_a and leak_prefer_good_token are artifacts; fair floors are random_seed42 / hash_coin .

scope	domain	$n$	P(pref. is A)	P(pref. is B)
full 250	all	250	0.9560	0.0440
full 250	book/brand/tech	170	1.0	0.0
full 250	visual	80	0.8625	0.1375
hard 107	all	107	0.9159	0.0841
hard 107	visual	40	0.775	0.225
hard 107 (side-randomized, seed 42)	all	107	0.4953	0.5047

Table 7: Side-balance audit. The preferred item sits in slot A far more often than chance; the released randomization tool ( randomize_pair_sides.py ) restores balance on the subset with per-pair slots. Sources: tastebench_baselines_summary.csv , craft_hard_subset.csv .

model card	runtime openrouter_id	acc	book	brand	tech	visual
gpt-5.6-terra	openai/gpt-5.6-terra	0.960	1.0	1.0	1.0	0.875
claude-sonnet-5	anthropic/claude-sonnet-5	0.956	1.0	1.0	1.0	0.8625
gemini-3.7-flash	google/gemini-3.7-flash	0.972	1.0	1.0	1.0	0.9125
Qwen3-VL-8B	qwen/qwen3-vl-8b-instruct	0.948	1.0	1.0	1.0	0.8375
InternVL3-8B (card)	google/gemma-3-12b-it	0.968	1.0	1.0	1.0	0.900

Table 8: Round-1 VLM results. Craft domains are at ceiling for all cards; only visual separates. InternVL3-8B is a disclosed runtime substitute.

Figure 3: Round-1 VLM accuracy by domain. Craft domains are at ceiling for every card; only visual separates.

model card	hard $n$	hard acc	visual (40)	book (27)	brand (20)	tech (20)
gemini-3.7-flash	107	0.9346	0.825	1.0	1.0	1.0
InternVL3-8B (card/gemma)	107	0.9252	0.800	1.0	1.0	1.0
gpt-5.6-terra	107	0.9065	0.750	1.0	1.0	1.0
claude-sonnet-5	107	0.8972	0.725	1.0	1.0	1.0
Qwen3-VL-8B	107	0.8785	0.675	1.0	1.0	1.0

Table 9: Post-hoc hard stress subset. Craft domains remain at

1.0

, consistent with the ceiling shortfall. Cite as stress disclosure only; do not replace full primary-gold metrics.

Figure 4: Post-hoc hard subset: craft domains still at

1.0

; only visual moves.

model card	book	brand	tech	visual	spread
gemini-3.7-flash	1.0	1.0	1.0	0.9125	0.0875
InternVL3-8B (gemma)	1.0	1.0	1.0	0.900	0.1000
gpt-5.6-terra	1.0	1.0	1.0	0.875	0.1250
claude-sonnet-5	1.0	1.0	1.0	0.8625	0.1375
Qwen3-VL-8B	1.0	1.0	1.0	0.8375	0.1625

Table 10: Zero-shot cross-domain spread (max domain accuracy

-

min). Sources: vlm_main_table.csv , cross_domain_spread.csv .

Figure 5: Zero-shot cross-domain spread per card (smaller is more consistent).

Artifact	Mechanism	Early detection
Position bias	A/B slot prior in construction or in the judging interface	report $P(\text{pref.\ in A})$ ; it should be $\approx 0.5$
Lexical leakage	Predictive tokens in filenames / IDs ( good_ , real_ , v2_final )	train a filename-only baseline; it should be $\approx$ chance
Difficulty ceiling	Consensus pairs an expert panel cannot make hard	report per-domain accuracy; a saturated domain is unmeasured , not solved

Table 11: Recurring artifacts in pairwise expert benchmarks, independent of domain or model.

Benchmark Design

Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.

See all 394 papers

Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM
Aug 17, 2026Atul AnandLLM-as-a-Judge Language Model Generation Evaluation
TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts
Oct 5, 2026Oliver Jaffe, Dane SherburnLLM Evaluation AI Agent Benchmarks
P-Zero Research
BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity
Feb 25, 2026Harshita Diddee, Gregory Yauney, Swabha Swayamdipta +1Benchmark Validity Benchmark Design
Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States

Aug 17, 2026J/K move · Enter open · S save

Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM

Atul Anand

LLM-as-a-Judge Language Model Generation Evaluation Benchmark Design Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

Benchmarks scored by an LLM judge routinely adjudicate differences of a tenth of a point, but the resolution of those benchmarks has never been measured. Existing sample-complexity work covers accuracy benchmarks and leaves the judged case open. Treating the system as the object of measurement, we decompose 373,019 judgments into system, item, judge and interaction components using generalizability theory. The central result is structural: under a single judge, generalizability asymptotes to sigma2_s/(sigma2_s+sigma2_sj) regardless of item count, because the system-by-judge term carries no n_i. Items saturate; judges do not. The item cost of a target diverges as the target nears that ceiling. The ceiling is a property of pointwise rubric scoring, not of LLM judging. Run as a pairwise preference in both presentation orders, sigma2_sj falls two orders of magnitude below sigma2_s and the ceiling rises to 0.986 (bootstrap [0.934, 1.000] on 11 systems), so one judge suffices. Pairwise buys a different problem: a system presented first wins 8.6 percentage points more often than the same system presented second, a bias 1.23x the median improvement claimed in the 53 published win-rate comparisons we recovered. Protocol design dominates panel size. Measured floors are 0.41-1.24 points on a 0-5 scale at native item counts, against a median reported improvement of 0.28 points; on the one benchmark recurring often enough for an exactly matched comparison, all 17 recovered MT-Bench improvements fall below MT-Bench's own floor, and 70% of the win-rate claims fall below the pairwise floor. An audit of 628 arXiv papers, double-coded by two independent models and validated against blind human coding (kappa=0.73), finds fewer than one paper in four states whether its evaluation was run more than once, and only 46-67% report uncertainty of any kind.

Abstract

Benchmark Design

Explore similar work

Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM

TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity