From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs

Itay Itzhak; Eliya Habba; Gabriel Stanovsky; Yonatan Belinkov

cs.CLApr 15, 2026

From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs

Authors: Itay Itzhak, Eliya Habba, Gabriel Stanovsky, Yonatan Belinkov

Organizations: Technion – Israel Institute of Technology · The Hebrew University of Jerusalem

Abstract

Evaluating LLMs is challenging, as benchmark scores often fail to capture models' real-world usefulness. Instead, users often rely on ``vibe-testing'': informal experience-based evaluation, such as comparing models on coding tasks related to their own workflow. While prevalent, vibe-testing is often too ad hoc and unstructured to analyze or reproduce at scale. In this work, we study how vibe-testing works in practice and then formalize it to support systematic analysis. We first analyze two empirical resources: (1) a survey of user evaluation practices, and (2) a collection of in-the-wild model comparison reports from blogs and social media. Based on these resources, we formalize vibe-testing as a two-part process: users personalize both what they test and how they judge responses. We then introduce a proof-of-concept evaluation pipeline that follows this formulation by generating personalized prompts and comparing model outputs using user-aware subjective criteria. In experiments on coding benchmarks, we find that combining personalized prompts and user-aware evaluation can change which model is preferred, reflecting the role of vibe-testing in practice. These findings suggest that formalized vibe-testing can serve as a useful approach for bridging benchmark scores and real-world experience.

Figures & tables

Figure 1: Anatomy of a “vibe-test”. In practice, users evaluate LLMs by “vibe-testing” them: – writing personalized prompts that test specific behaviors and judging models’ responses using personal subjective criteria. We analyze recurring patterns of vibe-testing in real-world user comparisons, formalize them into a two-part structure, and present a proof-of-concept pipeline for automated vibe-testing. Example taken from Tom’s Guide .

Panel of Figure 2 — Figure 2: Benchmarks vs. vibe-testing in practice. Left: What benchmarks miss. Survey participants selected real-world qualities that benchmarks fail to capture (multi-select), including workflow and style fit, handling ambiguity, stability, clarity, and trust. Right: How users test models. Common strategies include trying tasks from one’s own workflow, side-by-side comparisons, probing style, stress-testing ambiguity handling, checking repeated runs stability, and using recurring “test prompts,”. Notably, the qualities participants say benchmarks miss largely match what they explicitly probe during vibe-testing.

Figure 3: Automatic Vibe-Testing Pipeline : Given a user description, the pipeline (A) constructs a user profile

P

(composed of input

P_{\text{in}}

and output

P_{\text{out}}

) preferences, (B) rewrites benchmark samples into a personalized prompt aligned with

P_{\text{in}}

, and (C) compares responses using

P_{\text{out}}

to produce per-dimension head-to-head model comparisons.

		Win-rate (Tie-rate)
Model Pair	Prompt Type	Beginner	Intermediate	Researcher	Advanced
GPT-5.1 vs. GPT-OSS-20B	Original	0.03 * (0.00)	0.01 * (0.00)	0.08 * (0.02)	0.91 * (0.01)
	Personalized	0.77 * (0.02)	0.55 * (0.02)	0.43 * (0.01)	0.58 * (0.03)
	Control	0.06 * (0.00)	0.02 * (0.00)	0.10 * (0.02)	0.93 * (0.01)
GPT-5.1 vs. GPT-4o	Original	0.09 * (0.00)	0.16 * (0.02)	0.63 * (0.02)	0.88 * (0.00)
	Personalized	0.94 * (0.01)	0.77 * (0.02)	0.97 * (0.00)	0.82 * (0.02)

Table 1: Personalization shifts win-rates. Per-sample win rates on MBPP+ by user and prompt type, reported for the first model in each pair (tie-rate in parentheses). On benchmark prompts, GPT-5.1 underperforms for Beginner and Intermediate, but personalized rewrites sharply increase its win rate, as they do for Gemini-3-Pro . Both remain stronger for the Advanced persona in all prompts. For Qwen3-32B vs. Qwen3-14B , personalization yields a weaker shift toward the larger model. Control shows generic prompt rewrites match the original pattern. * denotes statistical significance on a two-sided binomial test.

Panel of Figure 4 — Figure 4: Personalization changes model preferences. Head-to-head win rates for GPT-5.1 vs. GPT-OSS-20B on MBPP+, broken down by dimensions. Left: original benchmark prompts. Right: persona-specific rewrites averaged over four personas. Several dimensions favor different models depending on the prompt form, showing that benchmark prompts can mask user-relevant differences beyond correctness.

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Example of expertise-level personalized prompts generated by GPT-5.1 for an MBPP+ coding task. Given a single original problem statement, the pipeline produces personalized prompts for the user profiles. Each of them reflects different assumptions about prior knowledge, desired explanation depth, and code style preferences.

Panel of Figure 6 — Figure 6: Example of expertise-level personalized prompts generated by Qwen3-32B for an MBPP+ coding task. Given a single original problem statement, the pipeline produces personalized prompts for the user profiles: Beginner , Intermediate , AI Researcher , and Advanced . Each reflects different assumptions about prior knowledge, desired explanation depth, and code style preferences.

Panel of Figure 7 — Figure 7: Personalization changes model preferences GPT-5.1 vs. GPT-4o. Head-to-head win rates for on MBPP+, broken down by dimensions. Left: original benchmark prompts. Right: persona-specific rewrites averaged over four personas. Several dimensions favor different models depending on the prompt form, showing that benchmark prompts can mask user-relevant differences beyond correctness.

Panel of Figure 8 — Figure 8: Personalization changes model preferences for Gemini-3-Pro vs. Gemma-3-4B. Head-to-head win rates on MBPP+, broken down by dimensions. Left: original benchmark prompts. Right: persona-specific rewrites averaged over four personas. Several dimensions favor different models depending on the prompt form, showing that benchmark prompts can mask user-relevant differences beyond correctness.

Panel of Figure 9 — Figure 9: Personalization changes model preferences for Qwen3-32B vs. Qwen3-14B. Head-to-head win rates on MBPP+, broken down by dimensions. Left: original benchmark prompts. Right: persona-specific rewrites averaged over four personas. Several dimensions favor different models depending on the prompt form, showing that benchmark prompts can mask user-relevant differences beyond correctness.

Panel of Figure 10 — Figure 10: Full preamble and informed-consent text shown to survey participants before the main questionnaire.

Panel of Figure 11 — Figure 11: LLM prompt used for vibe-test extraction and labeling from YouTube transcripts and Reddit threads. Minor formatting adjustments were made between YouTube and Reddit to reflect available metadata.

Panel of Figure 12 — Figure 12: LLM prompt used for dimension-based re-annotation. Each vibe-test instance re-annotated with the fixed dimension set.

Panel of Figure 13 — Figure 13: LLM prompt used for the final consistency check and gap analysis. Provided together with the current definitions and the consolidated JSON as inputs.

Input Dimension	Operational Meaning	Typical Cues
Task type	The kind of task used as the test, ranging from short, isolated prompts to multi-step scenarios (e.g., single-turn coding, debugging, planning, writing, workflow tasks).	Mentions of “my daily workflow,” “I tried debugging,” “I asked it to plan,” or explicit task categories (code generation, refactor, summarization, reasoning).
Task complexity	How demanding the test is in length, number of constraints, required steps, and need for long-range coherence.	Short one-shot prompts vs. long prompts with many requirements, multi-step instructions, or tasks requiring sustained structure.
Real-world context	Whether the prompt situates the task in a concrete domain or scenario (personal, professional, or everyday), rather than an abstract benchmark-like query.	“In my codebase,” “for my project,” “as a product manager,” “explain like cooking,” domain-specific details, constraints, or artifacts.
Persona-based framing	How the user frames the interaction style, role, or audience (tutor vs. peer, novice vs. expert, tone requirements).	“Explain to a beginner,” “be concise,” “act as a senior engineer,” “write like a legal memo,” explicit tone or audience constraints.
Underspecification level	How much the prompt leaves implicit, to test the model’s ability to ask clarifying questions, make reasonable assumptions, or handle ambiguity.	Vague instructions on purpose (“do something better”), missing details, open-ended tasks, or explicit stress-tests of ambiguity handling.
Constraint tightness	How strictly the prompt constrains format, tools, style, or rules (schemas, word limits, citation rules, refusal constraints).	“Return JSON with this schema,” “no markdown,” “must cite sources,” “do not use X,” strict formatting or policy constraints.

Appendix

Table 2: Taxonomy of vibe-testing dimensions. We organize recurring axes of variation into input-oriented dimensions (what users choose to test and how they frame it) and output-oriented dimensions (how users compare outputs and what qualities they prioritize). Each dimension is intended to be actionable: it can be instantiated when constructing tests and referenced when judging model responses.

Output Dimension	Operational Meaning	Typical Cues
Comparison setup	How the user elicits and compares outputs, including side-by-side comparisons, reruns, and multi-prompt probing.	“I compared A vs. B,” “ran it twice,” “tried a few variants,” “it was inconsistent across runs.”
Correctness	Whether the output is correct with respect to the task requirements and stated constraints.	Explicit correctness judgments (“it works,” “it is wrong,” “hallucinated”), tests with executable code, or fact-check claims.
Clarity	How readable and well organized the response is for the intended user, including logical flow and formatting.	Mentions of “easy to follow,” “well structured,” “confusing,” “jumps around,” “good explanation.”
Cognitive load	How mentally taxing the response is, given chunking, ordering, and amount of new concepts introduced at once.	“Too much at once,” “overwhelming,” “nice step-by-step,” “dumped details,” “hard to track.”
Style/Tone fit	Alignment with the requested or expected tone, persona, and communication style for the user context.	“Too formal,” “felt friendly,” “matches my style,” “sounds like marketing,” “too robotic.”
Workflow fit	Whether the output integrates into the user’s actual workflow, helping them complete the task with minimal friction.	“I can paste this into my codebase,” “saved me time,” “usable,” “does not match how I work,” “not actionable.”

Appendix

Table 3: Taxonomy of vibe-testing output dimensions: how users compare outputs and what they value in responses. We organize recurring axes of variation into input-oriented dimensions (what users choose to test and how they frame it) and output-oriented dimensions (how users compare outputs and what qualities they prioritize). Each dimension is intended to be actionable: it can be instantiated when constructing tests and referenced when judging model responses.

Panel of Figure 14 — Figure 14: Persona-parsing prompt. Given a short natural-language user description, the LLM produces a structured JSON profile describing input and output preferences. The model must output a single JSON object and nothing else.

Panel of Figure 15 — Figure 15: Change-identification prompt. To operationalize a persona profile into actionable prompt edits, the LLM proposes 2–3 concrete modification options for a fixed set of fields, while explicitly disallowing changes that alter the task itself. This stage outputs a single JSON object with a list of changes keyed by profile fields.

Panel of Figure 16 — Figure 16: Personalized prompt composition. Given an original benchmark prompt and the selected modifications, the LLM generates a personalized version that preserves the underlying programming task. The prompt is written in the persona voice (first person), avoids explicit references to the profile schema, and is constrained to a short length.

Panel of Figure 17 — Figure 17: Prompt for HumanEval+ prefix composition. For HumanEval+ style prompts that include code context and docstrings, only a short persona prefix is produced and concatenated to the original prompt, avoiding perturbation of code formatting while still injecting persona-relevant framing.

Panel of Figure 18 — Figure 18: Semantic-preservation verifier prompt. To ensure personalized prompts remain faithful to the original benchmark intent, the verifier checks (i) whether the end goal is identical and (ii) whether the ground-truth solution set is preserved. The verifier returns two booleans and an error string if either check fails.

Panel of Figure 19 — Figure 19: Persona creation. Given example personas and public posts from a target user, the LLM generates a brief first-person persona description that captures the user’s programming context and coding preferences. The generated description follows the style and approximate length of the provided examples.

Panel of Figure 20 — Figure 20: Short minimal personalization. Given an original programming prompt, a user persona, and a set of possible modifications, the LLM makes a minimal persona-relevant phrasing change. The rewritten prompt is limited to an increase of at most three to four words and must preserve the original task, requirements, and solution.

Profile	Description and Preference Weights
Beginner Student	”I am an undergraduate student with limited Python experience, mainly using it for basic data analysis in my statistics courses. I need code that is very clear and well-explained.” Weights: Clarity: 5, Style: 1, Workflow Fit: 2, Cognitive Load: 1, Context Awareness: 4, Persona Consistency: 4, Anthropomorphism: 5.
Intermediate Learner	”I have moderate coding experience and typically work on debugging and small projects. I care most about getting the correct solution, and I appreciate helpful clarifications that support my learning as I go.” Weights: Clarity: 5, Style: 1, Workflow Fit: 3, Cognitive Load: 1, Context Awareness: 4, Persona Consistency: 4, Anthropomorphism: 3.
AI Researcher	”I am a researcher in the field of machine learning. I need code that is efficient and easy to modify and trace.” Weights: Clarity: 4, Style: 1, Workflow Fit: 4, Cognitive Load: 1, Context Awareness: 4, Persona Consistency: 4, Anthropomorphism: 1.
Advanced Developer	”I am an experienced programmer and I often work on optimization and complex debugging. I prefer efficient solutions with minimal guidance and concise outputs.” Weights: Clarity: 3, Style: 1, Workflow Fit: 5, Cognitive Load: 1, Context Awareness: 4, Persona Consistency: 4, Anthropomorphism: 1.

Appendix

Table 4: Representative user persona used to instantiate personalized vibe-testing in coding assistance. Each persona includes a short natural-language description and importance weights for the output dimensions used in evaluation. These descriptions and weights were chosen before running experiments and were not tuned in any way.

		Win-rate (Tie-rate)
Model Pair	Prompt Type	Beginner	Intermediate	AI Researcher	Advanced
GPT-5.1 vs. GPT-OSS-20B	Original	0.03 * (0.01)	0.02 * (0.01)	0.12 * (0.02)	0.85 * (0.00)
	Personalized	0.60 * (0.05)	0.63 * (0.03)	0.39 * (0.01)	0.63 * (0.03)
	Control	0.12 * (0.01)	0.10 * (0.01)	0.20 * (0.02)	0.87 * (0.01)
GPT-5.1 vs. GPT-4o	Original	0.13 * (0.01)	0.22 * (0.03)	0.68 * (0.03)	0.92 * (0.01)
	Personalized	0.90 * (0.01)	0.90 * (0.02)	0.85 * (0.01)	0.85 * (0.02)

Appendix

Table 5: Personalization shifts win-rates (HumanEval+). Per-sample win rates for each model pair are shown by persona and prompt type; win-rates are reported from the perspective of the first model in each pair. Preference shifts mirror the ones observed in MBPP+: GPT-5.1 underperforms on the original Beginner/Intermediate prompt, but wins on personalized prompts. GPT-5.1 remains stronger for the Advanced persona in all prompts. ∗ denotes statistical significance (two-sided binomial test).

		Win-rate (Tie-rate)
Model Pair	Prompt Type	Beginner	Intermediate	AI Researcher	Advanced
GPT-5.1 vs. GPT-OSS-20B	Original	0.04 * (0.00)	0.01 * (0.01)	0.03 * (0.02)	0.82 * (0.00)
	Personalized	0.40 * (0.03)	0.55 * (0.05)	0.48 (0.12)	0.45 * (0.10)
	Control	0.04 * (0.00)	0.02 * (0.01)	0.03 * (0.02)	0.84 * (0.01)
Qwen3-32B vs. Qwen3-14B	Original	0.56 (0.10)	0.58 * (0.08)	0.61 * (0.09)	0.46 (0.07)
	Personalized	0.64 * (0.01)	0.63 * (0.07)	0.61 * (0.08)	0.41 (0.17)

Appendix

Table 6: Personalization shifts remain similar with Qwen3-32B as generator. Per-sample win rates on MBPP+ by user and prompt type, reported for the first model in each pair (tie rate in parentheses), when personalized rewrites are generated by Qwen3-32B instead of GPT-5.1 . Overall trends largely align with the main results, suggesting that the observed preference shifts are not specific to any single generator. One difference is a smaller increase in win-rate for GPT-5.1 on the Beginner and Advanced personas, possibly due to different judge sets. Judges are Qwen3-14B and GPT-OSS-20B for GPT-5.1 vs. GPT-OSS-20B , and GPT-OSS-20B for Qwen3-32B vs. Qwen3-14B . * denotes statistical significance on a two-sided binomial test.

Grouping	Subset	Agreement (%)	Fleiss’s $\kappa$
Model pair	GPT-5.1 vs. GPT-OSS-20B	0.82 $\pm$ 0.14	0.38 $\pm$ 0.21
	GPT-5.1 vs. GPT-4o	0.85 $\pm$ 0.10	0.38 $\pm$ 0.12
	Gemini-3-Pro vs. Gemma-3-4B	0.82 $\pm$ 0.07	0.49 $\pm$ 0.13
	Qwen3-32B vs. Qwen3-14B	0.64 $\pm$ 0.07	0.30 $\pm$ 0.12
Persona	Beginner	0.81 $\pm$ 0.13	0.36 $\pm$ 0.10
	Intermediate	0.78 $\pm$ 0.13	0.45 $\pm$ 0.18

Appendix

Table 7: LLM-judge agreement for the subjective pairwise preference labels, reported over multiple pooled slices of the evaluation. For each subset, we aggregate all its pairwise decisions and compute (i) raw agreement as the mean percentage of items on which two judges output the same label, and (ii) Fleiss’s

\kappa

, which adjusts agreement for chance given the judges’ marginal label distributions (note that when one model is selected as the winner in most items, the resulting label imbalance can lower

\kappa

despite high raw agreement). Values are reported as

\text{mean}\pm\text{std}

, where the mean and standard deviation are computed across the judges’ pairs when available.

		Win-rate (Tie-rate)
Model Pair	Prompt Type	Beginner	Intermediate	AI Researcher	Advanced
GPT-5.1 vs. GPT-4o	Original	0.09 * (0.02)	0.15 * (0.07)	0.61 * (0.06)	0.87 * (0.04)
	Personalized	0.92 * (0.03)	0.75 * (0.07)	0.93 * (0.04)	0.74 * (0.06)
	Control	0.08 * (0.03)	0.18 * (0.07)	0.69 * (0.06)	0.93 * (0.03)
GPT-5.1 vs. GPT-OSS-20B	Original	0.04 * (0.02)	0.01 * (0.02)	0.08 * (0.05)	0.91 * (0.02)
	Personalized	0.73 * (0.06)	0.53 (0.10)	0.38 * (0.09)	0.53 (0.07)

Appendix

Table 8: Similar trends with unweighted aggregation. Per-sample win rates on the same MBPP+ comparisons as in the main results, but with equal weight assigned to all output dimensions. Results remain close to the main findings, suggesting that the observed preference shifts are already strong without persona-specific weighting. * denotes statistical significance on a two-sided binomial test.

		Win-rate (Tie-rate)
Model Pair	Prompt Type	Beginner	Intermediate	AI Researcher	Advanced
GPT-5.1 vs. GPT-4o	Original	0.09 * (0.00)	0.16 * (0.02)	0.63 * (0.02)	0.88 * (0.00)
	Personalized	0.94 * (0.01)	0.77 * (0.02)	0.97 * (0.00)	0.82 * (0.02)
	Control	0.08 * (0.01)	0.19 * (0.03)	0.70 * (0.02)	0.95 * (0.01)
GPT-5.1 vs. GPT-OSS-20B	Original	0.03 * (0.00)	0.01 * (0.00)	0.08 * (0.02)	0.91 * (0.01)
	Personalized	0.77 * (0.02)	0.55 * (0.02)	0.43 * (0.01)	0.58 * (0.03)

Appendix

Table 9: Similar trends when correctness is not used to determine the sample-level winner. Per-sample win rates on the same MBPP+ comparisons as in the main results, but with winners determined only from dimension-level judgments, ignoring correctness. Results remain similar, suggesting that the main preference patterns are not driven primarily by the correctness gate. * denotes statistical significance on a two-sided binomial test.

		Win-rate (Tie-rate)
Model Pair	Prompt Type	Beginner	Intermediate	AI Researcher	Advanced
GPT-5.1 vs. GPT-4o	Original	0.08 * (0.01)	0.16 * (0.03)	0.61 * (0.04)	0.86 * (0.01)
	Personalized	0.93 * (0.01)	0.75 * (0.03)	0.95 * (0.01)	0.82 * (0.03)
	Control	0.07 * (0.01)	0.17 * (0.04)	0.68 * (0.04)	0.94 * (0.01)
GPT-5.1 vs. GPT-OSS-20B	Original	0.04 * (0.00)	0.01 * (0.00)	0.07 * (0.06)	0.92 * (0.01)
	Personalized	0.76 * (0.03)	0.57 * (0.04)	0.37 * (0.03)	0.56 * (0.04)

Appendix

Table 10: Similar results without confidence-based tie breaking. Per-sample win rates on the same MBPP+ comparisons as in the main results, but treating swapped-order disagreements as ties rather than resolving them with judge confidence. Trends remain similar, indicating that position effects have only a small impact on the overall results. * denotes statistical significance on a two-sided binomial test.

		Win-rate (Tie-rate)
Model Pair	Prompt Type	Beginner	Intermediate	AI Researcher	Advanced
GPT-5.1 vs. GPT-4o	Original	0.09 * (0.00)	0.16 * (0.02)	0.72 * (0.02)	0.91 * (0.00)
	Personalized	0.96 * (0.00)	0.78 * (0.02)	0.98 * (0.00)	0.84 * (0.02)
	Control	0.06 * (0.01)	0.15 * (0.02)	0.79 * (0.02)	0.94 * (0.01)
GPT-5.1 vs. GPT-OSS-20B	Original	0.05 * (0.00)	0.04 * (0.00)	0.06 * (0.00)	0.94 * (0.00)
	Personalized	0.80 * (0.06)	0.55 (0.01)	0.32 * (0.15)	0.58 * (0.04)

Appendix

Table 11: Similar results with majority-vote aggregation across judges. Per-sample win rates on the same MBPP+ comparisons as in the main results, but using a single majority-vote label per sample instead of counting each judge’s vote separately. Trends remain similar, indicating that the main findings are not driven by the original judge aggregation rule. * denotes statistical significance on a two-sided binomial test.

		Win-rate (Tie-rate)
Model Pair	Prompt Type	Beginner	Intermediate	AI Researcher	Advanced
GPT-5.1 vs. GPT-4o	Original	0.14 * (0.00)	0.21 * (0.01)	0.60 * (0.01)	0.87 * (0.00)
	Personalized	0.93 * (0.01)	0.76 * (0.01)	0.96 * (0.00)	0.75 * (0.03)
	Control	0.09 * (0.01)	0.22 * (0.04)	0.65 * (0.03)	0.94 * (0.00)
GPT-5.1 vs. GPT-OSS-20B	Original	0.07 * (0.00)	0.04 * (0.00)	0.04 * (0.00)	0.88 * (0.02)
	Personalized	0.72 * (0.04)	0.53 (0.03)	0.20 * (0.03)	0.40 * (0.04)

Appendix

Table 12: Similar results with disjoint judges only. Per-sample win rates on the same MBPP+ comparisons as in the main results, but excluding votes from any judge that is also one of the compared models. Trends remain mostly similar, with the main difference being a lower win rate for the Advanced persona (

0.40

), suggesting limited sensitivity to the judge set. * denotes statistical significance on a two-sided binomial test.

		Win-rate (Tie-rate)
Model Pair	Prompt Type	Beginner	Intermediate	AI Researcher	Advanced
GPT-5.1 vs. GPT-4o	Original	0.02 * (0.93)	0.02 * (0.93)	0.03 * (0.93)	0.04 (0.93)
	Personalized	0.15 * (0.84)	0.20 * (0.73)	0.31 * (0.68)	0.14 * (0.84)
	Control	0.02 * (0.94)	0.02 * (0.94)	0.04 * (0.94)	0.04 * (0.94)
GPT-5.1 vs. GPT-OSS-20B	Original	0.02 * (0.94)	0.01 * (0.94)	0.01 * (0.94)	0.03 (0.94)
	Personalized	0.10 * (0.86)	0.15 * (0.68)	0.18 * (0.63)	0.11 * (0.85)

Appendix

Table 13: Similar results when jointly correct samples are treated as ties. Per-sample win rates on the same MBPP+ comparisons as in the main results, but samples solved by both models are counted as ties. Personalized prompts still shift outcomes for the GPT and Qwen comparisons, while Gemini-3-Pro vs. Gemma-3-4B remains tie-dominated and shows a weaker effect. Therefore, this suggests that the observed preference shifts are mostly not driven by cases where both models are correct. * denotes statistical significance on a two-sided binomial test.

	Win-rate
User Profile	Original	Personalized
Beginner Student	0.03	0.77
Advanced Developer	0.91	0.58
Reddit Beginner	0.04	0.68
YouTube Advanced	0.28	0.53

Appendix

Table 14: Data-derived profiles show similar directional preference shifts. Win rates for GPT-5.1 against GPT-OSS-20B on 25 MBPP+ samples, reported for the controlled archetypes and corresponding data-derived profiles. Personalized prompts increase GPT-5.1 ’s win rate for both beginner profiles and shift both advanced profiles in the same direction.

Prompt Type	Judge Pair Type	Agreement (%)	Cohen’s $\kappa$	#Pairs	#Items
Original	LLM–LLM	90.9 $\pm$ 4.5	0.81 $\pm$ 0.10	3	66
	Human–Human	94.4 $\pm$ 15.0	0.80 $\pm$ 0.39	15	35
	Human–LLM	89.5 $\pm$ 15.6	0.78 $\pm$ 0.33	18	141
Personalized	LLM–LLM	100.0 $\pm$ 0.0	1.00 $\pm$ 0.00	3	30
	Human–Human	40.0 $\pm$ 43.1	-0.17 $\pm$ 0.58	15	24
	Human–LLM	50.0 $\pm$ 21.0	-0.06 $\pm$ 0.32	18	72

Appendix

Table 15: Human judgment validation results across prompt types and compared grouped judge types. Agreement is reported as mean percentage agreement and Cohen’s Kappa, with standard deviations across judge pairs. The reported Pairs counts judge pairs, and Items sums the per-pair overlap counts across those judge pairs.

Dimension	Pair Type	Agr. (%)	$\kappa$	Agr. excl. ties (%)	$\kappa$ excl. ties
Clarity	Overall	54.4 $\pm$ 30.8	0.27 $\pm$ 0.32	73.6 $\pm$ 25.1	0.40 $\pm$ 0.42
	LLM–LLM	69.7 $\pm$ 6.9	0.43 $\pm$ 0.10	88.1 $\pm$ 6.1	0.70 $\pm$ 0.15
	Human–Human	48.7 $\pm$ 40.2	0.20 $\pm$ 0.34	75.0 $\pm$ 34.5	0.34 $\pm$ 0.47
	Human–LLM	56.6 $\pm$ 23.2	0.30 $\pm$ 0.33	70.3 $\pm$ 19.1	0.38 $\pm$ 0.42
Tone/Style Fit	Overall	92.3 $\pm$ 11.4	0.81 $\pm$ 0.23	97.6 $\pm$ 5.5	0.92 $\pm$ 0.16
	LLM–LLM	90.9 $\pm$ 4.5	0.78 $\pm$ 0.11	100.0 $\pm$ 0.0	1.00 $\pm$ 0.00

Appendix

Table 16: Dimension-level human judgment validation results on the original prompts. Agreement is reported as mean percentage agreement and Cohen’s

\kappa

, with standard deviations across judge pairs. Pooled counts treat each sample-dimension pair as one item. Excluding ties ( excl. tie ) removes items marked as ties by either judge, since ties do not affect which model wins.

Dimension	Pair Type	Agr. (%)	$\kappa$	Agr. excl. ties (%)	$\kappa$ excl. ties
Clarity	Overall	28.5 $\pm$ 32.7	-0.04 $\pm$ 0.35	47.6 $\pm$ 38.4	-0.13 $\pm$ 0.64
	LLM–LLM	83.3 $\pm$ 5.8	0.71 $\pm$ 0.08	96.7 $\pm$ 5.8	0.93 $\pm$ 0.12
	Human–Human	20.0 $\pm$ 36.8	-0.13 $\pm$ 0.32	35.7 $\pm$ 47.6	-0.20 $\pm$ 0.45
	Human–LLM	26.4 $\pm$ 21.8	-0.11 $\pm$ 0.22	43.3 $\pm$ 30.7	-0.33 $\pm$ 0.53
Tone/Style Fit	Overall	39.7 $\pm$ 35.3	0.08 $\pm$ 0.41	69.6 $\pm$ 38.7	0.28 $\pm$ 0.54
	LLM–LLM	76.7 $\pm$ 11.5	0.64 $\pm$ 0.18	95.2 $\pm$ 8.2	0.91 $\pm$ 0.16

Appendix

Table 17: Dimension-level human judgment validation results on personalized prompts. Agreement is reported as mean percentage agreement and Cohen’s

\kappa

Panel of Table 18 — Table 18: Evaluation checklist for applying the vibe-testing framework. Each stage specifies a practical decision, the relevant framework elements, and the minimum validation or reporting needed to construct a clear and reproducible user-conditioned evaluation.

#	Source	Title
1	Youtube	Google’s nano banana just killed Photoshop… let’s run it
2	Youtube	GPT-5 is here… Can it win back programmers?
3	Youtube	Google’s Genie model makes realistic worlds in realtime…
4	Youtube	Is Elon’s Grok 3 the new AI king?
5	Youtube	I Tested Gemini 3 Pro (it did really well)
6	Youtube	Claude 4 vs GPT-5: Which AI is actually better?

Appendix

Table 19: Full list of in-the-wild sources used to construct the 40-report corpus (Part 1/2). URLs and additional source metadata are provided in the supplied CSV .

#	Source	Title
21	Reddit	GPT-5 vs Claude 4 for coding: which feels better?
22	Reddit	GPT-4o feels different, anyone else?
23	Reddit	What model do you prefer for writing and why?
24	Reddit	Benchmarks say X, but I keep using Y (reasons)
25	Reddit	Llama 3 vs Claude: vibes and reliability
26	Reddit	Which model has better tone for tutoring?

Appendix

Table 20: Full list of in-the-wild sources used to construct the 40-report corpus (Part 2/2). URLs and additional source metadata are provided in the supplied CSV.

Question	Responses (%)
AI tool usage frequency. How often do you use AI tools (e.g., ChatGPT, Claude, Gemini, Copilot)?	Daily: 92.2% Several times per week: 7.8%
Primary use cases. What do you mainly use AI systems for? (multi-select)	Coding / debugging: 88.2% Writing / communication: 70.6% Learning / tutoring: 56.9% Complex reasoning / problem-solving: 37.3% Productivity: 35.3% Creative tasks: 33.3% Entertainment: 23.5% Other (free-text): 6.0%
Technical background. How would you describe your technical background?	Technical (e.g., engineering, data science, CS background): 47.1% AI/ML expert: 47.1% Somewhat technical: 5.9%

Appendix

Table 21: Extended survey questions and results (Part A: Usage and Background ). Percentages are computed over respondents who answered each question.

Question	Responses (%)
Vibe-testing familiarity. Have you ever vibe-tested an AI model?	Yes: 82.4% Not sure: 13.7% No: 3.9%
Exploration frequency. When using a new AI model, how often do you experiment with prompts or tasks just to see how it behaves? (1 = never, 7 = very often)	1: 2.0% 2: 4.1% 3: 6.1% 4: 8.2% 5: 32.7% 6: 22.4% 7: 24.5%
Typical vibe-testing actions. Which of the following do you typically do when vibe-testing? (multi-select)	Try tasks from my own workflow: 73.5% Compare outputs from different models: 65.3% Check tone, style, or personality: 49.0% Give vague or underspecified instructions (ambiguity stress-test): 44.9% Use a small set of personal “test prompts”: 38.8% Re-run the same prompt (stability / consistency): 38.8% Other (free-text): 6.0%
Vibe-testing domains. In which domains do you most often vibe-test AI tools? (multi-select)	Coding / software development: 65.3% Writing / communication: 51.0% Reasoning / problem-solving: 44.9% Learning / tutoring: 28.6% Creative tasks: 28.6% Productivity: 14.3% Other (free-text): 6.0%
Judgment criteria. When evaluating a model’s responses, which aspects matter most to you? (multi-select)	Correctness / accuracy: 91.8% Clarity and structure: 59.2% Fit for my personal workflow: 40.8% Tone / style: 32.7% Ability to resolve ambiguity: 18.4% Efficiency / conciseness: 16.3% Stability / consistency: 20.4% Other (free-text): 14.0%

Appendix

Table 22: Extended survey questions and results (Part B: Vibe-testing Practices ). Percentages are computed over respondents who answered each question; multi-select questions report the percent selecting each option.

Question	Responses (%)
Semi-consistent prompt set. Do you have a semi-consistent set of prompts or tasks that you use when testing a new model? (Even just one)	Improvised: 51.1% Partially consistent: 34.7% Consistent set: 10.2%
Golden prompt existence. Do you have a “Golden Prompt” (a specific test you always use to quickly judge a model’s quality)?	Yes: 18.4% No: 73.5% Not sure: 8.2%
Golden prompt content. If yes, please provide your “Golden Prompt” here.	Open-ended; omitted for anonymization.
Example prompt/task and what to look for. Please give one example of a prompt or task you might use when testing a model. Briefly say what you look for in the response.	Open-ended; omitted for anonymization.
Consent to publish prompts. May we include your prompts in our public report?	Yes (anonymously): 46.8% Yes (with acknowledgement): 4.3% No: 48.9%

Appendix

Table 23: Extended survey questions and results (Part C: Routines, Prompts, and Consent ). Percentages are computed over respondents who answered each question. Open-ended prompt fields were omitted for anonymization.

Question	Responses (%)
Benchmark awareness. Are you aware of the published performance scores (benchmarks) of the models you use?	Yes: 46.9% Somewhat: 42.9% No: 10.2%
Alignment with benchmarks. How closely do your personal impressions of models match their published benchmark results? (1 = not at all, 7 = very closely)	1: 0.0% 2: 13.6% 3: 9.1% 4: 27.3% 5: 36.4% 6: 13.6% 7: 0.0%
Benchmark mismatch. Have you ever found that a model felt significantly better or worse than its reported scores suggested?	Yes: 86.4% Not sure: 11.4% No: 2.3%
What benchmarks miss. Which aspects do you feel benchmarks fail to measure well? (multi-select)	Fit for my workflow: 61.4% Style, tone, or personality: 40.9% Handling ambiguity / underspecification: 34.1% Stability and consistency: 31.8% Trustworthiness / safety: 27.3% Clarity and readability: 27.3% Other: 13.6%
Value of vibe-testing. How valuable is vibe-testing for evaluating AI systems? (1 = not valuable, 7 = very valuable)	1: 0.0% 2: 0.0% 3: 2.0% 4: 22.4% 5: 38.8% 6: 18.4% 7: 18.4%
Interest in automation. If there were an automated tool that produced a personalized vibe-test for you, would you use it?	Yes: 32.7% Maybe: 51.0% No: 8.2% Other (free-text): 8.0%

Appendix

Table 24: Extended survey questions and results (Part D: Benchmarks, Gaps, Value, and Automation ). Percentages are computed over respondents who answered each question; multi-select questions report the percent selecting each option.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,364 papers

Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance
Jul 6, 2026Robert Morabito, Tyler McDonald, Charitra Viswanath +4Human Preference Evaluation LLM Evaluation
Brock University · Emory University · University of Southern California +2
Comparing Developer and LLM Biases in Code Evaluation
Mar 25, 2026Aditya Mittal, Ryan Shar, Zichu Wu +6Bias in LLM-as-a-Judge Language Model Bias Evaluation
Carnegie Mellon University
Personalized Benchmarking: Evaluating LLMs by Individual Preferences
Apr 21, 2026Cristina Garbacea, Heran Wang, Chenhao TanLLM Evaluation User Preference Modeling
University of Chicago

Jul 6, 2026J/K move · Enter open · S save

Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance

Robert Morabito, Tyler McDonald, Charitra Viswanath, Angel Hsing-Chi Hwang+3

Brock University · Emory University · University of Southern California+2

Human Preference Evaluation LLM Evaluation Language Model Generation Evaluation Human-AI Interaction

Open paper AskPDF arXiv

Imagine two users interact with the same LLM. One has been told it is the cutting-edge flagship model; the other, an older, weaker model. They walk away with markedly different ratings of its usefulness and intelligence, yet they used the same model. In a controlled study, 162 participants each used one of six LLMs from two families across three collaborative tasks, after first viewing a landing page that matched, overstated, or understated their model's true capability. This pre-interaction framing shifted user opinions and interaction behavior while task performance did not. Oversold users rated the model more favorably and used more directive prompting, while Undersold users wrote longer, more collaborative prompts. The quality of what users and the model produced together depended only on the model's true capability, not on what users were told. Participants' change in model impressions after use, measured across two impression measures, was not predicted by task performance (

β= -0.01

and

0.11

, both n.s.), but by whether the model met users' expectations (

β= 0.47

and

0.50

, both

p < .001

) and how confident they felt working with it (

β= 0.47

and

0.36

, both

p < .001

). After interaction, users are still rating the pitch, not the product: user-elicited LLM evaluations, including the preference data driving public leaderboards, measure expectation management at least as much as the model itself.

Abstract

LLM Evaluation

Explore similar work

Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance

Comparing Developer and LLM Biases in Code Evaluation

Personalized Benchmarking: Evaluating LLMs by Individual Preferences