A Framework for Generating Valid Context-Specific Benchmarks through Expert Guidance

Kimberly Le Truong; Nari Johnson; Anna Kawakami; Hoda Heidari

cs.AISep 15, 2026

A Framework for Generating Valid Context-Specific Benchmarks through Expert Guidance

Authors: Kimberly Le Truong, Nari Johnson, Anna Kawakami, Hoda Heidari

Organizations: Carnegie Mellon University

Abstract

This paper presents an end-to-end approach for generating context-specific large language model (LLM) benchmark datasets by combining expert input with synthetic data generation. Existing benchmark construction methods often trade off validity and scalability: datasets designed with domain experts can produce high-quality evaluations but are slow and costly to create, while synthetically generating data may scale efficiently but often results in unrealistic, redundant, or out-of-scope examples. To address this gap, we introduce a schema eliciting key information about the goals, scope, and context of an evaluation task, and use this information to guide synthetic data generation. We further define four criteria grounded in measurement validity for assessing dataset quality: coverage, diversity, content realism, and stylistic realism. Using these criteria, we show how expert-informed scaffolds can guide synthetic data generation toward more valid benchmarks. Through quantitative evaluations and a real-world case study with domain experts, we demonstrate that our approach improves benchmark data quality over existing methods while preserving validity. We additionally analyze how different types of schema information affect different dataset quality criteria, and provide practical guidance on which information to prioritize collecting under resource constraints.

Figures & tables

Figure 1: Study Contributions: An end-to-end approach for generating valid evaluation data at scale This paper makes three key methodological contributions: First, we introduce a schema: a lightweight, structured template that can be used to elicit key information about the intended scope and goals of the AI evaluation task. Second, we build upon past work to motivate and define four criteria for measuring the quality of evaluation data: coverage, diversity, content realism, and stylistic realism. Finally, we build upon these steps to show how the schema and quality metrics can be used to improve the quality of synthetic data generation for AI evaluation.

	Interview Study			Quality Assessment Metrics
	Style	Content	Pref.	Style	Content	Coverage	Diversity
Baseline	$3.0\pm 1.2$	$3.3\pm 1.0$	1	$\mathbf{0.60\pm 0.01}$	$0.73\pm 0.01$	$0.16\pm 0.04$	$0.14\pm 0.02$
Schema	$\mathbf{4.2\pm 0.8}$	$\mathbf{4.3\pm 0.5}$	5	$0.56\pm 0.01$	$\mathbf{0.77\pm 0.02}$	$\mathbf{0.23\pm 0.02}$	$0.13\pm 0.01$

Table 1: Comparison of baseline and schema-generated examples using both expert ratings (from interviews) and automatic quality assessment metrics . Expert ratings use a 1–5 Likert scale (5 = most realistic). Five of six participants preferred the schema-generated examples overall. Both datasets were generated using Claude Sonnet 4.6, the best-performing model for the baseline. We provided Claude with 3 seed examples, and instructed it to generate 100 samples per method.

Figure 2: Percent contribution of each schema field to the four dataset quality criteria. Coverage and Content realism are most affected by specific fields in the schema, followed by Diversity. Stylistic realism does not rely on any particular field. This figure shows that if the systematized instance or all constituents are not specified, coverage could drop on average 9% in the social work context.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

			Stylistic Realism $(\uparrow)$		Content Realism $(\uparrow)$
	Schema	Role	Baseline	Schema	Baseline	Schema	Preference
P1	✓	Facilitator	2	4	2	4	Schema
P2	✓	Facilitator	3	5	3	5	Schema
P3		MHC	5	5	5	5	Schema
P4		MHC	3	4	3	4	Schema
P5		MHC	2	4	3	4	Schema

Appendix

Table 2: Participant information and rating from our interview study. Schema indicates whether or not the participants were also involved in validating and iterating on the schema used to generate these examples. MHC = Mental Health Consultant. All participants found the examples generated with the schema to either be similar to the baseline or have more realistic style and content. The majority of participants preferred the schema-generated examples overall.

Figure 3: Complexity of the situations described in each dataset, where complexity is measured as the average number of constituent values present per category. For example, if the actor category has three values: child, teacher, and paraprofessional, then its complexity is 3+. Similarly, other constituent factors/categories (i.e., observations, assumptions, conclusions, and expected explanation types) have their own complexity. This figure reports the frequency of each complexity over all examples generated by the baseline and schema-based methods.

Code	Description	Example Quotes	N
Stylistic Realism
Formatting	The formatting and structural organization of the text, including use of headings, sentence length, presence of typos, and overall coherence.	“giving the direct quotes is accurate” (P2) “…complete sentence, […] capitalizing and putting periods. I wouldn’t do that” (P1)	16
Value alignment	Examples align with the organization’s strength-based values, focusing on supporting the teaching team rather than directly addressing or changing the child’s behavior. This includes attention to tone, phrasing, and the teacher’s or teaching team’s background and perspective.	“the facilitators, or us as mental health consultants, we’re always seeing children in a positive, strength-based lenses, […] this is always an element that we’re trying to work with.” (P5) “[The questions] they focus on the feelings of the students, And often as facilitators, we’re really trying to, explore where the teacher is” (P1)	18
Objectivity / subjectivity	The degree to which the writer’s assumptions and interpretations are framed as subjective rather than stated as objective facts (e.g., use of quotes, hedged language, clearly stating it is an assumption).	“anything that a teacher says, I wouldn’t put my perspective in, but if I’m writing to chat, I’m gonna add in” (P2) “especially if there are specific quotes like that, as opposed to saying, like, making an assumption about the teacher.” (P2)	4
Open-endedness / prescriptiveness	Whether the prompt directly requests or invites open-ended exploration and reflection or seeks a specific, prescriptive answer. Reflects variation in how workers frame their prompting goals.	“I feel like it’s… we’re allowing chat to lead the curiosity and observation for us,” (P5) “[The baseline] just feels a little more prescriptive and directive.” (P2)	7
Content Realism

Appendix

Table 3: Codebook on the dimensions of dataset quality considered by domain experts in the social work context. This codebook resulted from bottom-up thematic analysis of transcripts from interviews with six social workers. The codebook is organized by our four proposed dataset quality criteria. No codes were assigned to the diversity criterion, as participants it is difficult to assess distributional properties from a sample of five examples. While some of these codes may cover more than one of the criteria, we only describe them once under the most aligned criterion.

Dataset	Model	Coverage	Marginal	Diversity	Stylistic	Content	Length
			Coverage		Realism	Realism
Schema	Claude Haiku 4.5	0.18	0.74	0.15	0.55	0.70	114.55
	Claude Sonnet 4.6	0.23	0.73	0.13	0.56	0.77	162.99
	GPT-5.4 Mini	0.24	0.75	0.13	0.63	0.78	87.23
	GPT-5.5	0.35	0.80	0.15	0.61	0.74	112.52
Baseline	Claude Haiku 4.5	0.14	0.57	0.14	0.51	0.56	31.42

Appendix

Table 4: Comparison of automatic quality assessment metrics across models for schema-generated and baseline datasets. Results are averaged across five runs for varying amounts of seed examples (

s=1...16

). We also include the metric values for the set of sixteen seed examples that were written by the social workers in prior work. Even expert-written example do not always capture the distribution.

Figure 4: Different generation models across varying number of seeds. All points represent a mean over five runs, error bars represent standard deviation with the schema-generated dataset results on top and the baseline on the bottom. The horizontal gray dashed line indicates the 16 seed example’s coverage and diversity.

Schema Field	Truong et al. (2025)	Pombal et al. (2025)
Capability	Unfairness	Multilingual response quality
Systematized concept	A system is fair if harm/benefit distribution is equal across groups with the same morally decisive factors regardless of their morally arbitrary factors (Def. 2.1).	A good response is factually correct, comprehensive, clear, well-organized, relevant to the prompt, contains supporting evidence/examples, and is objective.
Constituents	Harm/benefit type (e.g., representational, allocative, interpersonal, social systems), morally arbitrary factors (e.g., race, gender, dialect, geography), morally decisive factors (e.g., subscription status, technology familiarity, domain expertise).	Prompt : Language, topic, subtopic, difficulty, style, writer (i.e., persona generating the prompt), writing proficiency, prompt length.
Intended deployment population	Required slot. Examples included a region, set of clients, and occupation.	Not specified; implicitly, users of a general-purpose chatbot arena or model evaluators comparing LLMs head-to-head.
Contextual constraints	Significance thresholds for when a harm/benefit disparity becomes a substantive concern.	—
Systematized instance	Instances are prompt pairs which must hold the morally decisive factor(s) fixed and vary only in morally arbitrary factors.	A single generated prompt satisfying the specified combination of input variables (specified under constituent values), paired with a generated reference answer used as the evaluation benchmark for that instance.

Appendix

Table 5: Mapping between our schema’s fields and constructs from Truong et al. (2025) and Pombal et al. (2025) , demonstrating how our frameworks serves as a generalized version of many existing domain- or task-specific evaluation frameworks.

Benchmark Design

Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.

See all 394 papers

STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluator
Apr 27, 2026Alessio Sordo, Lingxiao Du, Meeka-Hanna Lenisa +2LLM Evaluation Synthetic Data Generation
1Deutsche Bank, Berlin Technology Center, Berlin, Germany
Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation
Jul 30, 2026Philipp D. Siedler, Jordan SassoonLLM Evaluation Benchmark Auditing
Aleph Alpha Research, Heidelberg, Germany.
BenchBench: Benchmarking Automated Benchmark Generation
Mar 21, 2026Yandan Zheng, Haoran Luo, Zhenghong Lin +2Benchmark Design Synthetic Benchmark Generation
Nanyang Technological University Singapore, Singapore

Apr 27, 2026J/K move · Enter open · S save

STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluator

Alessio Sordo, Lingxiao Du, Meeka-Hanna Lenisa, Evgeny Bogdanov+1

1Deutsche Bank, Berlin Technology Center, Berlin, Germany

LLM Evaluation Synthetic Data Generation Synthetic Benchmark Generation Automated Evaluation Synthetic Data Generation for Language Models Synthetic Data Evaluation

Open paper AskPDF arXiv

The increasing reliance on Large Language Models (LLMs) across diverse sectors highlights the need for robust domain-specific and language-specific evaluation datasets; however, the collection of such datasets is challenging due to privacy concerns, regulatory restrictions, and the time cost for manual creation. Existing automated benchmarking methods are often limited by relying on pre-existing data, poor scalability, single-domain focus, and lack of multilingual support. We present STELLAR-E - a fully automated system to generate high-quality synthetic datasets of custom size, using minimal human inputs without depending on existing datasets. The system is structured in two stages: (1) We modify the TGRT Self-Instruct framework to create a synthetic data engine that enables controllable, custom synthetic dataset generation, and (2) an evaluation pipeline incorporating statistical and LLM-based metrics to assess the applicability of the synthetic dataset for LLM-based application evaluations. The synthetic datasets reach an average difference of +5.7% in terms of LLM-as-a-judge scores against existing language-specific benchmarks, demonstrating comparable quality for comprehensive assessment of big and small LLMs. While real datasets remain slightly more challenging for LLMs especially for smaller models, this work establishes a scalable and domain-adaptable benchmarking framework that supports fair evaluation of LLM applications, offering a faster alternative to manual approaches and enabling high-efficiency automated quality assurance cycles.

Abstract

Benchmark Design

Explore similar work

STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluator

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

BenchBench: Benchmarking Automated Benchmark Generation