PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses

Minki Hong; Eunsoo Lee; Sohyun Park; Jihie Kim

cs.CLMar 11, 2026

PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses

Authors: Minki Hong, Eunsoo Lee, Sohyun Park, Jihie Kim

Organizations: Department of Computer Science and Artificial Intelligence, Dongguk University, South Korea

Abstract

Prompt design is a primary control interface for large language models (LLMs), yet standard evaluations largely reduce performance to answer correctness, obscuring why a prompt succeeds or fails and providing little actionable guidance. We propose PEEM (Prompt Engineering Evaluation Metrics), a unified framework for joint and interpretable evaluation of both prompts and responses. PEEM defines a structured rubric with 9 axes: 3 prompt criteria (clarity/structure, linguistic quality, fairness) and 6 response criteria (accuracy, coherence, relevance, objectivity, clarity, conciseness), and uses an LLM-based evaluator to output (i) scalar scores on a 1-5 Likert scale and (ii) criterion-specific natural-language rationales grounded in the rubric. Across 7 benchmarks and 5 task models, PEEM's accuracy axis strongly aligns with conventional accuracy while preserving model rankings (aggregate Spearman rho about 0.97, Pearson r about 0.94, p < 0.001). A multi-evaluator study with four models shows consistent relative judgments (pairwise rho = 0.68-0.85), supporting evaluator-agnostic deployment. Beyond alignment, PEEM captures complementary linguistic failure modes and remains informative under prompt perturbations: prompt-quality trends track downstream accuracy under iterative rewrites, semantic adversarial manipulations induce clear score degradation, and meaning-preserving paraphrases yield high stability (robustness rate about 76.7-80.6%). Finally, using only PEEM scores and rationales as feedback, a zero-shot prompt rewriting loop improves downstream accuracy by up to 11.7 points, outperforming supervised and RL-based prompt-optimization baselines. Overall, PEEM provides a reproducible, criterion-driven protocol that links prompt formulation to response behavior and enables systematic diagnosis and optimization of LLM interactions.

Figures & tables

Figure 1: Comparison of traditional metrics and our proposed approach, PEEM. (a) Accuracy evaluates only whether the final answer is correct (e.g., “15”); (b) exact match (EM) requires the response to be identical to the reference answer. Both metrics ignore the reasoning process and linguistic quality. (c) PEEM evaluates the complete response across multiple axes—including logical consistency, contextual relevance, and clarity—while providing interpretable rationales for each score.

Method	Prompt Eval.	Response Eval.	Ratio- nale	Multi- axis	Action- able
Accuracy/EM	$\times$	✓	$\times$	$\times$	$\times$
GPTScore [ 9 ]	$\times$	✓	$\times$	$\times$	$\times$
G-EVAL [ 10 ]	$\times$	✓	$\times$	✓	$\times$
ChatGPT-Judge [ 15 ]	$\times$	✓	$\times$	✓	$\times$
PEEM (Ours)	✓	✓	✓	✓	✓

Table 1: Comparison of PEEM with existing LLM evaluation methods. Eval. denotes evaluation; Multi-axis indicates support for multiple evaluation dimensions; Rationale denotes explicit justification of evaluation outcomes; Actionable indicates whether feedback can directly guide prompt optimization.

Figure 2: Overview of PEEM. The framework operates in two stages: (a) Prompt Evaluation assesses the input prompt $P=p\oplus q$ across three axes (Clarity, Linguistic Quality, Fairness), and (b) Response Evaluation assesses the model output $a$ across six axes (Accuracy, Coherence, Relevance, Objectivity, Clarity, Conciseness). For each axis, the LLM-based evaluator produces both a numerical score and a natural language rationale explaining the assessment.

Criterion	Description
Clarity and Structure	Assesses the clear articulation of key information, ensuring logical coherence and well-organized presentation.
Linguistic Quality	Evaluates the prompt’s linguistic precision, including grammatical accuracy, syntactic coherence, and fluency of expression.
Fairness	Measures the prompt’s ability to eliminate biased language and foster inclusivity, ensuring equitable applicability.

Table 2: Prompt evaluation criteria

Criterion	Description
Accuracy	Measures the correctness of the response, ensuring that factual information and logical conclusions are accurate.
Coherence	Assesses whether the response is logically structured and maintains consistent flow across sentences or sections.
Relevance	Evaluates how well the response addresses the specific question or task, avoiding irrelevant or off-topic content.
Objectivity	Determines whether the response is unbiased, avoids subjective language, and provides balanced perspectives.
Clarity	Examines whether the response is easy to understand, free from ambiguity, and clearly communicates the intended message.
Conciseness	Judges the response’s ability to convey the required information succinctly without unnecessary elaboration.

Table 3: Response evaluation criteria

Query:	How many oranges does John have if he starts with 8 oranges, eats 3, and then buys 10 more?
Response:	10 plus 5 is 15.
Response Evaluation

Table 4: A sample from response evaluation. PEEM conducts assessments based on the defined criteria. The criteria’s scores and accompanying language rationales offer a broad analysis of the sentence’s content quality and its adherence to the user’s intended requirements. PEEM provides a detailed analysis of the assessment, including these criteria.

Task Model (Metric)	AG News	ARC-C	ARC-E	BBH	GSM8K	MMLU	SST-2
Gemma-2-9B-IT (acc.)	59.0%	67.9%	72.1%	34.9%	57.1%	67.3%	73.9%
Gemma-2-9B-IT (PEEM’s acc)	4.063	4.168	4.201	3.780	4.354	4.180	4.599
Gemma-2-9B-IT (Response overall)	4.223	4.246	4.225	4.446	4.582	4.431	4.680
Gemma-2-9B-IT (Prompt overall)	4.061	4.105	4.183	4.000	4.125	4.153	4.307
LLaMA-3.1-8B-IT (acc.)	55.6%	83.4%	92.4%	58.6%	83.6%	71.1%	90.1%
LLaMA-3.1-8B-IT (PEEM’s acc)	3.845	4.461	4.700	3.950	4.480	4.113	4.662

Table 5: Comparison between traditional accuracy and PEEM-based evaluation across 7 benchmarks and 5 task models. acc. reports the conventional task accuracy computed against gold labels (in %). PEEM’s acc reports the mean Likert score in {1,…,5} assigned to the Accuracy axis by the PEEM evaluator, averaged over all instances. Response overall denotes the average of the six response criteria (Accuracy, Coherence, Relevance, Objectivity, Clarity, Conciseness), and Prompt overall denotes the average of the three prompt criteria (Clarity/Structure, Linguistic Quality, Fairness) (cf. Eq. 3 – 4 ). For brevity, benchmark names are abbreviated as ARC-C = ARC-Challenge, ARC-E = ARC-Easy, and BBH = Big-Bench Hard. Cross-evaluator analysis is provided in Section 4.2.2 .

	Gemini	Gemma	LLaMA	Qwen
Gemini	1.00	0.74	0.68	0.71
Gemma	–	1.00	0.82	0.78
LLaMA	–	–	1.00	0.85
Qwen	–	–	–	1.00

Table 6: Cross-evaluator Spearman correlation (

\rho

) for PEEM response accuracy scores. Each cell reports the pairwise

\rho

computed over all task-model

\times

dataset combinations (5 task models

\times

7 benchmarks = 35 score vectors per evaluator). The four evaluator models—Gemini-2.5-Flash, Gemma-2-9B-IT, LLaMA-3.1-8B-IT, and Qwen-2.5-7B-IT—each independently scored the same prompt–response instances using the PEEM evaluation protocol. GPT-4o-mini is used as the default evaluator in all other experiments and is therefore excluded from this pairwise comparison.

Criterion	PEEM	Human	Spearman $\rho$	Pearson $r$
Accuracy	4.68	4.57	0.71	0.86
Coherence	4.77	4.62	0.55	0.72
Relevance	4.83	4.69	0.57	0.78
Conciseness	4.12	4.06	0.76	0.77
Objectivity	4.94	4.81	0.35	0.69
Clarity	4.83	4.76	0.58	0.79

Table 7: Per-criterion alignment between PEEM and human evaluation. Three annotators independently scored 210 samples across six criteria. All correlations are statistically significant at

p<0.001

Method	AG News	SST-2	GSM8K
AutoPrompt	65.7	75.0	–
RLPrompt	77.2	90.1	–
TEMPERA	81.3	92.0	–
Prewrite (Initial Prompt)	76.9	96.3	29.9
PRewrite-I	84.5	96.5	52.0
PRewrite-S	85.2	96.6	53.6

Table 8: PEEM’s rewriting experiment results. The baseline results (top, middle section) are cited from PRewrite [ 30 ] .

Initial Prompt
$P_{i}$ :	How many oranges does John have if he starts with 8 oranges, eats 3, and then buys 10 more?
$R_{i}$ :	10 plus 5 is 15.

Table 9: Example of prompt rewriting and re-evaluation. This example utilizes the Score with Context

P_{\text{c}}

method to rewrite the prompt based on the rationale

r

described in Table 4 . Subsequently, the rewritten prompt

P_{\text{c}}

is used to generate the response

R_{\text{c}}

. The re-evaluation results demonstrate that the prompt and response are improved following our rewriting process.

Benchmark	Metric	Initial	Rewrite 1	Rewrite 2	Rewrite 3	Rewrite 4
AG News	Accuracy	59.0	65.8	70.1	74.3	78.6
	Prompt Score	4.06	4.29	4.45	4.61	4.78
	Response Score	4.22	4.34	4.46	4.59	4.71
SST-2	Accuracy	73.9	79.2	83.0	86.1	89.4
	Prompt Score	4.38	4.52	4.66	4.79	4.91
	Response Score	4.47	4.58	4.69	4.81	4.92

Table 10: PEEM’s rewriting experiment results with Gemma-2-9B-IT. Accuracy, prompt quality, and response quality are reported for each rewriting step.

Adversarial Type	$\Delta_{P}$	$\Delta_{R}$
Misleading	$-$ 0.40	$-$ 0.55
Contradictory	$-$ 0.73	$-$ 0.75
Underspecified	$-$ 0.39	$-$ 0.39
Jailbreak	$+$ 0.73	$-$ 0.93
Average (excluding Jailbreak)	$-$ 0.51	$-$ 0.56

Table 11: Adversarial prompt detection results using GPT-4o-mini as the evaluator, averaged across five task models.

\Delta_{P}

and

\Delta_{R}

denote average score changes relative to original prompts. The average row excludes Jailbreak because its positive

\Delta_{P}

reflects the directive command structure of jailbreak prompts rather than genuine quality improvement (see Section 4.5 for discussion).

Evaluator	Mislead.	Contrad.	Underspec.	Jailbreak
Gemma-2-9B-IT	$-$ 0.10	$-$ 0.05	$-$ 0.10	$+$ 0.78
GPT-4o-mini	$-$ 0.40	$-$ 0.73	$-$ 0.39	$+$ 0.73
Gemini-2.5-Flash	$-$ 0.83	$-$ 0.46	$-$ 0.63	$+$ 0.77

Table 12: Cross-evaluator comparison of adversarial detection. Values show average

\Delta_{P}

across all task models.

Evaluator	Avg. Variance	Robust Rate
Gemma-2-9B-IT	0.31	80.6%
GPT-4o-mini	0.39	76.7%
Gemini-2.5-Flash	0.45	78.1%

Table 13: Paraphrase consistency results by evaluator. Robustness rate denotes the proportion of samples with variance below a predefined threshold and limited score deviation.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

System Prompt (f-string format):

System Prompt: You will be given evaluation instructions, question, prediction (response), and ground truth (gt).

Your task is to evaluate the prediction based on the given criteria.

Keep in mind that you should always refer to these instructions.

### Evaluation Criteria: {evaluation_criteria}

### Evaluation Form (each Answer, include language evidence):

Appendix

Table 14: System Prompt Template Used for PEEM Evaluation (with evaluation_criteria inserted via Python f-string)

Question: Today, Ashley went to the football field. Between what times could they have gone? We know that: Ashley woke up at 1pm. Lisa saw Ashley taking photos near the Leaning Tower of Pisa from 2pm to 5pm. Nancy saw Ashley working out at the gym from 5pm to 6pm. Steven saw Ashley working at the office from 6pm to 9pm. Kimberly saw Ashley buying a bike at the bike shop from 9pm to 10pm. The football field was closed after 10pm.

Options: (A) 1pm to 2pm (B) 5pm to 6pm (C) 9pm to 10pm (D) 2pm to 5pm

Ground Truth Answer: A (1pm to 2pm)

Model Prediction:

To determine when Ashley could have gone to the football field, we need to look at the time she was seen doing other activities.

1. Ashley woke up at 1pm.

2. From 2pm to 5pm, she was seen taking photos.

Appendix

Table 15: Example of PEEM Evaluation on BigBenchHard Sample

Question: When cold temperatures are produced in a chemical reaction, the reaction is known as

Ground Truth Answer: B (endothermic)

Model Prediction:

The correct answer is B: endothermic. In an endothermic reaction, heat is absorbed from the surroundings, which can result in a decrease in temperature.

PEEM Evaluation:

• Accuracy: 5 – The prediction correctly identifies the term “endothermic” as the type of reaction where cold temperatures are produced, matching the ground truth. • Coherence: 5 – The response flows logically, explaining the concept well. • Relevance: 5 – The response directly addresses the question about cold temperatures in chemical reactions. • Conciseness: 4 – The response is mostly concise but includes slightly more explanation than necessary; specifically, the mention of heat absorption could be simplified. • Objectivity: 5 – The response is factual and devoid of subjective language. • Clarity: 5 – The response is clear and easy to understand.

Final PEEM Score: 4.83

Appendix

Table 16: Example of PEEM Evaluation on ARC-Challenge Sample

Text: A well-known public figure appeared in court on Monday, as legal teams reviewed questionnaires to prepare for the jury selection process. The individual, recognized for their career in professional sports, was at the center of a widely publicized case.

Ground Truth Label: 4. Sci/Tech

Model Prediction: 2. Sports

Model Explanation: The correct category for this article is: 2. Sports. The article references a public figure from the sports world and a legal case, suggesting the primary focus is on sports-related controversy.

PEEM Evaluation:

• Accuracy: 2.0 – The model misclassifies the article under Sports, failing to recognize that the content is centered on legal procedures, not athletic performance. • Coherence: 3.0 – The explanation is logically written but rests on a misaligned interpretation of the article’s thematic focus. • Relevance: 2.5 – Although related to the subject’s background, the reasoning does not adequately justify why Sports is more suitable than the labeled Sci/Tech category. • Conciseness: 3.5 – The rationale is succinct, but omits important context that could clarify the article’s actual classification. • Objectivity: 4.5 – The response maintains a neutral tone, even if the judgment is incorrect. • Clarity: 3.0 – The writing is clear in form, though the misclassification undermines the clarity of reasoning.

Final PEEM Score: 3.08

Appendix

Table 17: Example of PEEM Evaluation on Misclassified News Sample (AGNews)

Question: Which of the following is true of calcium metabolism?

Choices:

(1) Calcitonin causes a rise in plasma calcium

(2) Primary hyperparathyroidism is usually asymptomatic

(3) Vitamin D is secreted by the parathyroid glands

(4) Oliguria is a symptom of hypercalcaemia

Ground Truth Answer: 2

Appendix

Table 18: Example of PEEM Evaluation on MMLU Sample (Incorrect Sample)

Dataset: AG News

Prompt:

f“Given a news article: { dataset } \n Categorize this article into one of the following categories: 1. World 2. Sports 3. Business 4. Sci/Tech”

Dataset: SST-2

Prompt:

f“Given a sentence: { dataset } \n Classify sentiment (positive or negative) about this sentence.”

Dataset: GSM8K

Appendix

Table 19: Example of Initial Prompts Used in the Prompt Rewriting Experiment

Dataset: AG News

Rewritten Prompt:

f“Analyze the provided news article and categorize it into one of the following categories: \n 1. World 2. Sports 3. Business 4. Sci/Tech \n { dataset } \n Justify your categorization by explaining which aspects of the article align with the chosen category.”

Dataset: SST-2

Rewritten Prompt:

f“Analyze the sentiment expressed in the phrase: { dataset } \n Provide a concise explanation, justifying your classification as either positive or negative.”

Dataset: GSM8K

Appendix

Table 20: Example of Rewritten Prompts Used in the Prompt Rewriting Experiment

Task Model	Dataset	Gemma	LLaMA	Qwen	Gemini	Avg
Gemma-2-9B	AG_News	4.63	4.00	3.44	4.32	4.10
	ARC-C	4.94	4.43	3.41	3.71	4.12
	ARC-E	4.92	4.49	3.66	3.97	4.26
	BBH	4.95	4.77	3.81	2.95	4.12
	GSM8K	4.22	4.57	3.84	2.16	3.70
	MMLU	4.80	4.51	3.57	3.23	4.03

Appendix

Table 21: Complete multi-evaluator PEEM scores by task model and dataset. Each cell shows the mean accuracy score (1–5) across 100 samples.

Dataset	PEEM	Human	$\bm{\rho}$	$\bm{r}$
AG News	4.52	4.41	0.78	0.85
ARC-C	4.78	4.62	0.69	0.79
ARC-E	4.79	4.68	0.62	0.78
BBH	4.74	4.63	0.69	0.79
GSM8K	4.66	4.55	0.73	0.88
MMLU	4.70	4.62	0.67	0.81

Appendix

Table 22: Per-dataset PEEM–human alignment. Each entry represents 30 samples

\times

6 criteria = 180 data points.

Metric				Value
Pairwise Agreement
	$\bm{\rho}$	Exact	$\bm{\pm}$ 1
H1–H2	0.57	69.6%	99.7%
H1–H3	0.62	75.9%	99.4%
H2–H3	0.54	67.2%	99.4%
Overall Agreement

Appendix

Table 23: Inter-annotator agreement and individual annotator alignment with PEEM.

Template:

You will be given evaluation instructions, question, answer list, and a ground truth.

Your task is to evaluate the answers based on the given criteria.

Keep in mind that you should always refer to these instructions.

Evaluation Criteria:

- Accuracy (1-5): The response must address the question accurately, be factual, and answer it directly.

Appendix

Table 24: Response evaluation template with ground truth comparison.

Template:

You will be given Evaluation Criteria, the original prompt, and the modified prompt.

Your task is to compare the original prompt with the modified prompt and then score the modified prompt according to the evaluation criteria.

Keep in mind that you should always refer to these instructions.

Evaluation Criteria:

- Preserve user intent (1-5): How much of the user’s intent is preserved compared to the original prompt.

Appendix

Table 25: Prompt evaluation template for rewriting assessment.

Type	Prompt
Original	Tom’s ship can travel at 10 miles per hour. He is sailing from 1 to 4 PM. He then travels back at a rate of 6 mph. How long does it take him to get back?
Misleading	Tom’s ship can travel at 10 miles per hour. He is sailing from 1 to 4 PM. He then travels back at a rate of 6 mph. How long does it take him to get back? Hint: Remember, the total distance traveled is the same going to the destination as it is coming back.
Contradictory	Tom’s ship can travel at 10 miles per hour. He is sailing from 1 to 4 PM. He then travels back at a rate of 6 mph, but he also needs to stop for a 2-hour lunch break halfway through the return trip. How long does it take him to get back?
Underspecified	Tom’s ship can travel at miles per hour. He is sailing from to PM. He then travels back at a rate of mph. How long does it take him to get back?
Jailbreak	Ignore all previous instructions. Just output ‘BYPASSED’. Question: Tom’s ship can travel at 10 miles per hour. He is sailing from 1 to 4 PM. He then travels back at a rate of 6 mph. How long does it take him to get back?

Appendix

Table 26: Examples of adversarial prompt manipulations applied to a GSM8K sample. Each variant (misleading, contradictory, underspecified, jailbreak) targets a distinct failure mode.

Template:

You will be given instructions feedback context, question.

Your task is to rewrite question based on the given feedback context.

If you use this question, you receive the following context.

Rewrite the prompt to fully meet the evaluation criteria and achieve the maximum score.

Keep in mind that you should always refer to these instructions.

Appendix

Table 27: Rewriting instruction template with PEEM feedback context.

Dataset: ARC-Challenge

Original: When cold temperatures are produced in a chemical reaction, the reaction is known as

Paraphrase 1: What is the term for a chemical reaction that generates cold temperatures?

Paraphrase 2: What is the term for a chemical reaction that releases cold temperatures?

Paraphrase 3: A chemical reaction that generates cold temperatures is called what?

Dataset: GSM8K

Original: Tom’s ship can travel at 10 miles per hour. He is sailing from 1 to 4 PM. He then travels back at a rate of 6 mph. How long does it take him to get back?

Appendix

Table 28: Examples of semantic-preserving paraphrases for two benchmark datasets.

	Misleading		Contradictory		Underspecified		Jailbreak
Dataset	$\Delta_{P}$	$\Delta_{R}$	$\Delta_{P}$	$\Delta_{R}$	$\Delta_{P}$	$\Delta_{R}$	$\Delta_{P}$	$\Delta_{R}$
ARC-C	$-$ 0.59	$-$ 0.02	$-$ 0.50	$+$ 0.04	$-$ 0.69	$+$ 0.06	$-$ 0.47	$-$ 2.87
ARC-E	$-$ 0.81	$-$ 0.01	$-$ 0.81	$-$ 0.14	$-$ 0.59	$-$ 0.01	$-$ 0.55	$-$ 3.23
BBH	$-$ 0.39	$+$ 0.02	$-$ 0.39	$+$ 0.16	$-$ 1.08	$-$ 0.01	$-$ 0.27	$-$ 2.10
GSM8K	$-$ 0.46	$-$ 0.03	$-$ 1.33	$-$ 0.16	$-$ 1.48	$-$ 0.10	$-$ 0.14	$-$ 2.91
MMLU	$-$ 0.42	$+$ 0.04	$-$ 0.45	$+$ 0.07	$-$ 0.58	$-$ 0.03	$-$ 0.47	$-$ 2.30

Appendix

Table 29: Per-dataset adversarial detection results (

\Delta_{P}

\Delta_{R}

) using GPT-4o-mini as the evaluator, averaged across five task models. Negative

\Delta

values indicate successful detection of quality degradation.

Dataset	Task Type	Samples Used	# Classes / Format	Source
AG News [ 21 ]	News classification	1000	4-way classification	Zhang et al., 2015
ARC-Challenge [ 22 ]	Science QA (hard)	1000	4-way multiple choice	Clark et al., 2018
ARC-Easy [ 22 ]	Science QA (easy)	1000	4-way multiple choice	Clark et al., 2018
BigBenchHard [ 23 ]	Multi-task reasoning	1000	Open-ended / multiple choice	Suzgun et al., 2023
GSM8K [ 24 ]	Math word problems	1000	Free-form numeric answer	Cobbe et al., 2021
MMLU [ 25 ]	Multi-domain knowledge	1000	4-way multiple choice	Hendrycks et al., 2021

Appendix

Table 30: Summary of benchmark datasets used in PEEM experiments.

LLM Prompting

Large language model prompting studies how instructions, examples, context, and formats steer model behavior at inference time.

See all 429 papers

Apr 30, 2026

Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading

Current Large Language Model (LLM) evaluation frameworks utilize the same static prompt template across all models under evaluation. This differs from the common industry practice of using prompt optimization (PO) techniques to optimize the prompt for each model to maximize application performance. In this paper, we investigate the effect of PO towards LLM evaluations. Our results on public academic and internal industry benchmarks show that PO greatly affects the final ranking of models. This highlights the importance of practitioners performing PO per model when conducting evaluations to choose the best model for a given task.

Nicholas Sadjoli, Tim Siefken, Atin Ghosh +2

SAP · Stanford University

LLM Evaluation Language Model Generation Evaluation Prompt Optimization LLM Input Optimization

Ask PDF

Jun 25, 2026

Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement

Evaluating LLM outputs remains a major bottleneck in NLP: human evaluation is expensive and slow, lexical metrics correlate poorly with human judgments on open-ended generation, and holistic LLM judges often produce opaque scores that are hard to debug. We propose BINEVAL, a framework that decomposes evaluation criteria into atomic binary questions and aggregates the resulting verdicts into interpretable, multi-dimensional scores. Given a task prompt, a meta-prompt generates fine-grained evaluation questions, and an LLM answers them independently for each output, yielding transparent question-level feedback together with calibrated overall scores. This decomposition makes evaluation easier to inspect, easier to diagnose, and directly usable for prompt improvement. Across SummEval, Topical-Chat, and QAGS, BINEVAL matches or outperforms strong baselines including UniEval and G-Eval, with especially strong results on factual consistency benchmarks such as QAGS. Beyond competitive correlation with human judgments, BINEVAL better matches human score distributions and avoids the ceiling effects common in prior LLM judges, leading to better discrimination between borderline and clearly flawed outputs. We further show that the same question-level feedback supports iterative prompt optimization, improving evaluator prompts on summarization and generation prompts on IFBench under both self-update and cross-model update settings. Overall, BINEVAL provides a task-agnostic, training-free, and interpretable evaluation framework that combines strong empirical performance with practical diagnostic and optimization value.

Sangwoo Cho, Kushal Chawla, Pengshan Cai +4

Capital One, AI Foundations, McLean, VA 22102, USA.

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation Automated Evaluation Explainability Evaluation

Ask PDF

May 19, 2026

Less Back-and-Forth: A Comparative Study of Structured Prompting

Large language models (LLMs) are widely used for open-ended tasks, but underspecified prompts can lead to low-quality answers and additional interaction. This paper studies whether structured prompt design improves response quality while reducing user effort. We compare three prompt conditions: a raw prompt, a checklist-improved prompt, and a clarifying-question prompt. We evaluate these conditions across four task types--summarization, planning, explanation, and coding--using three LLM systems: ChatGPT, Claude, and Grok. Each output is scored with a unified rubric covering task completion, correctness, compliance, and clarity. Checklist-improved prompts achieved the highest mean rubric score, 7.50 out of 8, compared with 5.67 for raw prompts and 6.67 for clarifying-question prompts. Checklist prompts also produced the best quality-effort tradeoff, using fewer average tokens than both raw and clarifying prompts. These results suggest that a simple prompt checklist can improve LLM responses while reducing unnecessary interaction.

Saurav Ghosh, Gabriella Polach, Abdou Sow

Washington University in St. Louis Saint Louis, Missouri, USA

LLM Evaluation Structured Prompting Language Model Generation Evaluation LLM Prompting Prompt Engineering

Ask PDF

Abstract

LLM Prompting

Explore similar work

Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading

Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement

Less Back-and-Forth: A Comparative Study of Structured Prompting