Structured Prompts Improve Evaluation of Language Models
Organizations: Stanford University
Abstract
As language models (LMs) are increasingly adopted across domains, high-quality benchmarking frameworks are essential for guiding deployment decisions. However, frameworks such as Holistic Evaluation of Language Models (HELM) typically evaluate models under a single static prompt configuration, even though model behavior depends strongly on prompt choice. As a result, reported scores can reflect prompt choice as much as model capability. Declarative prompting frameworks, such as DSPy, offer a scalable way to operationalize this evaluation protocol by evaluating models under structured prompting strategies rather than static prompts. Here, we conduct a reproducible DSPy+HELM benchmarking study and quantify how prompt choice impacts benchmark outcomes. Using five prompting methods, we evaluate four frontier and two open-source LMs across seven benchmarks against HELM baseline scores and find that prompt choice can materially impact leaderboards. Structured prompting improves performance (by 6% on average), alters comparisons (leaderboard rankings shift on benchmarks); chain-of-thought provides the strongest cost-performance tradeoff while more advanced optimizers offer smaller additional gains. Beyond the known intuition that prompts affect outputs, our benchmarking study integrates structured prompting into an established evaluation framework and quantifies how prompt choice alone impacts leaderboards. We open-source (i) DSPy+HELM Evaluation (https://github.com/stanford-crfm/helm/pull/3893) and (ii) Prompt Optimization Pipeline (https://github.com/StanfordMIMI/dspy-helm).
Figures & tables
| Benchmark | Input Output | Task | Samples |
| MMLU-Pro | Reasoning Question Answer | Multi-Task Reasoning | 1,000 |
| GPQA | Graduate Question Answer | Graduate-Level QA | 446 |
| GSM8K | Math Problem Solution | Numeric Problem-Solving | 1,000 |
| MedCalc-Bench | Patient Note Computed Value | Computational Reasoning | 1,000 |
| Medec | Medical Narrative Errors | Error Classification | 597 |
| HeadQA | Medical Question Answer | USMLE-Style QA | 1,000 |
| Model | API Identifier | Release | Context | Reasoning | Open-Source |
| Claude 3.7 Sonnet | anthropic/claude-3-7-sonnet-20250219 | 02/19/2025 | 200k | ✗ | ✗ |
| Gemini 2.0 Flash | google/gemini-2.0-flash-001 | 02/01/2025 | 1000k | ✗ | ✗ |
| GPT 4o | openai/gpt-4o-2024-05-13 | 05/13/2024 | 128k | ✗ | ✗ |
| o3 Mini | openai/o3-mini-2025-01-31 | 01/31/2025 | 200k | ✓ | ✗ |
| Llama 3.3 70B | meta-llama/Llama-3.3-70B-Instruct | 12/06/2024 | 128k | ✗ | ✓ |
| Qwen3 4B | Qwen/Qwen3-4B-Instruct-2507 | 08/06/2025 | 256k | ✗ | ✓ |
| Prompting Method | Claude 3.7 Sonnet | Gemini 2.0 Flash | GPT 4o | o3 Mini | Llama 3.3 70B | Qwen3 4B |
| HELM Baseline | 64.8% 3.3 | 61.4% 3.4 | 61.0% 3.3 | 70.9% 3.3 | 57.2% 3.4 | 47.8% 3.3 |
| Zero-Shot Predict | 65.1% 3.4 | 61.7% 3.4 | 59.7% 3.3 | 73.2% 3.2 | 59.2% 3.4 | 49.3% 3.4 |
| Zero-Shot CoT ★ | 69.4% 3.2 | 66.2% 3.4 | 65.7% 3.3 | 72.7% 3.2 | 64.0% 3.4 | 59.0% 3.4 |
| BFRS ★ | 69.3% 3.3 | 66.2% 3.3 | 65.9% 3.2 | 73.1% 3.2 | 64.2% 3.3 | 60.3% 3.4 |
| MIPROv2 ★ | 69.8% 3.3 | 66.2% 3.3 | 65.3% 3.3 | 73.1% 3.1 | 62.1% 3.4 | 60.4% 3.4 |
| Highest Baseline ★ | +5.0% | +4.8% | +4.9% | +2.3% | +7.0% | +12.6% |
| Benchmark | Prompting Method | Claude 3.7 Sonnet | Gemini 2.0 Flash | GPT 4o | o3 Mini | Llama 3.3 70B | Qwen3 4B |
| MMLU-Pro | HELM Baseline | 76.3% 2.7 | 66.1% 3.0 | 62.2% 3.0 | 77.1% 3.1 | 64.7% 3.0 | 44.9% 3.1 |
| Zero-Shot Predict | 77.7% 2.7 | 70.3% 2.8 | 60.7% 3.0 | 78.4% 3.1 | 62.7% 3.0 | 41.7% 3.0 | |
| Zero-Shot CoT | 79.7% 2.5 | 75.3% 2.8 | 67.6% 3.0 | 76.2% 3.1 | 68.5% 2.9 | 66.2% 3.0 | |
| BFRS | 80.1% 2.5 | 75.4% 2.7 | 71.1% 2.8 | 76.5% 3.1 | 68.5% 2.9 | 68.6% 2.9 | |
| MIPROv2 | 80.6% 2.4 | 75.3% 2.6 | 68.7% 2.9 | 76.1% 3.1 | 56.5% 3.1 | 68.6% 2.9 | |
| GPQA | HELM Baseline | 57.0% 4.7 | 53.4% 4.7 | 45.5% 4.7 | 57.6% 4.7 | 40.4% 4.5 | 34.3% 4.3 |
| Benchmark | Prompting Method | Claude 3.7 Sonnet | Gemini 2.0 Flash | GPT 4o | o3 Mini | Llama 3.3 70B | Qwen3 4B |
| MedCalc-Bench | HELM Baseline | 21.0% 2.5 | 15.8% 2.3 | 18.8% 2.4 | 34.0% 2.9 | 11.3% 2.0 | 4.7% 1.4 |
| Zero-Shot Predict | 20.6% 2.5 | 17.0% 2.4 | 15.7% 2.3 | 33.4% 3.0 | 9.9% 1.9 | 11.5% 2.0 | |
| Zero-Shot CoT | 35.3% 3.0 | 26.3% 2.7 | 26.6% 2.8 | 34.2% 3.0 | 22.5% 2.6 | 20.8% 2.5 | |
| BFRS | 34.1% 3.0 | 25.2% 2.7 | 27.0% 2.8 | 34.7% 3.0 | 20.0% 2.5 | 22.7% 2.6 | |
| MIPROv2 | 34.7% 3.0 | 25.4% 2.8 | 26.8% 2.8 | 34.3% 2.9 | 21.0% 2.5 | 21.5% 2.5 | |
| Medec | HELM Baseline | 62.8% 3.9 | 59.6% 4.0 | 58.0% 3.9 | 68.7% 3.7 | 52.9% 4.0 | 52.1% 4.0 |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,523 papers