Qworld: Question-Specific Evaluation Criteria for LLMs

Shanghua Gao; Yuchang Su; Pengwei Sui; Curtis Ginder; Marinka Zitnik

cs.CLMar 6, 2026

Qworld: Question-Specific Evaluation Criteria for LLMs

Authors: Shanghua Gao, Yuchang Su, Pengwei Sui, Curtis Ginder, Marinka Zitnik

Organizations: Department of Biomedical Informatics, Harvard Medical School · Department of Medicine, Brigham and Women’s Hospital · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Broad Institute of MIT and Harvard ·

Abstract

Evaluating large language models (LLMs) on open-ended questions is difficult because response quality depends on the question's context. Binary scores and static rubrics fail to capture these context-dependent requirements. Existing methods define criteria at the dataset level or generate them in a single pass, which limits their ability to explore the evaluation space implied by each question. We introduce One-Question-One-World (Qworld), a method that generates question-specific evaluation criteria using a recursive expansion tree. Given a question, Qworld decomposes it into scenarios, perspectives, and fine-grained binary criteria through hierarchical and horizontal expansion. The resulting criteria specify what a high-quality answer must address for that question. On HealthBench, Qworld covers 89% of expert-authored criteria and generates 79% novel criteria validated by human experts. Experts rate Qworld criteria higher in insight and granularity than those produced by prior methods. When applied to 11 frontier LLMs on HealthBench and Humanity's Last Exam, Qworld reveals capability differences in dimensions such as long-term impact, equity, error handling, and interdisciplinary reasoning that coarse rubrics do not capture. By generating evaluation criteria for each question, Qworld enables assessment of LLM responses that is tailored to the question rather than based on fixed task-level criteria.

Figures & tables

Figure 1: Given a question, Qworld generates question-specific evaluation criteria, which can be used by downstream evaluators (e.g., LLM-as-judge) to assess responses across diverse contexts.

Figure 2: Recursive expansion tree for generation of evaluation criteria in Qworld (d) in comparison with (a) chain-of-thought, (b) self-reflection generation, and (c) tree-decomposition generation.

	Automatic assessment		Human expert assessment
Method	Coverage $\uparrow$	Uniqueness $\uparrow$	Insight $\uparrow$	Granularity $\uparrow$
TICK	0.46	0.24	0.29	0.79
RocketEval	0.53	0.26	0.42	0.83
OpenRubrics	0.54	0.37	0.36	0.49
EvalAgent	0.83	0.50	0.40	0.65
Qworld	0.89	0.79	0.83	0.85

Table 1: Criteria-quality evaluation on HealthBench. We compare Qworld (and its retrieval-augmented version Qworld

{}_{\text{ret.}}

) against current methods on automatic metrics (Coverage, Uniqueness) and human expert ratings (Insight, Granularity). Qworld achieves best performance across all metrics. All metrics range in

[0,1]

Rank	$\Delta$	Model	$\text{s}_{\text{Healthbench}}$ ( $\%$ )	$\text{s}_{Qworld}$ ( $\%$ )
1	$\rightarrow 0$	GPT-5	62.6	35.4
2	$\uparrow 4$	Qwen3-30B	49.8	34.9
3	$\downarrow 1$	GPT-5-mini	61.7	34.8
4	$\downarrow 1$	DeepSeek-V3.2	53.0	31.8
5	$\rightarrow 0$	Gemini 3 Flash	52.5	30.4
6	$\downarrow 2$	Grok-4.1-Fast	52.5	30.1

Table 2: Performance of LLMs on HealthBench (left) and Humanity’s Last Exam (right).

s_{\text{HealthBench}}

and

s_{\text{HLE}}

are the official benchmark scores;

s_{Qworld}

uses Qworld-generated criteria;

\Delta

denotes rank change. Results with ∗ are taken from official reports. Models marked with

\dagger

are text-only and are evaluated on the text-only subset.

Figure 3: Example of Qworld-generated criteria for a medical question. Compared against expert-authored criteria on a hand numbness question, Qworld covers established requirements (e.g., considering alternative causes) and additionally identifies unique criteria such as warning patients about high-risk activities, demonstrating both high coverage and deeper contextual insight.

Figure 4: Human judges rate the value of Qworld unique criteria above 0.90 as the number of criteria increases, indicating that added criteria remain useful rather than redundant.

Figure 5: Evaluation using a taxonomy grouped from question-level criteria generated by Qworld ( b ), compared with a human-expert taxonomy ( a ) on HealthBench.

Figure 6: Evaluation using a taxonomy grouped from question-level criteria generated by Qworld on HLE. Qworld provides fine-grained evaluation beyond binary accuracy. It focuses on reasoning qualities, showing its context awareness.

Strategy	Cov. $\uparrow$	Uni. $\uparrow$
CoT	0.67	0.40
SR	0.84	0.70
TD	0.77	0.65
RET	0.89	0.79

Table 3: Left : Ablation. RET vs. Chain-of-Thought (CoT), Self-Reflection (SR), Tree Decomposition (TD) across Coverage (Cov.) and Uniqueness (Uni.). Center : Robustness. Coverage under different generator/judge pairs. Right : Scaling. Cov. & Uni. across the GPT-4.1 model family.

Figure 7: Qworld’s expansion efficiency. We track metric performance as the number of criteria grows through iterative expansion, comparing Qworld (blue) against Self-Reflection (orange). At equal criteria counts, Qworld consistently achieves higher scores, demonstrating that gains come from discovering novel evaluation dimensions rather than inflating volume.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Step-by-step illustration of the Recursive Expansion Tree. (a) The generation pipeline: hierarchical decomposition (question $\to$ scenarios $\to$ perspectives $\to$ criteria), horizontal expansion at each level ( $\times\,w_{\ell}$ ), and cross-branch review and consolidation after each decomposition. (b) One fully expanded subtree for a HealthBench heat-rash question, from the question to a scenario, its perspectives, and the resulting binary criteria. See Appendix D.2 for the complete case study: 13 scenarios, 26 perspectives, and 36 Qworld-generated criteria, compared against 14 physician-authored HealthBench criteria.

Figure 9: Average model scores when the same suite of frontier LLMs is evaluated using criteria from different sources. Each bar reports the mean score aggregated over all models; error bars indicate variability across models under the same criteria source. All other automatically generated criteria yield uniformly high scores suggesting coarse or weakly grounded criteria that under-penalize omissions. In contrast, Qworld produces substantially lower scores and larger variance comparable to experts.

Method	Specificity $\uparrow$	Implicitness $\uparrow$
TICK	0.03	0.73
RocketEval	0.09	0.76
OpenRubrics	0.02	0.89
EvalAgent	0.04	0.83
Qworld	0.09	0.87
Qworld ${}_{\text{ret.}}$	0.10	0.89

Appendix

Table 4: Specificity and Implicitness scores for all methods on HealthBench. Both metrics range in

[0,1]

. Specificity is an NIWF-based score measuring domain-specific vocabulary density; Implicitness (

1-\text{WO}

) measures the degree to which a criterion introduces properties not explicitly stated in the instruction. Qworld achieves best performance on both metrics.

Panel of Figure 10 — Figure 10: Ablation on expansion steps. Coverage, Uniqueness, and Specificity as a function of expansion depth $K$ on HealthBench.

Hierarchy	Coverage $\uparrow$	Uniqueness $\uparrow$	# Criteria
3-level (scenario $\to$ perspective $\to$ criterion)	0.919	0.786	42.0
2-level (perspective $\to$ criterion)	0.894	0.770	38.6
1-level (criterion only)	0.811	0.617	18.5

Appendix

Table 5: Ablation on the three-level hierarchy of RET. Removing the scenario level, and further the perspective level, degrades both Coverage and Uniqueness on the 100-question HealthBench subset.

Method	Coverage $\uparrow$	Uniqueness $\uparrow$	Tokens / q
Qworld (ours)	0.919	0.786	98,124
CoT + repeated sampling	0.814	0.487	98,616
Tree of Thoughts + repeated sampling	0.809	0.639	118,137

Appendix

Table 6: Comparison with CoT and Tree of Thoughts extended by sequential repeated-sampling aggregation at matched (or higher) per-question token budgets, on the same 100-question HealthBench subset with the GPT-4.1 judge.

Method	Coverage	Uniqueness	Total tok/q	Calls/q	Est. cost/q
TICK	0.46	0.23	851	1	$0.003
RocketEval	0.45	0.21	705	1	$0.002
OpenRubrics	0.61	0.37	2,526	1	$0.007
Qworld-skill (1-call)	0.87	0.59	4,320	1	$0.016
EvalAgent	0.85	0.50	396,208	95.2	$0.91
Qworld (multi-call)	0.92	0.79	98,124	56.4	$0.38

Appendix

Table 7: Criteria-generation cost on a 100-question HealthBench subset. Top: single-call methods, including Qworld-skill, a one-call distilled version of Qworld. Bottom: multi-call methods. Calls per question serve as a transparent proxy for latency.

Method	GPT-5 score	Judging tok/q	Cost/q ($)	Avg. criteria/q
TICK	0.929	7,673	0.018	4.4
RocketEval	0.926	7,375	0.017	4.2
OpenRubrics	0.900	11,502	0.027	6.6
EvalAgent (EA-Full)	0.759	91,440	0.214	53.8
Qworld (top-5)	0.566	8,714	0.021	5.0
Qworld (top-10)	0.533	17,325	0.041	10.0

Appendix

Table 8: Judging cost and resulting GPT-5 score on 100 HealthBench questions with the GPT-4.1 judge. A lower score indicates a stricter evaluation that surfaces more failures. Qworld’s judging cost is adjustable via the top-

K

criteria by importance weight.

Figure 11: Distributions of generated criteria. Left: per-question criteria count on HealthBench. Right: distribution of criteria points on HealthBench.

Figure 12: Ranking stability and score drops with more criteria. As criteria count increases, model rankings remain largely stable while absolute scores consistently decrease, indicating that added criteria are meaningful (not noise) and impose a stricter evaluation standard that mitigates benchmark saturation.

Panel of Figure 13 — Figure 13: Qworld-generated raw criteria tag overviews on HealthBench (top) and HLE (bottom).

Panel of Table 9 — Table 9: LLM-based judge prompts for Coverage (top) and Uniqueness (bottom). Both implement the judge function $\mathcal{J}$ described above.

Method	Insight (0–100)	Granularity (0–100)
RocketEval	47.4 $\pm$ 10.0	86.7 $\pm$ 10.7
EvalAgent	44.7 $\pm$ 3.8	70.0 $\pm$ 8.9
OpenRubrics	35.3 $\pm$ 2.1	43.0 $\pm$ 10.5
TICK	31.1 $\pm$ 22.2	83.0 $\pm$ 11.2
Qworld	77.7 $\pm$ 6.1	87.3 $\pm$ 10.3

Appendix

Table 10: Same-batch human study: per-method mean

\pm

std across three annotators on the raw 1–3 scale rescaled to 0–100 via

(x-1)\times 50

Dimension	Near-agreement rate	Krippendorff $\alpha$ (ordinal)
Insight	89.4%	0.381
Granularity	96.5%	0.488

Appendix

Table 11: Inter-annotator agreement in the same-batch study. Near-agreement rate is the percentage of items where annotators gave the same or adjacent scores rather than opposite endpoints of the 3-point scale.

Criterion point value	Total number of criteria	$P$ (human Value $=3$ )
5–6	17	0.76
7	43	0.84
8	106	0.86
9	71	0.93
10	87	0.95

Appendix

Table 12: Alignment between Qworld-assigned criterion point values and human importance judgments over 324 criteria from 32 questions. Higher point values correspond to criteria that humans more often rate as highest-value (Value

=3

Criterion	GPT-5 only	Claude only
Explains the mechanism (Eustachian tube), defines terminology in context	✓	—
Differentiates normal popping from symptoms suggestive of underlying pathology	✓	—
Flags unilateral symptoms as more concerning, urges urgent care if sudden hearing change	✓	—
Frames possibilities probabilistically (range of causes, “may indicate”)	✓	—
Clearly states AI’s limitation and refuses to interpret hearing-test results	—	✓
Discourages self-treatment / OTC remedies without professional evaluation	—	✓

Appendix

Table 13: Criteria passed by only one of the two models on the ear-popping question. Both models receive an identical overall score of 0.441.

Aspect	GPT-5 only	Claude only
States 400 mg/dL is a dangerous level requiring immediate action	✓	—
Tells the user not to delay their insulin dose	✓	—
Prompts checking ketones (urine / blood meter)	✓	—
Advises against carbs / sugary food and against driving until controlled	✓	—
Avoids prescription-like dosing language	—	✓
Avoids recommending insulin without confirming type / method	—	✓

Appendix

Table 14: Aspects satisfied by only one of the two models on the diabetes-emergency question. The overall scores are 0.444 (GPT-5) and 0.415 (Claude-Opus-4.5).

Panel of Table 15 — Table 15: Case Study: Scenarios and Perspectives Generated by Qworld for a Heat Rash Management Question (Part 1 of 6). A user in a hot, humid climate asks for prevention tips and portable relief items for recurring neck heat rashes. Given this question, Qworld first identifies 13 distinct scenarios that characterize different contexts in which the question is relevant, then elicits 26 fine-grained perspectives capturing dimensions of response quality. These are subsequently expanded into 36 checkable evaluation criteria (Parts 2–4), compared against 14 physician-authored expert criteria. This decomposition illustrates how Qworld transforms a seemingly simple health query into a rich, multi-dimensional evaluation framework.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering
Feb 27, 2026Rafid Ishrak Jahan, Fahmid Shahriar Iqbal, Sagnik Ray ChoudhuryLLM Evaluation LLM-as-a-Judge
University of North Texas, Denton, Texas, USA
Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
Jun 25, 2026Sangwoo Cho, Kushal Chawla, Pengshan Cai +4LLM Evaluation LLM-as-a-Judge
Capital One, AI Foundations, McLean, VA 22102, USA.
Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation
Oct 14, 2026Haocheng Yang, Licheng Pan, Yiming Dong +7Language Model Generation Evaluation Rubric-Based Evaluation
National University of Singapore · Zhejiang University · Alibaba Group +6

Feb 27, 2026J/K move · Enter open · S save

LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering

Rafid Ishrak Jahan, Fahmid Shahriar Iqbal, Sagnik Ray Choudhury

University of North Texas, Denton, Texas, USA

LLM Evaluation LLM-as-a-Judge Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

Long-form question answering (LFQA) demands nuanced evaluation of multi-sentence explanatory responses, yet existing metrics often fail to reflect human judgment. We present LFQA-HP-1M, a large-scale dataset comprising 1.3M human pairwise preference annotations for LFQA. We propose nine rubrics for answer quality evaluation, and show that simple linear models based on these features perform comparably to state-of-the-art LLM evaluators. We further examine transitivity consistency, positional bias, and verbosity biases in LLM evaluators and demonstrate their vulnerability to adversarial perturbations. Overall, this work provides one of the largest public LFQA preference datasets and a rubric-driven framework for transparent and reliable evaluation.

Abstract

LLM Evaluation

Explore similar work

LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering

Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement

Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation