LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering

Rafid Ishrak Jahan; Fahmid Shahriar Iqbal; Sagnik Ray Choudhury

cs.CLFeb 27, 2026

LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering

Authors: Rafid Ishrak Jahan, Fahmid Shahriar Iqbal, Sagnik Ray Choudhury

Organizations: University of North Texas, Denton, Texas, USA

Abstract

Long-form question answering (LFQA) demands nuanced evaluation of multi-sentence explanatory responses, yet existing metrics often fail to reflect human judgment. We present LFQA-HP-1M, a large-scale dataset comprising 1.3M human pairwise preference annotations for LFQA. We propose nine rubrics for answer quality evaluation, and show that simple linear models based on these features perform comparably to state-of-the-art LLM evaluators. We further examine transitivity consistency, positional bias, and verbosity biases in LLM evaluators and demonstrate their vulnerability to adversarial perturbations. Overall, this work provides one of the largest public LFQA preference datasets and a rubric-driven framework for transparent and reliable evaluation.

Figures & tables

Figure 1: Workflow for our framework: from multisource dataset aggregation and LFQA filtering to rubric-based evaluation and model comparison.

Sample Set	Model	Precision	Recall	F1 Score
First 100 initial definition	llama-4	0.7500	0.2553	0.3810
	gpt-4o	0.9091	0.2128	0.3448
	gemini-2.5	0.8750	0.2979	0.4444
Second 100 initial definition	llama-4	0.6905	0.6558	0.6528
	gpt-4o	0.7133	0.6485	0.6382
	gemini-2.5	0.7313	0.7346	0.7293

Table 1: LLMs evaluation metrics (Precision, Recall, F1) on the first and second 100 samples, with both initial and updated definition.

Figure 2: Effect of batch size on filtering performance for Llama-4. Batch sizes above 10 provide marginal cost gains but a large performance drop.

Rubrics	Definition	Score range
Specificity	measures the precision and depth in explanation over general statements.	0/1/2, CT
Grammar	examines grammatical correctness, adhering to the formal writing style.	1-5, CO
Fluency	measures how clearly the answer communicates.	1-3, Co
Completeness	checks whether the answer addresses all the queries asked by the question.	1-5, CT
Coherence	captures how logically and cohesively the answer is organized.	1-5, CO
Relevance	evaluates whether the answer directly addresses the given question.	1-5, CO

Table 2: LFQA evaluation rubric definitions and score ranges. CT/CO denotes whether scores are categorical or continuous.

Feature (Rubric Aspect)	Weight
Specificity	0.0420
Grammar	-0.0378
Fluency	0.0919
Completeness	0.4065
Coherence	0.5910
Relevance	-0.3082

Table 3: Learned LR weights indicate contribution toward human preference

Model	Acc.	Prec.	Rec.	F1
Logistic Regression	0.6826	0.6821	0.6816	0.6817
LLM-as-a-judge
GPT-4o	0.6943	0.6970	0.6960	0.6942
Gemini-2.5-flash	0.6826	0.6864	0.6847	0.6822
Llama-4-Scout	0.6840	0.6899	0.6868	0.6833
Rubrics

Table 4: Comparative performance of LLM-based evaluators and logistic regression models

Choices	Transitive	Non-Transitive	Not Testable
A $\mid$ B	A	A	B
B $\mid$ C	B	B	B
C $\mid$ A	A	C	–

Table 5: Illustration of transitive, cyclic (3-cycle), and non-testable preference cases. In the final column, the model selects option B in two cases, thereby rendering the third option irrelevant for transitivity detection.

Model	$\Delta$ Acc.	$\Delta$ Prec.	$\Delta$ Rec.	$\Delta$ F1
GPT-4o	+0.0052	+0.0043	+0.0009	+0.0024
Gemini-2.5-flash	-0.0196	-0.0201	-0.0252	-0.0238
Llama-4-Scout	-0.0168	-0.0190	-0.0228	-0.0208

Table 6: Change (

\Delta

) in performance metrics after introducing positional bias

Model	Human Short $\rightarrow$ LLM Long	Human Long $\rightarrow$ LLM Short	Bias Direction
GPT-4o	248	171	Long-biased
Llama-4	232	199	Nearly balanced
Gemini-2.5-flash	248	184	Long-biased

Table 7: Cross-bias instances where model choice conflicted with human-correct answer length.

Perturbation	$\Delta$ Acc.(%)	$\Delta$ Prec.(%)	$\Delta$ Rec.(%)	$\Delta$ F1(%)
TextFooler	-2.64%	-2.60%	-2.61%	-2.65%
DeepWordBug	-10.67%	-11.05%	-10.89%	-10.69%

Table 8: Relative performance degradation (

\Delta

%) of GPT-4o under adversarial perturbations.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Acc.	Prec.	Rec.	F1
Logistic Regression	0.6826	0.6821	0.6816	0.6817
LLM-as-a-judge
GPT-4o	0.6818	0.6841	0.6834	0.6817
Gemini-2.5-flash	0.6926	0.6962	0.6947	0.6923
Llama-4-Scout	0.6891	0.6941	0.6917	0.6887
Rubrics

Appendix

Table 9: Performance analysis of LLM evaluators at

T=1.0

without majority voting

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Qworld: Question-Specific Evaluation Criteria for LLMs
Mar 6, 2026Shanghua Gao, Yuchang Su, Pengwei Sui +2LLM Evaluation
Department of Biomedical Informatics, Harvard Medical School · Department of Medicine, Brigham and Women’s Hospital · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University +2
How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation
Sep 1, 2026Elitsa Yotkova, Violeta Kastreva, Petar Velkov +4LLM Answer Verification Language Model Generation Evaluation
Sofia University “St. Kliment Ohridski” · ETH Zürich · University of Zurich +1
Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users
Jun 18, 2026Haw-Shiuan Chang, Jeffrey Gomez, Mehul Patwari +2Reward Modeling LLM Alignment
University of Massachusetts, Amherst, USA · York University, Canada

Mar 6, 2026J/K move · Enter open · S save

Qworld: Question-Specific Evaluation Criteria for LLMs

Shanghua Gao, Yuchang Su, Pengwei Sui, Curtis Ginder+1

Department of Biomedical Informatics, Harvard Medical School · Department of Medicine, Brigham and Women’s Hospital · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University+2

LLM Evaluation

Open paper AskPDF arXiv

Evaluating large language models (LLMs) on open-ended questions is difficult because response quality depends on the question's context. Binary scores and static rubrics fail to capture these context-dependent requirements. Existing methods define criteria at the dataset level or generate them in a single pass, which limits their ability to explore the evaluation space implied by each question. We introduce One-Question-One-World (Qworld), a method that generates question-specific evaluation criteria using a recursive expansion tree. Given a question, Qworld decomposes it into scenarios, perspectives, and fine-grained binary criteria through hierarchical and horizontal expansion. The resulting criteria specify what a high-quality answer must address for that question. On HealthBench, Qworld covers 89% of expert-authored criteria and generates 79% novel criteria validated by human experts. Experts rate Qworld criteria higher in insight and granularity than those produced by prior methods. When applied to 11 frontier LLMs on HealthBench and Humanity's Last Exam, Qworld reveals capability differences in dimensions such as long-term impact, equity, error handling, and interdisciplinary reasoning that coarse rubrics do not capture. By generating evaluation criteria for each question, Qworld enables assessment of LLM responses that is tailored to the question rather than based on fixed task-level criteria.

Abstract

LLM Evaluation

Explore similar work

Qworld: Question-Specific Evaluation Criteria for LLMs

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users