HARDEN: Constrained Evolutionary Search for Harder, Answer-Preserving Evaluation Cases

Aditya Kumaran; Rahul Singhal; Karime Maamari; Amine Mhedhbi; Pradyumna Tambwekar

cs.AISep 24, 2026

HARDEN: Constrained Evolutionary Search for Harder, Answer-Preserving Evaluation Cases

Authors: Aditya Kumaran, Rahul Singhal, Karime Maamari, Amine Mhedhbi, Pradyumna Tambwekar

Organizations: Distyl AI · Polytechnique Montréal · Mila – Quebec AI Institute

Abstract

Language models are often evaluated on curated benchmarks that underrepresent the complexity of enterprise deployments. We introduce HARDEN, a constrained evolutionary search method to adapt the input of existing evaluation cases into more challenging variants while keeping their expected outputs fixed. HARDEN searches along generated domain-specific complexity axes while enforcing feasibility constraints such as preserving task semantics, realism, and execution validity. Across FinQA, PubMedQA, and ContractNLI and three Qwen3.5 model scales (35B-A3B, 122B-A10B, and 397B-A17B), HARDEN reduces task-model accuracy by 22.7% on average and by up to 49.9% relative to single-pass baselines using the same feasibility checks. These results show that evolutionary search can produce substantially harder valid evaluation cases.

Figures & tables

Figure 1: Task model accuracy by mutation method and model size. Lower accuracy indicates more effective hardening. Error bars are 95% case-bootstrap CIs ( Efron and Tibshirani, 1993 ) based on ten responses per case.

Figure 2: Mean-normalized discrete semantic entropy by mutation method and task model size. Higher indicates greater task-model output uncertainty. Error bars show 95% case-bootstrap confidence intervals.

Figure 3: Successfully hardened cases (out of 200) by mutation method and task-model size. A case is counted when it passes both LM-based checks and reduces sampled task-model accuracy relative to its original.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Dimension	Operational high-score anchor
D1	Tabular numeric density and structure	A genuine financial table, typically with at least eight labeled line items, at least two value columns, and at least 25% numeric cells.
D2	Financial line-item and disclosure vocabulary	At least six distinct financial terms or line items, such as revenue, cost of sales, operating income, total assets, cash flow, or fiscal year.
D3	Authentic accounting numeric formatting	At least three authentic conventions, including thousands separators, parenthesized negatives, currency or scale markers, period headers, percentages, per-share values, or footnote markers.
D4	Grounded numerical-reasoning intent	The question names a metric or period present in the report and requests a computable value, change, difference, ratio, percentage, or aggregate.

Appendix

Table 1: The target-blind “Financial-report numerical-reasoning input realism rubric” used in our FinQA experiment. Scores use a 1–5 scale.

Axis	Example mutation
Table-label compression	Replace verbose row and column descriptions with authentic filer shorthand.
Table-lookup ambiguity	Add plausible sibling rows, subtotals, or comparative-period columns.
Numeric-format heterogeneity	Mix currency symbols, separators, parenthetical negatives, and dash zeros.
Financial-terminology variation	Use standard financial synonyms across the question and source.
Narrative-distractor density	Add non-decisive figures and footnote detail from related disclosures.
Cross-reference indirection	Route a fact through a resolvable footnote or remove a duplicate restatement.

Appendix

Table 2: Generated FinQA complexity-axis catalog, with abbreviated examples.

Original

HARDEN

MUTUAL NON-DISCLOSURE AGREEMENT Each undersigned party (the “Receiving Party”) understands that…

MUTUAL NON-DISCLOSURE AGREEMENT WITNESSETH: WHEREAS , CombiChem, Inc., a Delaware corporation, and DuPont Pharmaceuticals Company, a Delaware general partnership (each a “party” and collectively the “parties”), are engaged in discussions concerning a possible business transaction of mutual interest; and WHEREAS , in the course of such discussions each party may find it desirable to disclose to the other certain of its proprietary and confidential information…; and WHEREAS , the parties desire to establish the terms and conditions governing the disclosure, protection and use of such information; NOW, THEREFORE , in consideration of the mutual covenants and promises contained herein…, the parties agree as set forth below. Each undersigned party (the “Receiving Party”) understands that…

IN WITNESS WHEREOF, the parties have executed this Agreement as of the day and year set forth below. Date: 3-10-99 …

IN WITNESS WHEREOF, the parties have caused this Agreement to be executed by their respective duly authorized representatives as of the day and year set forth below. Date: 3-10-99 … Address for Notices: CombiChem, Inc., Attn: General Counsel, San Diego, California DuPont Pharmaceuticals Company, Attn: Legal Department, Wilmington, Delaware

Appendix

Table 3: Excerpt of a ContractNLI case before and after evolution. The mutation prepends authentic recitals and pads the execution formalities of a real mutual NDA; ellipses mark elided unchanged text.

Benchmark	Task model	Original	Few-Shot	Few-Shot (web-search)	HARDEN
FinQA	Qwen 3.5 35B	71.6	69.1 (69.8)	66.3 (68.5)	49.4
FinQA	Qwen 3.5 122B	72.4	69.5 (71.1)	69.4 (70.0)	59.6
FinQA	Qwen 3.5 397B	73.8	72.1 (72.7)	72.1 (72.3)	60.5
PubMedQA	Qwen 3.5 35B	76.1	80.6 (80.2)	81.9 (81.9)	50.4
PubMedQA	Qwen 3.5 122B	77.7	81.3 (80.7)	83.4 (83.4)	33.5
PubMedQA	Qwen 3.5 397B	76.8	82.1 (81.5)	82.1 (82.0)	37.9

Appendix

Table 4: Task model accuracy (%) under each mutation method. Few-Shot and Few-Shot (web-search) entries report unfiltered (constraint-filtered) accuracy: first on all 200 mutations, then after cases that fail either LM-based check revert to the original case (Appendix A.8 ).

Benchmark	Task model	Original	Few-Shot	Few-Shot (web-search)	HARDEN
FinQA	Qwen 3.5 35B	0.17	0.22 (0.20)	0.26 (0.21)	0.44
FinQA	Qwen 3.5 122B	0.14	0.15 (0.13)	0.17 (0.15)	0.32
FinQA	Qwen 3.5 397B	0.09	0.13 (0.11)	0.12 (0.10)	0.22
PubMedQA	Qwen 3.5 35B	0.16	0.11 (0.11)	0.14 (0.14)	0.35
PubMedQA	Qwen 3.5 122B	0.11	0.10 (0.10)	0.10 (0.10)	0.39
PubMedQA	Qwen 3.5 397B	0.08	0.09 (0.08)	0.10 (0.10)	0.32

Appendix

Table 5: Mean normalized discrete semantic entropy under each mutation method. Sample counts follow Table 4 . Few-Shot and Few-Shot (web-search) entries use the same unfiltered (constraint-filtered) convention as Table 4 .

Benchmark	Task model	Original	Few-Shot	Few-Shot (web-search)	HARDEN
FinQA	Qwen 3.5 35B	0.17	0.22 (0.20)	0.26 (0.21)	0.44
FinQA	Qwen 3.5 122B	0.13	0.15 (0.13)	0.17 (0.15)	0.31
FinQA	Qwen 3.5 397B	0.09	0.13 (0.11)	0.12 (0.10)	0.22
PubMedQA	Qwen 3.5 35B	0.15	0.11 (0.11)	0.13 (0.13)	0.34
PubMedQA	Qwen 3.5 122B	0.11	0.10 (0.10)	0.10 (0.10)	0.39
PubMedQA	Qwen 3.5 397B	0.08	0.09 (0.08)	0.10 (0.10)	0.32

Appendix

Table 6: Mean likelihood-weighted semantic entropy under each mutation method. Few-Shot and Few-Shot (web-search) entries report unfiltered (constraint-filtered) values.

Benchmark	Task model	Original	Few-Shot	Few-Shot (web-search)	HARDEN
FinQA	Qwen 3.5 35B	100.9	106.2 (94.0)	113.8 (92.6)	140.8
FinQA	Qwen 3.5 122B	61.8	83.5 (66.7)	78.8 (65.6)	70.3
FinQA	Qwen 3.5 397B	22.3	20.3 (21.5)	30.9 (25.7)	25.1
PubMedQA	Qwen 3.5 35B	56.7	58.8 (58.5)	66.8 (66.6)	78.0
PubMedQA	Qwen 3.5 122B	48.1	47.1 (47.0)	52.3 (52.1)	56.6
PubMedQA	Qwen 3.5 397B	16.7	19.8 (19.4)	24.2 (24.2)	23.5

Appendix

Table 7: Mean sequence negative log-likelihood of the greedy completion under each mutation method. Higher values indicate lower sequence likelihood and are sensitive to completion length. Few-Shot and Few-Shot (web-search) entries report unfiltered (constraint-filtered) values.

Benchmark	Task model	Original	Few-Shot	Few-Shot (web-search)	HARDEN
FinQA	Qwen 3.5 35B	0.101	0.103 (0.101)	0.097 (0.096)	0.111
FinQA	Qwen 3.5 122B	0.076	0.080 (0.078)	0.075 (0.072)	0.081
FinQA	Qwen 3.5 397B	0.032	0.027 (0.029)	0.032 (0.031)	0.034
PubMedQA	Qwen 3.5 35B	0.210	0.210 (0.209)	0.212 (0.212)	0.227
PubMedQA	Qwen 3.5 122B	0.150	0.149 (0.149)	0.146 (0.146)	0.167
PubMedQA	Qwen 3.5 397B	0.055	0.051 (0.051)	0.053 (0.053)	0.059

Appendix

Table 8: Mean token negative log-likelihood of the greedy completion under each mutation method. This is sequence NLL divided by token count; it is not exponentiated perplexity. Higher values indicate lower mean token likelihood. Few-Shot and Few-Shot (web-search) entries report unfiltered (constraint-filtered) values.

Benchmark	Task model	Hardened cases	Accuracy (%)	Semantic entropy
FinQA	Qwen 3.5 35B	126 of 200	$92.2\rightarrow 57.0$	$0.13\rightarrow 0.56$
FinQA	Qwen 3.5 122B	101 of 200	$93.4\rightarrow 68.1$	$0.09\rightarrow 0.45$
FinQA	Qwen 3.5 397B	61 of 200	$86.2\rightarrow 42.5$	$0.17\rightarrow 0.57$
PubMedQA	Qwen 3.5 35B	121 of 200	$78.3\rightarrow 36.0$	$0.24\rightarrow 0.57$
PubMedQA	Qwen 3.5 122B	142 of 200	$92.1\rightarrow 29.9$	$0.12\rightarrow 0.52$
PubMedQA	Qwen 3.5 397B	122 of 200	$94.5\rightarrow 30.7$	$0.10\rightarrow 0.49$

Appendix

Table 9: Original-versus- HARDEN comparison restricted to successfully hardened cases (non-hardened cases excluded).

Panel of Figure 4 — Figure 4: Redacted correctness-study introduction screens for FinQA (top) and PubMedQA (bottom). The crops omit the embedded video preview and show no participant or platform identifiers.

Panel of Figure 5 — Figure 5: Redacted realism-study introduction screens for FinQA (top) and PubMedQA (bottom), including the benchmark-specific realism definitions and task descriptions. No participant or platform identifiers are shown.

Panel of Figure 6 — Figure 6: Participant-facing response controls. The original-case explanation screen (top) is followed by the modified-case correctness judgment and justification screen (middle); the realism judgment and rationale screen is shown at bottom. These controls are identical across FinQA and PubMedQA, so only one capture of each is included.

Benchmark	Source	Agentic check	Human majority
FinQA	Official	90%	60%
FinQA	GPT-5.6 Sol	95%	70%
FinQA	GPT-3.5 Turbo	0%	60%
PubMedQA	Official	100%	95%
PubMedQA	GPT-5.6 Sol	100%	100%
PubMedQA	GPT-3.5 Turbo	90%	60%

Appendix

Table 10: Percentage of cases labeled realistic by the agentic check and the two-of-three participant majority. Each row contains 20 cases.

Benchmark	Agreement	Cohen’s $\kappa$	Gwet’s AC1	Positive agr.	Negative agr.	$\phi$ /MCC
FinQA	51.7%	$-0.031$	0.090	61.3%	35.6%	$-0.031$
PubMedQA	85.0%	0.135	0.820	91.7%	18.2%	0.182

Appendix

Table 11: Realism agreement between the agentic check and participant majority over 60 cases per benchmark.

Benchmark	TP/FN/FP/TN	Accuracy	$\phi$ /MCC	Cohen’s $\kappa$	Specificity	McNemar Holm $p$
FinQA	10/0/9/1	55.0%	0.229	0.100	10.0%	0.0039
PubMedQA	10/0/10/0	50.0%	n/a	0.000	0.0%	0.0039

Appendix

Table 12: Correctness agreement on 20 gold-labeled mutated pairs per benchmark. Confusion counts are TP/FN/FP/TN with correct as positive.

Benchmark	Manual incorrect	Counterfactual	Label flip	Manual correct
FinQA	5/8	5/5	5/5	3/3

Appendix

Table 13: Correctness-check agreement on the co-author-agreement evaluation set, by construction method. Numerators are matches to the agreed co-author verdict; denominators exclude split judgments.

Benchmark	TP/FN/FP/TN	Accuracy (95% CI)	Correct recall	Incorrect specificity
FinQA	3/1/2/15	85.7% (65.4–95.0)	75.0%	88.2%

Appendix

Table 14: Secondary correctness performance on the co-author-agreement evaluation set, with correct as the positive class. Confidence intervals are Wilson 95% intervals for accuracy.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Brittlebench: Quantifying LLM robustness via prompt sensitivity
Feb 27, 2026Angelika Romanou, Mark Ibrahim, Candace Ross +8Prompt Sensitivity LLM Evaluation
FAIR at Meta · EPFL · Northeastern University
Self-Improving Language Models with Bidirectional Evolutionary Search
May 27, 2026Guowei Xu, Zhenting Qi, Huangyuan Su +4Evolutionary Optimization LLM Planning
Harvard University · MIT
QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks
Apr 20, 2026Taylor Lundy, Narun K. Raman, Kevin Leyton-BrownLLM Evaluation Bayesian Optimization
Department of Computer Science University of British Columbia Vancouver, BC

Feb 27, 2026J/K move · Enter open · S save

Brittlebench: Quantifying LLM robustness via prompt sensitivity

Angelika Romanou, Mark Ibrahim, Candace Ross, Kerem Oktar+7

FAIR at Meta · EPFL · Northeastern University

Prompt Sensitivity LLM Evaluation Language Model Robustness

Open paper AskPDF arXiv

Existing evaluation methods largely rely on clean, static benchmarks, which can overestimate true model performance by failing to capture the noise and variability inherent in real-world user inputs. This is especially true for language models, which can face human-generated text queries containing mistakes, typos, or alternative ways of phrasing the same question. In this work, we introduce a theoretical framework for quantifying model sensitivity to prompt variants, or brittleness, that can enable us to disentangle data-induced difficulty from prompt-related variability. Using this framework, we design a novel evaluation pipeline, Brittlebench, to holistically evaluate the sensitivity of frontier models. We apply semantics-preserving perturbations to a suite of popular benchmarks, and observe model performance to degrade as much as 12%. However, these perturbations do not affect all models equally: even a single perturbation alters the relative ranking of models in 63% of cases, impacting conclusions about comparative model performance. Decomposing the total variance of both state-of-the-art open-weight and commercial models, we find that semantics-preserving input perturbations can account for up to half of the performance variance for a given model. Brittlebench highlights the need for more robust evaluations and models, and allows us to systematically understand model brittleness.

Abstract

LLM Evaluation

Explore similar work

Brittlebench: Quantifying LLM robustness via prompt sensitivity

Self-Improving Language Models with Bidirectional Evolutionary Search

QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks