Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation

Aashish Anantha Ramakrishnan; Ardavan Saeedi; Hamid Reza Hassanzadeh; Fazlolah Mohaghegh; Dongwon Lee

cs.CLFeb 26, 2026

Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation

Authors: Aashish Anantha Ramakrishnan, Ardavan Saeedi, Hamid Reza Hassanzadeh, Fazlolah Mohaghegh, Dongwon Lee

Organizations: The Pennsylvania State University · Optum AI

Abstract

With the widespread adoption of pre-trained Large Language Models (LLM), there exists a high demand for task-specific test sets to benchmark their performance in domains such as healthcare and biomedicine. However, the cost of labeling test samples while developing new benchmarks poses a significant challenge, especially when expert annotators are required. Existing frameworks for active sample selection offer limited support for generative Question Answering tasks, where option dynamics can affect model decision boundaries. In this paper, we present Generative Active Testing (GAT), an uncertainty-aware acquisition framework leveraging LLMs as surrogates for informing the sample selection process. Using a novel Statement Adaptation Module, we modify generative tasks into a pseudo-classification format, enabling the capture of sample-level uncertainties across unlabeled candidates. Our zero-shot acquisition functions reduce estimation error by ~40% compared to traditional sampling baselines, offering a scalable solution for cost-effective model benchmarking.

Figures & tables

Figure 1: An overview of the components of the Generative Active Testing Framework.

Figure 2: Step-wise flow diagram detailing tasks performed by the Statement Adaptation Module.

	$\Delta$ Conditional AU-ROC (Conf $\geq 0.7$ )
Strategy	PubMedQA	MedQA	AI2_ARC
MostConf	+3.9%	-6.1%	-49.3%
LeastConf	-49.9%	+25.6%	+11.0%
RunnerUp	+2.1%	+4.9%	+3.3%
Subset Size ( $N$ )	352 (70%)	394 (31%)	363 (59%)

Table 1: Ablation of Option Selection Strategies. Values represent the change (

\Delta

) in Conditional AUROC (Conf

\geq 0.7

), with the RunnerUp strategy showing consistent, robust improvements across all datasets.

Metric	PubMedQA	MedQA	AI2_ARC
Accuracy (QA)	0.74	0.59	0.80
Accuracy (Stmt)	0.77	0.64	0.74
Entropy $\sigma$ (QA)	0.25	0.41	0.41
Entropy $\sigma$ (Stmt)	0.16	0.12	0.09
Cond. AUROC (QA)	0.693	0.687	0.769
Cond. AUROC (Stmt)	0.707	0.721	0.794

Table 2: Impact of RunnerUp Statement Adaptation. Compared to the QA task, the statement adapted task has a lower spread of entropy

\sigma

values.

Panel of Figure 3 — Figure 3: Variance and Average Estimation Error. Results across acquisition runs on the PubMedQA dataset. Experimental settings: $\mathcal{F}_{\text{main}}=\text{Llama 3.1 8B}$ , $\mathcal{F}_{\text{surrogate}}=\text{NLIVerifier}$ , and $\mathcal{F}_{\text{oracle}}=\text{RunnerUp}$ .

Acquisition Function	AGNews $\mathcal{F}_{main}=$ Llama 3.1 8B	AGNews $\mathcal{F}_{main}=$ BARTClassifier
RandomAcq	0.00095	0.00031
SelfEntropyAcq	0.00153	0.00037
UCBAcq	0.00044	0.00032
MultiLM_CEAcq	0.00096	0.00022
UniformLM_CEAcq	0.00066	0.00024
MultiLMUniform_CEAcq	0.00045	0.00032

Table 3: Estimation Error AUC (

\downarrow

). Performance of acquisition functions on traditional text classification datasets such as AGNews.

Acquisition Function	PubMedQA	MedQA	AI2_ARC	Geo Mean AUC
RandomAcq	0.00339	0.00061	0.00058	0.00106
SelfEntropyAcq	0.01526	0.00055	0.00094	0.00199
LMClusterEntropyAcq	0.00531	0.00117	0.00049	0.00145
UCBAcq	0.01772	0.00099	0.00069	0.00230
MultiLM_CEAcq	0.00425	0.00051	0.00126	0.00140
UniformLM_CEAcq	0.00160	0.00041	0.00039	0.00064

Table 4: Estimation Error AUC (

\downarrow

). Performance of acquisition functions, where the geometric mean highlights robustness across multiple datasets. Experimental settings:

\mathcal{F}_{\text{main}}=\text{Llama 3.1 8B}

\mathcal{F}_{\text{surrogate}}=\text{NLIVerifier}

, and

\mathcal{F}_{\text{oracle}}=\text{RunnerUp}

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Acquisition Function	PubMedQA	MedQA	AI2_ARC
RandomAcquisition	0.00339	0.00061	0.00058
LMClusterAcquisition (Unbiased Estimator)	0.00531	0.00117	0.00049
LMClusterAcquisition (Biased Estimator)	0.00166	0.00084	0.00062

Appendix

Table 5: Estimation Error AUC (

\downarrow

). Performance of different Cluster Acquisition functions. Experimental settings:

\mathcal{F}_{\text{main}}=\text{Llama 3.1 8B}

\mathcal{F}_{\text{surrogate}}=\text{NLIVerifier}

, and

\mathcal{F}_{\text{oracle}}=\text{LeastConf}

Metric	PubMedQA	MedQA	AI2_ARC
Accuracy (QA)	0.75	0.59	0.80
Accuracy (Stmt)	0.44	0.77	0.82
Entropy $\sigma$ (QA)	0.25	0.41	0.41
Entropy $\sigma$ (Stmt)	0.18	0.18	0.12
Cond. AUROC (QA)	0.693	0.687	0.769
Cond. AUROC (Stmt)	0.347	0.863	0.853

Appendix

Table 6: Impact of Least Confident Statement Adaptation. Unlike the RunnerUp strategy, picking the least confident option results in high variance. While it improves discrimination on MedQA and AI2_ARC, it causes a catastrophic collapse in performance on PubMedQA (AUROC 0.69

\rightarrow

0.35), making it unreliable for general deployment.

Metric	PubMedQA	MedQA	AI2_ARC
Accuracy (QA)	0.75	0.59	0.80
Accuracy (Stmt)	0.69	0.60	0.43
Entropy $\sigma$ (QA)	0.25	0.41	0.41
Entropy $\sigma$ (Stmt)	0.23	0.10	0.07
Cond. AUROC (QA)	0.693	0.687	0.769
Cond. AUROC (Stmt)	0.720	0.645	0.390

Appendix

Table 7: Impact of Most Confident Statement Adaptation. Verifying the model’s top choice fails to consistently improve calibration. On AI2_ARC, the method suffers a severe collapse (AUROC 0.77

\rightarrow

0.39), and accuracy drops significantly, confirming that this strategy is prone to confirming the model’s original errors.

Model	Model ID
Llama 3.1 8B	meta-llama/Meta-Llama-3-8B
NLI Verifier	soumyasanyal/nli-entailment-verifier-xxl
GPT-4o	gpt-4o-2024-07-01-preview (Azure OpenAI)

Appendix

Table 8: Overview of models evaluated for Active Testing. For open-source models, this table shows the model names in Huggingface.

Acquisition Function	PubMedQA
RandomAcq	0.01289
AccuracyAcq	0.03950
SelfEntropyAcq	0.06692
UCBAcq	0.02147
UniformLM_CrossEntropyAcq	0.02337
UniformLM_JSDAcq	0.03831

Appendix

Table 9: Estimation Error AUC (

\downarrow

). Performance of acquisition functions without statement adaptation on the PubMedQA dataset. Experimental settings:

\mathcal{M}_{\text{main}}=\text{Llama 3.1 8B}

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Active Testing of Large Language Models via Approximate Neyman Allocation
May 11, 2026Zeli Liu, Jiancheng Zhang, Cong Liu +1LLM Evaluation Language Model Generation Evaluation
University of California, Riverside
Which LLM to pick? Online Active Model Selection for Large Language Models
Oct 1, 2026Alessandro Turrin, Patrik Okanovic, Torsten Hoefler +1Model Selection LLM Evaluation
TU Delft · ETH Zurich
Large Language Model Selection with Limited Annotations
May 24, 2026Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch +2Model Selection LLM Evaluation
TU Delft · ETH Zurich

May 11, 2026J/K move · Enter open · S save

Active Testing of Large Language Models via Approximate Neyman Allocation

Zeli Liu, Jiancheng Zhang, Cong Liu, Yinglun Zhu

University of California, Riverside

LLM Evaluation Language Model Generation Evaluation Semantic Entropy Generative AI Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) require reliable evaluation from pre-training to test-time scaling, making evaluation a recurring rather than one-off cost. As model scales grow and target tasks increasingly demand expert annotators, both the compute and labeling costs needed for each evaluation rise rapidly. Active testing aims to alleviate this bottleneck by approximating the evaluation result from a small but informative subset of the evaluation pool. However, existing approaches primarily target classification and break down on generative tasks. We introduce a novel active testing algorithm tailored to generative tasks. Our method leverages semantic entropy from surrogate models to stratify the evaluation pool and then conducts approximate Neyman allocation based on signals extracted from these surrogates. Across multiple language and multimodal benchmarks and a range of surrogate-target model pairs, our method significantly improves on baselines and closely tracks Oracle-Neyman, delivering up to 28% MSE reduction over Uniform Sampling and an average of 22.9% budget savings.

Abstract

LLM Evaluation

Explore similar work

Active Testing of Large Language Models via Approximate Neyman Allocation

Which LLM to pick? Online Active Model Selection for Large Language Models

Large Language Model Selection with Limited Annotations