How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure

Dipankar Sarkar

cs.CLSep 24, 2026

How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure

Authors: Dipankar Sarkar

Organizations: Skelf Research

Abstract

Evaluations of LLM systems routinely average over small prompt sets and report models as a ranked table. We ask how much confidence such a table deserves, using LLM-based prompt-structure inference as the case study: eight open model variants across five families and 8B to 675B parameters, caching disabled, 293 raw intermediate representations persisted. The measured phenomenon is unstable to begin with. Identical calls do not reliably recover identical structure, with mean node-set Jaccard from 0.39 to 0.96 and 72% of prompt-model cells never node-set-perfect. Auditing the evaluation weakens its conclusions further, and this is our main contribution. Under a joint cluster bootstrap over prompts, only the bottom of the ranking is firm: the two least reproducible models hold rank in 99% and 86% of replicates, the middle four in 27% to 48%, and the top two in 68% each, so the table identifies the worst model reliably but does not reliably identify the best. Two equally defensible rules for merging repeated campaigns change four of eight rows and move the study-wide headline by 7 percentage points. Checking the inferred structure against ground-truth annotations shows reproducibility cannot be read as accuracy. And four of the eight endpoints were withdrawn within ten weeks of measurement, so the study as specified can no longer be run. Small-sample LLM evaluations can therefore look far more definitive than their evidence supports. We recommend reporting rank stability, per-cell provenance, executed sensitivity comparisons, raw per-run outputs, and a measurement date alongside any ranking.

Figures & tables

Model	Family	$n$	Jaccard [95% CI]	ID-stab [95% CI]	Perfect	Rank held
minimax-m2.1	MiniMax	3	0.96 [0.89, 1.00]	0.96 [0.89, 1.00]	2/3	68%
gemma3:12b	Google	22	0.94 [0.90, 0.98]	0.92 [0.85, 0.98]	15/22	68%
mistral-large-3:675b	Mistral	19	0.80 [0.73, 0.86]	0.71 [0.62, 0.80]	3/19	48%
ministral-3:8b	Mistral	19	0.77 [0.66, 0.87]	0.71 [0.57, 0.83]	6/19	27%
gpt-oss:20b	OpenAI-style	20	0.76 [0.65, 0.85]	0.69 [0.56, 0.80]	4/20	47%
minimax-m2.7	MiniMax	6	0.70 [0.47, 0.89]	0.68 [0.44, 0.89]	2/6	48%

Table 1: Stability across eight model variants on the synthesised corpus, sorted by Jaccard.

n

is prompts completing at least 2 of

R

runs. Brackets are 95% percentile bootstrap intervals over 10,000 prompt resamples. "Rank held" is the fraction of joint-cluster-bootstrap replicates in which the model keeps the position shown here; prompts are resampled once per replicate and all models scored on the same draw, so the pairing is preserved.

Claim or assumption	Audit	Outcome
gpt-oss:120b is the least reproducible model	joint cluster bootstrap (§ 5 )	Holds. Stays last in 99% of replicates
minimax-m2.1 is the most reproducible model	joint cluster bootstrap (§ 5 )	Not supported. Holds rank in 68%; which of the top two leads is undetermined
The reported ranking is robust to campaign-merging choices	alternative merge rule (§ 5 )	Sensitive. Four of eight rows change; headline moves 7.1 points
Repeated inference stabilises after $r$ runs	tolerance specification (Appendix F )	Specification-dependent. 80 vs 50 of 98 cells under two readings of one flag
DeepSeek and Zhipu are structurally inaccessible	harness audit (§ 7 )	Withdrawn. Instrument is the leading candidate; cause unresolved
A stable IR is a correct IR	ground-truth annotations (§ 6 )	Not supported. The two orderings diverge

Table 2: Seven claims and assumptions examined in our self-audit. Only one remains strongly supported. Each row is established in the section named; none required new measurements.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Role in the study	Status on 2026-08-17
gpt-oss:20b	Table 1	available
gpt-oss:120b	Table 1	available
mistral-large-3:675b	Table 1	available
minimax-m2.7	Table 1	available
gemma3:12b	Table 1	retired 2026-07-15
ministral-3:8b	Table 1	retired 2026-07-15

Appendix

Table 3: Availability of every model named in this study, re-queried on 2026-08-17. Two bulk retirement waves are visible. Of the eight variants in Table 1 , four can no longer be queried; of the fifteen models named anywhere in the study, ten cannot.

Explore similar work

Sep 29, 2026cs.CL

Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation

Reproducibility is essential for scientific research, yet prior work shows that LLM outputs vary with hardware and batching. We identify an overlooked factor: the hidden injection of the current date into system prompts, which users cannot control and which changes every day. Across 9 recent LLMs and 6 datasets spanning multiple-choice QA (MCQA), math reasoning, code generation, and machine translation, performance varies solely with the current date, with deltas of up to 6% on MCQA, 14% on math reasoning, 7% on code generation, and 2.84 BLEU on machine translation. Model rankings also shift, affecting leaderboards. This date effect exceeds other sources of non-determinism, such as batch size and numerical precision. Standard prompting techniques -- chain-of-thought and few-shot prompting -- do not reduce the sensitivity; chain-of-thought even amplifies it. Our findings underscore the need for careful evaluation protocols to ensure reproducibility and fair comparisons in LLM research.

Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager +1

Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Mexico · University of Colorado Boulder, USA

Reproducibility Prompt Engineering Multiple-Choice Questions

Ask PDF

Jun 16, 2026cs.CL

Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs

Evaluating large language models (LLMs) is important for understanding their capabilities, comparing competing systems, and supporting the deployment of reliable models in practice. For open-ended tasks, pairwise evaluation has become a popular paradigm, in which two responses to the same prompt are compared and the resulting judgments are aggregated into an overall ranking. A central challenge of this paradigm is intransitivity: the induced comparison outcomes may fail to support any coherent global ranking. For example, one may observe cyclic preferences such as

A \succ B \succ C \succ A

, or inconsistencies involving ties such as

A \equiv B\equiv C\neq A

. Such contradictions make the resulting leaderboard unstable and challenging to interpret. In this paper, we propose a prompt perturbation framework for improving the consistency of pairwise LLM evaluation. Our approach generates perturbed variants of each prompt, uses the resulting comparison graphs to identify and filter out structurally inconsistent comparison patterns, and then applies standard ranking methods to the filtered comparisons. A key feature of the proposed framework is that graph-level structural consistency is incorporated explicitly into the evaluation pipeline before ranking aggregation. This provides a simple and principled way to reduce cyclic inconsistencies and improve the reliability of LLM rankings.

Dong Huang, Jianbo Sun, Pengkun Yang

Department of Statistics and Data Science, Tsinghua University

Large Language Model Evaluation Ranking Preference Alignment Learning Chart

Ask PDF

Jun 23, 2026cs.CL

On the Stability of Prompt Ranking in Large Language Model Evaluation

Prompt-based interaction has become a dominant paradigm for using large language models (LLMs), where multiple candidate prompts are evaluated and the top-ranked one is selected for downstream use. This workflow implicitly assumes that prompt rankings are stable under minor variations in evaluation conditions. In this paper, we systematically study prompt ranking stability under common sources of variability, including random seeds and limited evaluation subsets. Across three open-weight LLMs and two benchmark tasks, we find that while overall rank correlations are often moderate to high, the identity of the top-performing prompt frequently changes, leading to unreliable selection decisions. To address this issue, we propose a simple stability-aware selection strategy based on a lower confidence bound, which accounts for both performance and variance. Our results show that this approach improves robustness in unstable settings while remaining competitive in more stable regimes. These findings highlight the importance of accounting for evaluation uncertainty in prompt selection and LLM benchmarking.

Shaoshuai Du, Penghao Liang, Yixian Shen +3

University of Amsterdam, Amsterdam, Netherlands · Northeastern University, Boston, MA, USA · University of California San Diego, La Jolla, CA, USA +1

Large Language Model Evaluation Large Language Model Benchmarks Ranking

Ask PDF