Replicability, Reproducibility, and Robustness in Political and Social Science Research with Large Language Models

Joan C. Timoneda; Sebastián Vallejo Vera

cs.CLNov 4, 2026

Replicability, Reproducibility, and Robustness in Political and Social Science Research with Large Language Models

Authors: Joan C. Timoneda, Sebastián Vallejo Vera

Organizations: Assistant Professor, Department of Political Science, Purdue University. · Assistant Professor, Department of Political Science, University of Western Ontario.

Abstract

This chapter addresses the central reproducibility problem of research with large language models. LLM output is probabilistic and varies across models and versions, which can have significant effects on downstream statistical results. The chapter distinguishes between replicability, reproducibility, and robustness, and proposes a probabilistic standard based on the distribution of results, not identical outputs. It introduces a standardized prompt script, tests within-model variation for encoders and decoders, and develops error-structure analysis to diagnose where models err and whether their errors align. Using three datasets, the chapter shows that measurement choice can reverse the sign of downstream regression coefficients. Existing design-based corrections with gold-labeled subsamples restore validity, but come at the cost of estimate precision. The main contribution of the chapter is a stacked ensemble of open-weight models that improves precision in the correction step while retaining estimate validity.

Figures & tables

Figure 5.1 : Error overlap across the three tasks. Each cell reports the Jaccard overlap between two models’ error sets: for a pair of models, the number of documents both models misclassified divided by the number that either one misclassified. Values closer to one indicate that the two models err on the same observations; values closer to zero indicate that they err on different observations. Lines separate the five best decoder configurations from the fine-tuned encoder. Above each panel: the overlap that independent errors would produce. In Foy-Sutherland (2025) , the two hottest cells pair the same model under different prompts.

Figure 5.2 : Per-class miss rates for the five best decoder configurations and the fine-tuned encoder, by dataset. Each cell is the share of the class the model misclassifies; the vertical line separates the decoders from RoBERTa. Because each dataset ranks its own five best decoders, the decoder columns denote different models in each block. In Ballard et al. (2023) : GPT-mini FS, Mistral ZS, Llama FS, Gemma FS, Mistral FS. In Foy-Sutherland (2025) : Gemma FS, Gemma ZS, GPT-mini ZS, GPT-5.5 ZS, GPT-5.5 FS. In Trubowitz and Watanabe (2021) : GPT-5.5 FS, GPT-mini ZS, Gemma FS, Llama ZS, Mistral ZS. The encoder is more accurate than the decoders on the common classes but less accurate on several minority classes — Contrast in Foy-Sutherland and Unclear in Trubowitz and Watanabe .

Figure 5.3 : Shared decoder failures, and how many the encoder recovers. Each bar reports the share of a gold class that all five best decoders misclassify; the light portion is recovered by the fine-tuned encoder, the dark portion is missed by every model. The line on each bar marks the share that would be jointly misclassified if the decoders’ errors were independent — the gap between rule and bar measures how far decoder failures are correlated rather than coincidental. Class sizes in parentheses. On Trubowitz and Watanabe (2021) ’s Yes class the encoder recovers over half of the joint failures; on Unclear it recovers one document in forty-four, the codebook’s ceiling.

Panel of Figure 5 — Figure 5.4 : Naive (red) versus DSL-corrected (blue) estimates of $\beta_{Republican}$ in the Ballard et al. (2023) data, by hand-labeling budget. Dashed line and shaded band: gold-label benchmark and its 95% CI. Configurations ordered by uncivil-class F1.

Figure 5.7 : The single-model lottery: DSL-corrected estimates of

\beta_{Republican}

in the Ballard et al. (2023) data using the worst and best of the ten configurations, against the hand-labeled subsample alone, at three gold budgets. Numbers: mean 95% CI width. Dashed line and shaded band: gold-label benchmark and its 95% CI. Which configuration is best or worst is knowable only from a labeled validation sample — the single-model researcher draws blind.

Figure 5.8 : The single-model lottery and the ensemble alternative: DSL-corrected estimates of

\beta_{Republican}

in the Ballard et al. (2023) data using the worst and best of the ten decoder configurations, the hand-labeled subsample alone, and a stacked ensemble of the three open-weight few-shot configurations ( Timoneda, 2026b , see also) (weights fit on the validation half), at four gold budgets. Bottom row: the naive encoder estimate on the full corpus, uncorrected. Numbers: mean 95% CI width. Dashed line and shaded band: gold-label benchmark and its 95% CI.

Disclosure item	Information to report
Model identification	Name, version, and repository identifier of the LLM or language model used, including whether it is a base, fine-tuned, or otherwise modified model.
Model modifications	Any modifications applied to the model, including additional pre-training, fine-tuning, instruction tuning, adapters, prompting frameworks, or other architectural or procedural changes.
Prompt(s)	Full text of the final prompt, including system message, category definitions, few-shot examples, and required output format; the number of repeated runs; and a summary of prompt variants tested.
Training or fine-tuning data	Description of any additional data used to further pre-train, fine-tune, or adapt the model, including data source, size, selection criteria, preprocessing, and relevant exclusions.
Hyperparameters	Relevant training, fine-tuning, or inference hyperparameters, such as learning rate, batch size, number of epochs, optimizer, temperature, top- $p$ , maximum tokens, and random seeds where applicable.
Computational equipment	Hardware used for training, fine-tuning, or inference, including GPU or CPU type, number of devices, memory, and other relevant computational specifications.

Table 5.1 : Information to Disclose When Using LLMs

Scientific Reproducibility

Scientific reproducibility studies whether research methods, data, and software can be inspected and rerun to support published findings.

See all 117 papers

Jun 11, 2026

Automated reproducibility assessments in the social and behavioral sciences using large language models

Reproducibility in the social and behavioral sciences is typically evaluated by independent researchers who reanalyze the original data to assess whether the published findings can be recovered. However, such approaches are resource-intensive and difficult to scale. Here, we show that large language models (LLMs) can automate reproducibility assessments. Using N = 180 published studies with predefined claims from the behavioral and social sciences, we compare LLM-generated analyses with the original findings. For 11 studies, the LLM pipeline could not produce a viable effect size estimate. For the remaining studies, the LLM reached the same qualitative conclusion as the original study in 80% of cases, and recovered the original effect sizes (using a +/-0.05 tolerance in Cohen's d) in 24% of studies. In a subset with human reanalyses, the LLM reached the same qualitative conclusion as the original study in 95% of studies, similar to human reanalysts (83%), and the LLM recovered the original effect sizes using a +/-0.05 tolerance in 40% of studies, again broadly similar to human reanalysts (28%). Given the current capabilities and limitations of LLMs, the findings show that LLMs can support systematic audits of empirical results rather than substitute expert judgment. As such, LLMs can serve as a scalable screening tool to improve the rigor and reproducibility in empirical research.

Tobias Holtdirk, Pietro Marcolongo, Anna Steinberg Schulten +7

1LMU Munich, Munich, Germany · 2Munich Center for Machine Learning, Munich, Germany · University of Maryland, College Park, USA +1

AI-Assisted Scientific Research Scientific Reproducibility

Ask PDF

Mar 23, 2026

Can Large Language Models Detect Methodological Flaws? Evidence from Gesture Recognition for UAV-Based Rescue Operation Based on Deep Learning

Reliable evaluation is essential in machine learning research, yet methodological flaws-particularly data leakage-continue to undermine the validity of reported results. In this work, we investigate whether large language models (LLMs) can act as independent analytical agents capable of identifying such issues in published studies. As a case study, we analyze a gesture-recognition paper reporting near-perfect accuracy on a small, human-centered dataset. We first show that the evaluation protocol is consistent with subject-level data leakage due to non-independent training and test splits. We then assess whether this flaw can be detected independently by six state-of-the-art LLMs, each analyzing the original paper without prior context using an identical prompt. All models consistently identify the evaluation as flawed and attribute the reported performance to non-independent data partitioning, supported by indicators such as overlapping learning curves, minimal generalization gap, and near-perfect classification results. These findings suggest that LLMs can detect common methodological issues based solely on published artifacts. While not definitive, their consistent agreement highlights their potential as complementary tools for improving reproducibility and supporting scientific auditing.

Domonkos Varga

LLM Auditing LLM Evaluation Data Leakage

Ask PDF

Apr 23, 2026

Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results

Recent work has used LLM agents to reproduce empirical social science results with access to both the data and code. We broaden this scope by asking: Can they reproduce results given only a paper's methods description and original data? We develop an agentic reproduction system that extracts structured methods descriptions from papers, runs reimplementations under strict information isolation -- agents never see the original code, results, or paper -- and enables deterministic, cell-level comparison of reproduced outputs to the original results. An error attribution step traces discrepancies through the system chain to identify root causes. Evaluating four agent scaffolds and four LLMs on 48 papers with human-verified reproducibility, we find that agents can largely recover published results, but performance varies substantially between models, scaffolds, and papers. Root cause analysis reveals that failures stem both from agent errors and from underspecification in the papers themselves.

Benjamin Kohler, David Zollikofer, Johanna Einsiedler +2

ETH Zurich · University of Basel

AI Coding Agents LLM Agent Evaluation Scientific Reproducibility

Ask PDF

Abstract

Scientific Reproducibility

Explore similar work

Automated reproducibility assessments in the social and behavioral sciences using large language models

Can Large Language Models Detect Methodological Flaws? Evidence from Gesture Recognition for UAV-Based Rescue Operation Based on Deep Learning

Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results