Replicability, Reproducibility, and Robustness in Political and Social Science Research with Large Language Models

Joan C. Timoneda; Sebastián Vallejo Vera

cs.CLNov 4, 2026

Replicability, Reproducibility, and Robustness in Political and Social Science Research with Large Language Models

Authors: Joan C. Timoneda, Sebastián Vallejo Vera

Organizations: Assistant Professor, Department of Political Science, Purdue University. · Assistant Professor, Department of Political Science, University of Western Ontario.

Abstract

This chapter addresses the central reproducibility problem of research with large language models. LLM output is probabilistic and varies across models and versions, which can have significant effects on downstream statistical results. The chapter distinguishes between replicability, reproducibility, and robustness, and proposes a probabilistic standard based on the distribution of results, not identical outputs. It introduces a standardized prompt script, tests within-model variation for encoders and decoders, and develops error-structure analysis to diagnose where models err and whether their errors align. Using three datasets, the chapter shows that measurement choice can reverse the sign of downstream regression coefficients. Existing design-based corrections with gold-labeled subsamples restore validity, but come at the cost of estimate precision. The main contribution of the chapter is a stacked ensemble of open-weight models that improves precision in the correction step while retaining estimate validity.

Figures & tables

Figure 5.1 : Error overlap across the three tasks. Each cell reports the Jaccard overlap between two models’ error sets: for a pair of models, the number of documents both models misclassified divided by the number that either one misclassified. Values closer to one indicate that the two models err on the same observations; values closer to zero indicate that they err on different observations. Lines separate the five best decoder configurations from the fine-tuned encoder. Above each panel: the overlap that independent errors would produce. In Foy-Sutherland (2025) , the two hottest cells pair the same model under different prompts.

Figure 5.2 : Per-class miss rates for the five best decoder configurations and the fine-tuned encoder, by dataset. Each cell is the share of the class the model misclassifies; the vertical line separates the decoders from RoBERTa. Because each dataset ranks its own five best decoders, the decoder columns denote different models in each block. In Ballard et al. (2023) : GPT-mini FS, Mistral ZS, Llama FS, Gemma FS, Mistral FS. In Foy-Sutherland (2025) : Gemma FS, Gemma ZS, GPT-mini ZS, GPT-5.5 ZS, GPT-5.5 FS. In Trubowitz and Watanabe (2021) : GPT-5.5 FS, GPT-mini ZS, Gemma FS, Llama ZS, Mistral ZS. The encoder is more accurate than the decoders on the common classes but less accurate on several minority classes — Contrast in Foy-Sutherland and Unclear in Trubowitz and Watanabe .

Figure 5.3 : Shared decoder failures, and how many the encoder recovers. Each bar reports the share of a gold class that all five best decoders misclassify; the light portion is recovered by the fine-tuned encoder, the dark portion is missed by every model. The line on each bar marks the share that would be jointly misclassified if the decoders’ errors were independent — the gap between rule and bar measures how far decoder failures are correlated rather than coincidental. Class sizes in parentheses. On Trubowitz and Watanabe (2021) ’s Yes class the encoder recovers over half of the joint failures; on Unclear it recovers one document in forty-four, the codebook’s ceiling.

Panel of Figure 5 — Figure 5.4 : Naive (red) versus DSL-corrected (blue) estimates of $\beta_{Republican}$ in the Ballard et al. (2023) data, by hand-labeling budget. Dashed line and shaded band: gold-label benchmark and its 95% CI. Configurations ordered by uncivil-class F1.

Figure 5.7 : The single-model lottery: DSL-corrected estimates of

\beta_{Republican}

in the Ballard et al. (2023) data using the worst and best of the ten configurations, against the hand-labeled subsample alone, at three gold budgets. Numbers: mean 95% CI width. Dashed line and shaded band: gold-label benchmark and its 95% CI. Which configuration is best or worst is knowable only from a labeled validation sample — the single-model researcher draws blind.

Figure 5.8 : The single-model lottery and the ensemble alternative: DSL-corrected estimates of

\beta_{Republican}

in the Ballard et al. (2023) data using the worst and best of the ten decoder configurations, the hand-labeled subsample alone, and a stacked ensemble of the three open-weight few-shot configurations ( Timoneda, 2026b , see also) (weights fit on the validation half), at four gold budgets. Bottom row: the naive encoder estimate on the full corpus, uncorrected. Numbers: mean 95% CI width. Dashed line and shaded band: gold-label benchmark and its 95% CI.

Disclosure item	Information to report
Model identification	Name, version, and repository identifier of the LLM or language model used, including whether it is a base, fine-tuned, or otherwise modified model.
Model modifications	Any modifications applied to the model, including additional pre-training, fine-tuning, instruction tuning, adapters, prompting frameworks, or other architectural or procedural changes.
Prompt(s)	Full text of the final prompt, including system message, category definitions, few-shot examples, and required output format; the number of repeated runs; and a summary of prompt variants tested.
Training or fine-tuning data	Description of any additional data used to further pre-train, fine-tune, or adapt the model, including data source, size, selection criteria, preprocessing, and relevant exclusions.
Hyperparameters	Relevant training, fine-tuning, or inference hyperparameters, such as learning rate, batch size, number of epochs, optimizer, temperature, top- $p$ , maximum tokens, and random seeds where applicable.
Computational equipment	Hardware used for training, fine-tuning, or inference, including GPU or CPU type, number of devices, memory, and other relevant computational specifications.

Table 5.1 : Information to Disclose When Using LLMs

Scientific Reproducibility

Scientific reproducibility studies whether research methods, data, and software can be inspected and rerun to support published findings.

See all 117 papers

Automated reproducibility assessments in the social and behavioral sciences using large language models
Jun 11, 2026Tobias Holtdirk, Pietro Marcolongo, Anna Steinberg Schulten +7AI-Assisted Scientific Research Scientific Reproducibility
1LMU Munich, Munich, Germany · 2Munich Center for Machine Learning, Munich, Germany · University of Maryland, College Park, USA +1
Can Large Language Models Detect Methodological Flaws? Evidence from Gesture Recognition for UAV-Based Rescue Operation Based on Deep Learning
Mar 23, 2026Domonkos VargaLLM Auditing LLM Evaluation
Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results
Apr 23, 2026Benjamin Kohler, David Zollikofer, Johanna Einsiedler +2AI Coding Agents LLM Agent Evaluation
ETH Zurich · University of Basel

Jun 11, 2026J/K move · Enter open · S save

Automated reproducibility assessments in the social and behavioral sciences using large language models

Tobias Holtdirk, Pietro Marcolongo, Anna Steinberg Schulten, Felix Henninger+6

1LMU Munich, Munich, Germany · 2Munich Center for Machine Learning, Munich, Germany · University of Maryland, College Park, USA+1

AI-Assisted Scientific Research Scientific Reproducibility

Open paper AskPDF arXiv

Reproducibility in the social and behavioral sciences is typically evaluated by independent researchers who reanalyze the original data to assess whether the published findings can be recovered. However, such approaches are resource-intensive and difficult to scale. Here, we show that large language models (LLMs) can automate reproducibility assessments. Using N = 180 published studies with predefined claims from the behavioral and social sciences, we compare LLM-generated analyses with the original findings. For 11 studies, the LLM pipeline could not produce a viable effect size estimate. For the remaining studies, the LLM reached the same qualitative conclusion as the original study in 80% of cases, and recovered the original effect sizes (using a +/-0.05 tolerance in Cohen's d) in 24% of studies. In a subset with human reanalyses, the LLM reached the same qualitative conclusion as the original study in 95% of studies, similar to human reanalysts (83%), and the LLM recovered the original effect sizes using a +/-0.05 tolerance in 40% of studies, again broadly similar to human reanalysts (28%). Given the current capabilities and limitations of LLMs, the findings show that LLMs can support systematic audits of empirical results rather than substitute expert judgment. As such, LLMs can serve as a scalable screening tool to improve the rigor and reproducibility in empirical research.

Abstract

Scientific Reproducibility

Explore similar work

Automated reproducibility assessments in the social and behavioral sciences using large language models

Can Large Language Models Detect Methodological Flaws? Evidence from Gesture Recognition for UAV-Based Rescue Operation Based on Deep Learning

Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results