cs.SEJun 28, 2026

Citation Discipline in Spec-Driven Development: A Cross-Model Empirical Study of Output Determinism and Automated Hallucination Detection in LLM-Generated Code

Authors: Subham Panda

Abstract

Spec-Driven Development (SDD) frameworks guide Large Language Model (LLM)-powered code generation through formal specifications, yet they differ fundamentally in how they enforce traceability between requirements and generated code. This paper presents two controlled empirical studies comparing three SDD frameworks: traceSDDtraceSDD, which enforces mandatory per-line requirement citations using hierarchical REQ-XXX.Y.Z identifiers; SpecKitSpec Kit, which uses artifact-level traceability through user stories and acceptance criteria; and OpenSpecOpenSpec, which relies on post-hoc external trace maps. We measure two primary outcomes across two frontier LLMs -- Claude Sonnet 4.6 (N=20, 4 conditions, 240 implementations) and GLM-5-turbo (N=50, 4 conditions, 600 implementations): outputoutput determinismdeterminism (lexical similarity across independent LLM sessions) and automatedautomated hallucinationhallucination detectiondetection raterate (TDR). Our pre-registered analysis reveals a consistent, cross-model replicated trade-off: the uncited condition produces significantly higher determinism than the cited condition (Claude: d=−0.76d=-0.76, p=0.003p=0.003; GLM: d=−0.72d=-0.72, p<0.001p<0.001), while only the cited condition enables automated hallucination detection (TDR: Claude 86.4%, GLM 88.0%, vs 0% for all alternatives, FPR=0% across both studies). traceSDD (cited) significantly outperforms SpecKitSpec Kit on determinism (Claude: d=0.47d=0.47, p=0.049p=0.049; GLM: d=0.42d=0.42, p=0.003p=0.003) but not OpenSpec (Claude: d=0.18d=0.18, p=0.44p=0.44; GLM: d=0.14d=0.14, p=0.32p=0.32). These findings establish that citation annotations trade determinism for verifiability, and that this trade-off generalizes across model architectures.

Explore similar work

CardsList