cs.AIAug 23, 2026

Auditing the Synthetic Memoir: Measuring Scene-Level Confabulation in LLM-Generated Autobiography Against the Documented Record of the Life It Describes

Authors: Heather Renze

Organizations: Serenze Global

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

When a large language model (LLM) is asked to write a person's life, how much of what it writes actually happened? We present a scene-level case-study audit - the first quantified audit of LLM-generated autobiography against a subject-specific ground-truth corpus that we are aware of, based on an unsystematic literature search. The subject and the author of this paper are the same person: a 366-day "page-a-day" book of first-person anecdotal entries was drafted with a conversational LLM whose documented inputs were a template, two exemplar days, and each day's quote - not her corpus - and every day was subsequently audited at the anecdote-scene level against an independent verification corpus using a four-level rubric fixed before analysis. We define the verification-failure rate as the share of days not rated VERIFIED (scene positively corroborated): 354 of 366 days fail, 96.7% (Wilson 95% CI 94.4-98.1%). Only 12 days contain a corroborated scene; 19 days (5.2%) assert claims actively contradicted by the record; the dominant failure mode is grounded drift - real people, employers, and settings inside invented scenes - though its measured share varies across raters. Independent re-rating replicates the headline (no evidence the original rate was inflated) while showing that the four-way taxonomy has only fair-to-moderate reliability. Regenerating the same days with current named models reproduces 100% verification failure under the same inputs; grounding generation in the subject's corpus significantly improves the verification rate while leaving substantial residual failure (83.3%). We contribute the measurement, a reusable audit instrument whose WEAK/UNVERIFIED boundary we show to be unreliable, and a grounding remedy with quantified effect.

Figures & tables

VerdictDays% of 366
VERIFIED123.3%
WEAK22762.0%
UNVERIFIED10829.5%
CONTRADICTED / flagged195.2%
Failed verification (any non-VERIFIED class)35496.7%
Table 1: Scene-level verdicts over 366 generated day-narratives. “Failed verification” is any non- verified day (Section 4.4 ); the source summary’s approximate tally ( ≈\approx 350/366) is consistent with this exact count.
Figure 1: Verdict distribution, n=366n=366 .
Figure 2: Verdicts by month, n=366n=366 .
PremiseFamilyScreen hits
P1 Invented children / “the kids”invented relations5
P2 Blue-water sailor personainvented capability4
P3 Treatment-history misattributionmisattributed fact4
P4 Invented adult car accidentinvented event5
P5 Evernote role inflationmisattributed fact5
P6 AntarctiConf “on the ice” fictioninvented event3
Table 2: Recurring false premises; fixed keyword screens over audit rows (Section 4.5 ). Per-day lists: Appendix C .
Figure 3: Days whose audit row references each false premise (keyword screen).
Figure 4: Source-class coverage of the 252 sourced audit rows (multi-label).
DateVerdictAnecdote gist (abbreviated)
April 2VERIFIEDJoined Evernote as employee number eight; a mentor said "build a culture"
April 6VERIFIEDHit one hundred million users at Evernote; no celebration
May 9VERIFIEDLived on a sailboat in Mexico & an RV at Disney World; travel light
May 13VERIFIEDAsked every founder "what aren’t you saying?"; never turned it on her own dying marriage…
May 14VERIFIEDUnderestimated as a woman at THQ; winning College Jeopardy sealed the lesson
May 30VERIFIEDNear-death car accident as a teen (distracted driver); fear rode along for years
Table 3: All 12 days whose central scene was positively corroborated.
(a) Rater B (partial exposure)
OriginalVWUC
VERIFIED1200
WEAK120104
UNVERIFIED03161
CONTRADICTED0101
(b) Rater C (fully blind)
Table 4: Blind re-rates of two independent 60-day samples (seed 20260821): original labels (rows) vs. second raters (columns); V=VERIFIED, W=WEAK, U=UNVERIFIED, C=CONTRADICTED. Four-way agreement: B 63.3% ( κ=0.387\kappa=0.387 ), C 81.7% ( κ=0.573\kappa=0.573 ). Binary VERIFIED-vs-not raw agreement: B 95.0%, C 98.3% (binary κ\kappa degenerate and not quoted; Section 5.6 ).
ArmVERIFIEDWEAKUNVERIFIEDCONTRADICTEDVerification failure95% CI
A1: 2025 (orig. audit)11643098.3%[91.1, 99.7]
B: gpt-5.401590100.0%[94.0, 100.0]
B2: cl.-sonnet-5014460100.0%[94.0, 100.0]
C: gpt-5.4 grounded102624083.3%[72.0, 90.7]
Table 5: 2025-vs-2026 replication on the same 60-day sample (seed 20260821); final fully-blind ratings (raters D/E). “Failed verification” = any non- verified day.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

DatePremiseAnecdote gist (abbreviated)
February 7P6AntarctiConf “application” sat in drafts; scared to submit; “I submitted. I got in.”
March 26—“Months before diagnosis I lived in fear of cancer, imagined it in my body”
April 4P4Hospital bed after a car hit her, unsure she’d walk again; married Matt 2019
April 14P4Thanksgiving overeating; “after the car accident,” healing, pushed too hard
April 15P6/P7First keynote ON Antarctica, “standing on the seventh continent,” mic might freeze
April 19P4Decade of trying to be liked; then “a car hit me,” hospital bed, came out sharper
Appendix
Table 6: All 19 contradicted days. The Premise column reports keyword-screen output over gist+note text (Section 4.5 ); themes need not be visible in the abbreviated gist. “—” marks screen-negative days flagged during audit rather than by keyword screen.
More in this topic

Language Model Auditing

Language model auditing tests whether models behave reliably, fairly, and safely using behavioral probes and internal or deployment evidence.

See all 159 papers

Explore similar work

CardsList
Aug 20, 2026

PersonalBench: Measuring the Authorship Gap in LLM Personalization

Personalized text generation aims to make LLMs write in a specific individual's style, yet existing benchmarks measure task accuracy or preference alignment rather than whether the model's output actually resembles the target author's writing. We introduce PersonalBench, a benchmark that evaluates inference-time personalization methods through three independent lenses: LUAR (a trained authorship verification model), an LLM-as-judge, and automated stylometrics. Across 50 authors, 1,000 generations, and two model families (Qwen 3, GLM-4), we find that personalization methods do produce author-differentiated output (LUAR discriminates target authors within generated text at AUC=0.918) but this differentiation never crosses the human-LLM boundary. All methods achieve LUAR similarity to real authors in the range 0.484-0.508, below the cross-author human floor of 0.626 (ceiling 0.756). The LLM's own authorship fingerprint dominates: generated text is more distant from any human author than random humans are from each other. Methods are statistically indistinguishable from each other on LUAR (spread 0.024) despite appearing differentiated on the LLM judge, a discrepancy we trace to circularity between trait extraction and profile extraction. We validate that LUAR reliably measures authorship in our corpus (AUC=0.76 single-post, 0.96 multi-post). We release PersonalBench as a calibrated measuring stick: inference-time personalization modulates the LLM's style but does not bridge the gap to human authorship.

Yash Ganpat Sawant

Independent AI Researcher

Personalized Language ModelsPersonalized Text GenerationLanguage Model Generation EvaluationAuthorship Attribution
AskPDF
Jul 20, 2026

HALLMARK: How to Diagnose LLM Citation Verifiers and When to Deploy Them

GPTZero found 53 NeurIPS 2025 accepted papers with hallucinated citations, and venues now screen submissions with automated reference checkers. Rule- and Large Language Model (LLM)-based verifiers are emerging, but many are released without a shared evaluation and each flags a different kind of error, so it is unclear which to trust or when to deploy them. We address this with HALLMARK (Hallucination benchmark): 2,526 BibTeX entries spanning 14 hallucination types, with per-type and diagnostic sub-test labels and further temporal, cross-domain, and real-world evaluation splits. We evaluate zero-shot and tool-augmented LLMs, a DOI-lookup baseline, and our own co-designed rule-based verifier, bibtex-updater. Verifiers can miss hallucinations and wrongly flag correct papers, so every flag still needs human review. The false positive rate (FPR) determines the review effort and thus limits deployment. The tested models exhibit an order-of-magnitude spread in FPR; agentic lookups buy recall but inflate FPR, and neither smaller models nor prompt optimization reach the frontier's operating point. A two-stage cascade with bibtex-updater avoids that inflation, providing an upper bound, given the co-design. These rankings, however, depend on what counts as a hallucination: re-scoring the types that describe a real work cited inaccurately changes which verifier ranks first, and venues have not agreed on a definition. We call for a community definition and release per-type labels and a re-scoring protocol so that any agreed definition can be scored against every verifier.

Kayoon Kim, Wieland Brendel, Patrik Reizinger

Max Planck Institute for Intelligent Systems, ELLIS Institute Tübingen

LLM EvaluationCitation VerificationLLM Hallucination Detection
AskPDF
Apr 29, 2026

Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization

Stylistic personalization - making LLMs write in a specific individual's style, rather than merely adapting to task preferences - lacks evaluation grounded in authorship science. We show that grounding evaluation in authorship verification theory transforms what benchmarks can measure. Drawing on three measurement traditions - LUAR, a trained authorship verification model; an LLM-as-judge with decoupled trait matching; and classical function-word stylometrics - we evaluate four inference-time personalization methods across 50 authors and 1,000 generations. The theory-grounded metric, LUAR, provides what ad hoc alternatives cannot: calibrated baselines, with a human ceiling of 0.756 and a cross-author floor of 0.626, that give scores absolute meaning. All methods score below this floor, from 0.484 to 0.508, exposing an authorship gap invisible to uncalibrated metrics. The three metrics produce near-zero pairwise correlations, with absolute r less than 0.07, confirming that without theoretical grounding, metric choice determines conclusions: an LLM judge declares a clear winner while LUAR finds no meaningful differentiation. These findings demonstrate the theory-benchmark cycle in action: authorship theory exposes evaluation failures that ad hoc benchmarks miss.

Yash Ganpat Sawant

Independent AI Researcher

LLM EvaluationLLM PersonalizationLanguage Model Generation EvaluationAuthorship Attribution
AskPDF