Auditing the Synthetic Memoir: Measuring Scene-Level Confabulation in LLM-Generated Autobiography Against the Documented Record of the Life It Describes
When a large language model (LLM) is asked to write a person's life, how much of what it writes actually happened? We present a scene-level case-study audit - the first quantified audit of LLM-generated autobiography against a subject-specific ground-truth corpus that we are aware of, based on an unsystematic literature search. The subject and the author of this paper are the same person: a 366-day "page-a-day" book of first-person anecdotal entries was drafted with a conversational LLM whose documented inputs were a template, two exemplar days, and each day's quote - not her corpus - and every day was subsequently audited at the anecdote-scene level against an independent verification corpus using a four-level rubric fixed before analysis. We define the verification-failure rate as the share of days not rated VERIFIED (scene positively corroborated): 354 of 366 days fail, 96.7% (Wilson 95% CI 94.4-98.1%). Only 12 days contain a corroborated scene; 19 days (5.2%) assert claims actively contradicted by the record; the dominant failure mode is grounded drift - real people, employers, and settings inside invented scenes - though its measured share varies across raters. Independent re-rating replicates the headline (no evidence the original rate was inflated) while showing that the four-way taxonomy has only fair-to-moderate reliability. Regenerating the same days with current named models reproduces 100% verification failure under the same inputs; grounding generation in the subject's corpus significantly improves the verification rate while leaving substantial residual failure (83.3%). We contribute the measurement, a reusable audit instrument whose WEAK/UNVERIFIED boundary we show to be unreliable, and a grounding remedy with quantified effect.
Figures & tables
Verdict
Days
% of 366
VERIFIED
12
3.3%
WEAK
227
62.0%
UNVERIFIED
108
29.5%
CONTRADICTED / flagged
19
5.2%
Failed verification (any non-VERIFIED class)
354
96.7%
Table 1: Scene-level verdicts over 366 generated day-narratives. “Failed verification” is any non- verified day (Section 4.4 ); the source summary’s approximate tally ( ≈ 350/366) is consistent with this exact count.
Figure 1: Verdict distribution, n=366 .
Figure 2: Verdicts by month, n=366 .
Premise
Family
Screen hits
P1 Invented children / “the kids”
invented relations
5
P2 Blue-water sailor persona
invented capability
4
P3 Treatment-history misattribution
misattributed fact
4
P4 Invented adult car accident
invented event
5
P5 Evernote role inflation
misattributed fact
5
P6 AntarctiConf “on the ice” fiction
invented event
3
Table 2: Recurring false premises; fixed keyword screens over audit rows (Section 4.5 ). Per-day lists: Appendix C .
Figure 3: Days whose audit row references each false premise (keyword screen).
Figure 4: Source-class coverage of the 252 sourced audit rows (multi-label).
Date
Verdict
Anecdote gist (abbreviated)
April 2
VERIFIED
Joined Evernote as employee number eight; a mentor said "build a culture"
April 6
VERIFIED
Hit one hundred million users at Evernote; no celebration
May 9
VERIFIED
Lived on a sailboat in Mexico & an RV at Disney World; travel light
May 13
VERIFIED
Asked every founder "what aren’t you saying?"; never turned it on her own dying marriage…
May 14
VERIFIED
Underestimated as a woman at THQ; winning College Jeopardy sealed the lesson
May 30
VERIFIED
Near-death car accident as a teen (distracted driver); fear rode along for years
Table 3: All 12 days whose central scene was positively corroborated.
(a) Rater B (partial exposure)
Original
V
W
U
C
VERIFIED
1
2
0
0
WEAK
1
20
10
4
UNVERIFIED
0
3
16
1
CONTRADICTED
0
1
0
1
(b) Rater C (fully blind)
Table 4: Blind re-rates of two independent 60-day samples (seed 20260821): original labels (rows) vs. second raters (columns); V=VERIFIED, W=WEAK, U=UNVERIFIED, C=CONTRADICTED. Four-way agreement: B 63.3% ( κ=0.387 ), C 81.7% ( κ=0.573 ). Binary VERIFIED-vs-not raw agreement: B 95.0%, C 98.3% (binary κ degenerate and not quoted; Section 5.6 ).
Arm
VERIFIED
WEAK
UNVERIFIED
CONTRADICTED
Verification failure
95% CI
A1: 2025 (orig. audit)
1
16
43
0
98.3%
[91.1, 99.7]
B: gpt-5.4
0
1
59
0
100.0%
[94.0, 100.0]
B2: cl.-sonnet-5
0
14
46
0
100.0%
[94.0, 100.0]
C: gpt-5.4 grounded
10
26
24
0
83.3%
[72.0, 90.7]
Table 5: 2025-vs-2026 replication on the same 60-day sample (seed 20260821); final fully-blind ratings (raters D/E). “Failed verification” = any non- verified day.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Date
Premise
Anecdote gist (abbreviated)
February 7
P6
AntarctiConf “application” sat in drafts; scared to submit; “I submitted. I got in.”
March 26
—
“Months before diagnosis I lived in fear of cancer, imagined it in my body”
April 4
P4
Hospital bed after a car hit her, unsure she’d walk again; married Matt 2019
April 14
P4
Thanksgiving overeating; “after the car accident,” healing, pushed too hard
April 15
P6/P7
First keynote ON Antarctica, “standing on the seventh continent,” mic might freeze
April 19
P4
Decade of trying to be liked; then “a car hit me,” hospital bed, came out sharper
Appendix
Table 6: All 19 contradicted days. The Premise column reports keyword-screen output over gist+note text (Section 4.5 ); themes need not be visible in the abbreviated gist. “—” marks screen-negative days flagged during audit rather than by keyword screen.