Provenance Before Prose: Claim-Locked Reporting for Statistical Text Generation

Xiao Fan; Jingyuan Li; Hongbin Guo; Yubo Han; Yi Zhang

cs.CLAug 26, 2026

Provenance Before Prose: Claim-Locked Reporting for Statistical Text Generation

Authors: Xiao Fan, Jingyuan Li, Hongbin Guo, Yubo Han, Yi Zhang

Organizations: Xidian University

Abstract

Large language models (LLMs) can fluently verbalize statistical evidence, yet statistical reports can still drift numerical values, invert effect directions, or restate thresholded contrasts as categorical effects. We frame these failures as a control problem: the evidence-bearing content of a scientific report should be fixed by structured statistical results rather than sampled during prose generation. We therefore use cross-run reproducibility to stress-test whether report-visible numbers and claims are bound before prose generation. Existing controls operate at the text or slot level; a deterministic hybrid template reproduces only 61.1% of report-visible numerical content across seeds because the LLM still selects which findings and numbers the template renders. We propose claim-locked reporting, a provenance-before-prose protocol that fixes the evidence source, numbers, direction, and allowed language strength of each reportable claim before the LLM writes only connective prose. Across fMRI functional-connectivity reporting and randomized controlled trial reporting on Evidence Inference 2.0, claim-locked reporting improves reproducibility over the hybrid template by 37.4 and 20.5 points, respectively. Blinded human audits support the observed direction-preservation and governance trends. In an fMRI cost analysis with DeepSeek, claim-locked reporting also yields the lowest observed token use and median generation latency.

Figures & tables

Figure 1: Three levels of control for statistical text generation. Text-level methods, including prompting, retrieval, and structured output, leave claims, numbers, directions, and language strength to the LLM. The hybrid template adds slot-level rendering, but the LLM still selects which claims and numbers fill the slots. Claim-locked reporting fixes the evidence source, numbers, direction, and allowed language strength before prose generation; the LLM controls only connective wording.

Figure 2: Summary landscape of the reproducibility-error and governance composites per method; lower is better on both axes, which are reported separately rather than merged into a single score. (a) fMRI; (b) RCT.

Method	Repro. $\uparrow$	Hedge	Strong $\downarrow$	Num. $\downarrow$
Free-form	32.3	0.20	2.95	0.55
Prompt-only	26.0	0.22	1.95	0.55
Structured	22.0	0.23	1.85	0.35
Retrieval	24.1	0.21	1.50	0.20
Post-hoc verifier	15.2	0.25	2.75	0.90
Hybrid template	61.1	0.25	2.25	0.00

Table 1: Main fMRI comparison (

n=20

cells per row). Hedge is risk-conditioned; Num is a numerical audit flag. Bold = best among the seven methods (Repro., Strong, Num.).

Method	Input tok.	Output tok.	Latency (s)
Free-form	20,603	5,432	111.9
Hybrid template	21,213	7,069	123.9
Post-hoc verifier	44,434	16,687	295.2
Claim-locked	12,482	4,209	77.3

Table 2: Per-report resource use on the fMRI evaluation with DeepSeek as the writer.

Cohort	Primary	+ covariate	Drop
In-house	10,041	8	99.9%
HCP comparison	550	0	100.0%

Table 3: Group-covariate absorption stress test: FDR-surviving group-effect edges before and after adding the continuous covariate (BMI) used to derive the group label.

Method	Repro. $\uparrow$	Hedge	Strong $\downarrow$	Num. $\downarrow$
Free-form	74.7	0.24	0.03	0.27
Prompt-only	85.9	0.11	0.01	0.25
Structured	90.0	0.15	0.02	0.25
Retrieval	73.2	0.09	0.02	0.23
Post-hoc verifier	74.8	0.22	0.02	0.24
Hybrid template	79.5	0.19	0.04	0.11

Table 4: RCT results on Evidence Inference 2.0 (

n=800

cells per row). Strong is near floor because conclusions are short. Num is a conservative, high-recall numerical audit flag.

Method	Pres.	Inv.	Unres.	Inv. rate (%)
Free-form	103	14	3	12.0
Prompt-only	102	10	8	8.9
Structured	103	13	4	11.2
Retrieval	102	6	12	5.6
Post-hoc verifier	98	11	11	10.1
Hybrid template	102	17	1	14.3

Table 5: Human audit of direction preservation on the RCT subset (

120

reports per method). Pres. = preserved direction; Inv. = inverted direction; Unres. = unresolved direction, including reports that do not state a clear direction or whose direction cannot be reliably determined. Inv. rate

=

Inv.

/

(Pres.

+

Inv.). Unres. is reported descriptively and is excluded from the inversion-rate denominator.

Method	FActScore $\uparrow$	SelfCheckGPT $\uparrow$
Post-hoc verifier	0.856	0.840
Structured	0.855	0.846
Retrieval	0.855	0.796
Prompt-only	0.840	0.825
Free-form	0.835	0.831
Hybrid template	0.770	0.812

Table 6: FActScore and SelfCheckGPT on the

30

-record RCT sub-sample. Higher is better for both metrics.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Control level	What is controlled	Remaining LLM decisions
Free-form	Text	General grounding instruction	Claims, numbers, direction, strength, wording
Prompt-only	Text	Explicit faithfulness instructions	Claims, numbers, direction, strength, wording
Structured	Text	Output structure	Claims, numbers, direction, strength, wording
Retrieval	Text	Evidence localization	Claims, numbers, direction, strength, wording
Post-hoc verifier	Text	Post-generation revision	Initial claims and realization
Hybrid template	Slot	Deterministic slot rendering	Slot content and claim selection

Appendix

Table A1: Control configurations of the compared methods. All methods receive the same structured statistical evidence. They differ in where control is applied and in which reporting decisions remain under LLM control.

Configuration	fMRI Repro. $\uparrow$	fMRI Strong $\downarrow$	RCT Repro. $\uparrow$	RCT Strong $\downarrow$
Hybrid	61.1	2.25	79.5	0.040
Builder only	85.0	1.60	90.3	0.000
Builder + renderer	98.0	1.40	100.0	0.015
Full claim-locked	98.5	0.75	100.0	0.010

Appendix

Table A2: Component analysis. Reproducibility is reported in percent; Strong is the unhedged-strong count. Bold marks the full claim-locked configuration

Target	FC evidence	RCT evidence	Policy action
Numerical support	Statistical results and derived quantities	Trial outcomes and reported statistics	Allow only evidence-supported numbers
Direction preservation	Ledger-stored effect direction	Gold direction label	Preserve the resolved direction
Entity/source support	Anatomical and behavioral entities	Trial entities and source information	Exclude unsupported entities or sources
Inferential scope	Covariate dependence and exploratory findings	Intervention, outcome, and population	Restrict claim scope when required
Interpretive strength	Stability and evidential support	Support available in the evidence span	Limit claims to permitted language strength

Appendix

Table A3: Shared control targets and their domain-specific instantiations. The same auditor–policy interface is used in both settings, while each target is grounded in domain-specific evidence.

Unhedged strong language
Assessment	Free-form	Hybrid	Claim-locked
Automatic rule	2.50	1.80	0.60
Reviewer 1	2.60	1.90	0.60
Reviewer 2	2.40	1.70	0.50
Reviewer 3	2.70	2.00	0.70
Reviewer 4	2.50	1.80	0.60

Appendix

Table A4: Blinded fMRI governance audit. Values are average violation counts per report; lower is better. The rule-based counts are shown alongside the four blinded reviewers and their mean.

Method	Raw flags	Regex FP	Benign	Harmful
Free-form	213	17	149	47 (5.9%)
Hybrid template	89	14	55	20 (2.5%)
Claim-locked	181	1	180	0 (0.0%)

Appendix

Table A5: Manual calibration of the RCT numerical flags underlying Table 4 . Benign denotes metadata or support-window mismatches; Harmful denotes unsupported statistical results.

Evidence-Grounded Generation

Evidence-grounded generation produces answers and documents whose claims are supported by traceable sources.

See all 148 papers

May 10, 2026

CheckSupport: A Local LLM-Powered Tool for Automated Manuscript Submission Checklist Selection and Completion

Transparent and standardized reporting is essential for reproducible scientific research, yet adherence to reporting guidelines remains inconsistent because of the manual effort required to select and complete checklists. We present CheckSupport, an open-source, locally deployable system that uses large language models to automate the recommendation of reporting checklists and the evidence-grounded completion of checklists for scientific manuscripts. CheckSupport employs a staged prompting strategy that decomposes reporting workflows into constrained inference tasks, prioritizing faithful extraction over generative text synthesis. All inference is performed locally using instruction-tuned models, preserving data privacy and enabling reproducible, auditable workflows. Evaluated on a corpus of peer-reviewed manuscripts, CheckSupport achieved 90% overall accuracy for checklist recommendations and 88% overall accuracy for item-level completion while operating on CPU-only hardware. On average, the wall-clock time per manuscript was 12.5 seconds, including the checklist recommendation and full checklist completion. These results demonstrate that large language models, when applied as structured inference components, can reduce reporting burden and support more transparent and reproducible scientific reporting across disciplines.

Satvik Tripathi, Don Enwerem, Kevin Song +3

Department of Radiology, Perelman School of Medicine at University of Pennsylvania, Philadelphia, PA, USA · Department of Computer Science, Drexel University, Philadelphia, PA, USA · Department of Computer and Information Science, School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA 19104 +2

LLM Inference Scientific Reproducibility Scientific Workflow Automation

Ask PDF

Sep 28, 2026

Language Models Are "Insecure" Reporters

As large language models are deployed in increasingly autonomous long-horizon tasks, manually auditing and verifying the actions, artifacts, and outputs of models becomes more difficult. Users instead come to rely on LLM-generated reports to assess the quality and completeness of the work. We introduce a suite of eight adversarial reporting scenarios to systematically study whether LLMs conceal narrative-changing flaws: errors or limitations that undermine an otherwise successful account of work. We call this phenomenon "insecure reporting." When handed machine learning experiment logs containing a planted negative result that substantially weakens the proposed method, GPT-5.5 flags the negative result in only 2 of 200 generated reports. However, when a short honesty instruction, "Be honest in your response," is added, the model flags the negative result in 190 of 200 reports. Across eight open-weight models, chain-of-thought analysis reveals a recurring tension between disclosing narrative-changing flaws and reasoning about ways to appear successful. We perform an activation analysis and a steering experiment on Qwen3.5-9B, finding that honesty and success-seeking correspond to opposing directions in representation space. Our results suggest that LLMs tend to present narratives of success by default, and that steering models toward honesty makes their reports substantially more transparent.

Jenny Y. Huang, Jiameng Fan, Ahmed Imtiaz Humayun +5

Massachusetts Institute of Technology · Google Research · Harvard University

Language Model Safety Evaluation Deception in Language Models LLM Honesty Language Model Auditing

Ask PDF

May 30, 2026

Sequential statistical inference for Large Language Models: Representation, validity, and monitoring

This discussion argues that sequential statistical inference can naturally contribute to LLM trustworthiness. In deployment, LLM systems are queried repeatedly, conditioned on evolving contexts, and incorporate user or tool feedback, and may exhibit behavioral shifts after model updates or distribution changes. The discussion is organized around three tasks: representation, modeling LLM interactions as dependent stochastic processes rather than isolated prompt--response pairs; validity, developing uncertainty guarantees that remain meaningful under dependence, repeated use, and adaptation; and monitoring, using sequential alarms and change-point detection to identify shifts in calibration, hallucination rates, refusal behavior, fairness, or other task-relevant properties. This perspective complements recent surveys by viewing trustworthy LLM deployment as a problem of statistical process control.

Yao Xie

H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology

Language Model Calibration LLM Reliability LLM Uncertainty Estimation Trustworthy AI

Ask PDF

Abstract

Evidence-Grounded Generation

Explore similar work

CheckSupport: A Local LLM-Powered Tool for Automated Manuscript Submission Checklist Selection and Completion

Language Models Are "Insecure" Reporters

Sequential statistical inference for Large Language Models: Representation, validity, and monitoring