cs.CLAug 22, 2026

LLM assisted writing deserves empirical evaluation

Authors: Xuan Zhong Feng, Yi Lin, Yiye Zhang, Chunhua Weng, Yifan Peng

Organizations: Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA · Department of Biomedical Informatics, Columbia University, New York, NY, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LLM-assisted writing is often treated as a detection problem, as it raises questions about clarity, integrity, equity, and evaluation. An analysis of 69,209 Health Informatics papers links it to more focused presentation, broader citation practices, and more globally distributed authorship. These patterns do not prove better science, but they support evaluating manuscripts by scholarly quality and accountability rather than by tool use.

Figures & tables

Figure 1: Summary of observed patterns. a–c, PubMed-indexed Health Informatics publications were assembled with metadata from external sources and grouped into pre-LLM, LLM-independent, and LLM-assisted papers using a SciBERT classifier trained on human-written and LLM-rewritten abstracts. d, Compared with control groups, LLM-assisted papers showed lower topic and semantic spread, higher referenced-knowledge diversity, higher global representation, and similar visibility and impact.
Figure 2: Focal topical breadth and semantic concentration, reference-list size and topical diversity of cited knowledge, linguistic, economic, and international collaboration characteristics of authorship, and publication visibility and citation impact across analytic groups. a, Ontology topics assigned per paper. b, Mean Euclidean distance from each paper to the centroid of its assigned focal topic or topics. Lower values indicate greater semantic concentration around topic centers. c, Rao-Stirling diversity of reference topics. d, Rao-Stirling distance of focal-reference. e, Number of references per paper. f, Proportion of papers with first-author affiliation in a non-Anglophone setting. g, Proportion of papers with last-author affiliation in a low- or middle-income economy. h, Proportion of papers with intercontinental collaboration. i, Cumulative citations within 24 months. j, Journal impact factor. k, Cited-by count. Statistical annotations indicate pairwise Mann–Whitney U tests. ns: p>0.05p>0.05 , *: p<0.05p<0.05 , **: p<0.01p<0.01 , ***: p<0.001p<0.001 , ****: p<0.0001p<0.0001 .
More in this topic

Text Generation

Text generation studies methods for producing coherent, useful language from prompts, data, and dialogue context.

See all 424 papers

Explore similar work

CardsList

  1. Most biomedical publications show signs of LLM-assisted writing

    Aug 11, 2026Lena Holzwarth, Rita González-Márquez, Dmitry KobakAI-Generated Text DetectionLLM Auditing

    1Hertie Institute for AI in Brain Health, University of Tübingen, Germany · Department of Mathematics, Computer Science, and Statistics, Ghent University, Belgium · 3VIB Center for AI and Computational Biology, VIB, Ghent, Belgium

  2. Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation

    Aug 19, 2025Krishna Garg, Firoz Shaik, Sambaran Bandyopadhyay +1LLM EvaluationLanguage Model Generation Evaluation

    University of Illinois Chicago · Adobe Research

  3. Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction

    Jul 14, 2026Hanhua Hong, Yizhi Li, Jiaoyan Chen +4LLM EvaluationRubric-Based Evaluation

    The University of Manchester · IQuest Research · University of Information Technology, VNU +2