Do LLMs talk like us? This question intrigues a multitude of scholar and it is relevant in many fields, from education to academia. This work presents an interpretable statistical feature for distinguishing human written and LLMs generated dialogue. We introduce a lightweight metric derived from semantic categories distribution. Using the Empath lexical analysis framework, each text is mapped to a set of thematic intensity scores. We define semantic delta as the difference between the two most dominant category intensities within a dialogue, hypothesizing that LLM outputs exhibit stronger thematic concentration than human discourse. To evaluate this hypothesis, conversational data were generated from multiple LLM configurations and compared against heterogeneous human corpora, including scripted dialogue, literary works, and online discussions. A Welch t-test was applied to the resulting distributions of semantic delta values. Results show that AI-generated texts consistently produce higher deltas than human texts, indicating a more rigid topics structure, whereas human dialogue displays a broader and more balanced semantic spread. Rather than replacing existing detection techniques, the proposed zero-shot metric provides a computationally inexpensive complementary signal that can be integrated into ensemble detection systems. These finding also contribute to the broader empirical understanding of LLM behavioural mimicry and suggest that thematic distribution constitutes a quantifiable dimension along which current models fall short of human conversational dynamics.
Figures & tables
Figure 1: Our proposed metric fits into the broader AI detection methods ecosystem as a one-shot, "white-box" statistical tool.
Figure 2: Lexical Analysis Pipeline using the Empath Library. After initial Tokenization, each token is matched against a Pre-Defined Lexicon consisting of 200+ Pre-Built-in Clusters. The final output represents the statistical distribution of these semantic categories within the analyzed text.
Figure 3: Empath analysis of various AI generated dialogues consistently identifies a gap of intensity value between the leading two semantic categories, suggesting a high degree of thematic concentration in the model’s responses.
Figure 4: Empath analysis of various human-written texts consistently identifies a small gap of intensity value between the leading two semantic categories, suggesting a more dynamic and less rigid topic structure.
Figure 5: Comparison of mean Shannon entropy between human-written texts (Friends, Shakespeare, Reddit; aggregated as “Human”) and AI-generated texts (“AI”). The Human bar includes the three underlying human datasets to show internal variability and the overall mean. AI exhibits a slightly lower entropy than Human, indicating marginally higher predictability; although the gap is small, it is consistent with the expected trend of reduced lexical diversity in AI-generated content.
Figure 6: AI generated dialogues consistently showed a larger delta between Top1 and Top2 categories, in comparison with Human dialogues. A Welch T-Test was conducted to test the null hypothesis that the observed difference between the mean values is attributable to random noise. Since the p<0.05 , the null hypothesis was rejected. There is a statistical significative difference between AI and Human dialogues.
As AI systems integrate into online spaces, differentiating them from humans in conversations is increasingly important. We present Inverse Turing Bench, a benchmark that evaluates LLMs and other models on their ability to differentiate humans and AI in multi-turn text. The benchmark provides a collection of paired dialogue transcripts, wherein one dialogue is between two humans and the other is between a human and an AI. The task is to correctly identify which dialogue is human-only vs. human-AI. We evaluated a preliminary set of models against this benchmark, and found that GPTZero, Claude Opus-4.6, and GPT-5.5 achieve the highest accuracy: 89.41%, 77.92%, and 75.94% respectively. Our results suggest that statistical approaches to detection have semantic blind spots, but semantic approaches are susceptible to persona-prompting. Our work speaks to the Inverse Turing Test and motivates human-AI differentiation as a critical capability for AI systems. Our live benchmark can be found at https://huggingface.co/spaces/roc-hci/Inverse-Turing-Bench-Leaderboard.
William Hager, Ishika Rathi, Masum Hasan +1
University of Rochester · Independent Researcher · Stony Brook University
Large Language Models (LLMs) are now capable of generating highly fluent, human-like text. They enable many applications, but also raise concerns such as large scale spam, phishing, or academic misuse. While much work has focused on detecting LLM-generated text, only limited work has gone into understanding the stylistic differences between human-written and machine-generated text. In this work, we perform a large scale analysis of stylistic variation across human-written text and outputs from 11 LLMs spanning 8 different genres and 4 decoding strategies using Douglas Biber's set of lexicogrammatical and functional features. Our findings reveal insights that can guide intentional LLM usage. First, key linguistic differentiators of LLM-generated text seem robust to generation conditions (e.g., prompt settings to nudge them to generate human-like text, or availability of human-written text to continue the style); second, genre exerts a stronger influence on stylistic features than the source itself; third, chat variants of the models generally appear to be clustered together in stylistic space, and finally, model has a larger effect on the style than decoding strategy, with some exceptions. These results highlight the relative importance of model and genre over prompting and decoding strategies in shaping the stylistic behavior of machine-generated text.
Swati Rallapalli, Shannon Gallagher, Ronald Yurko +4
Software Engineering Institute, AI Division, Carnegie Mellon University · Department of Statistics & Data Science, Carnegie Mellon University
As LLMs become increasingly integrated into daily life, understanding how their presence will shape human linguistic behavior is an open question. We present a large-scale study of linguistic convergence in human-LLM dialogue, examining how humans and LLMs accommodate each other's linguistic style during multi-turn conversations. Using an asymmetric convergence metric on WildChat, a corpus of real-world ChatGPT transcripts, we find that while LLMs significantly overconverge toward their users on both function word and open-class features across eight languages, human convergence rates in this setting are broadly consistent with human-human baselines. These findings suggest that accommodation in human-LLM dialogue is asymmetric: while LLMs dramatically overfit to their users' style, humans linguistically accommodate LLMs no differently than they would another person.
Terra Blevins
Khoury College of Computer Sciences Northeastern University, Boston, MA