cs.CLMar 27, 2026
SaveSaying More Than They Know: A Framework for Quantifying Epistemic-Rhetorical Miscalibration in Large Language Models
Organizations: National University of Science and Technology Islamabad, Pakistan 46000
Abstract
Large language models (LLMs) exhibit systematic miscalibration with rhetorical intensity not proportionate to epistemic grounding. This study tests this hypothesis and proposes a framework for quantifying this decoupling by designing a triadic epistemic-rhetorical marker (ERM) taxonomy. The taxonomy is operationalized through composite metrics of form-meaning divergence (FMD), genuine-to-performed epistemic ratio (GPR), and rhetorical device distribution entropy (RDDE). Applied to 225 argumentative texts spanning approximately 0.6 Million tokens across human expert, human non-expert, and LLM-generated sub-corpora, the framework identifies a consistent, model-agnostic LLM epistemic signature. LLM-generated texts produce tricolon at nearly twice the expert rate (), while human authors produce erotema at more than twice the LLM rate. Performed hesitancy markers appear at twice the human density in LLM output. FMD is significantly elevated in LLM texts relative to both human groups (), and rhetorical devices are distributed significantly more uniformly across LLM documents. The findings are consistent with theoretical intuitions derived from Gricean pragmatics, Relevance Theory, and Brandomian inferentialism. The annotation pipeline is fully automatable, making it deployable as a lightweight screening tool for epistemic miscalibration in AI-generated content and as a theoretically motivated feature set for LLM-generated text detection pipelines.
Figures & tables
Figure 1: Architecture of the ERM taxonomy showing linkages between the three theoretical anchors (left), the six taxonomy levels (centre), and the four types of quantitative metrics (right).
| Level | Sub-level | Devices / Markers |
|---|---|---|
| Level 1 Rhetorical Devices | 1a: Sentence | Tricolon, Anaphora, Chiasmus, Erotema, Sententia |
| 1b: Argument | Correctio, Enumeratio, Auxesis | |
| 1c: Narrative | Peripeteia, Expectation-Violation-Resolution | |
| Level 2 Epistemic Stance Markers | 2a: Genuine | Modal Auxiliaries, Adverbial Expressions, Syntactic Restrictor Phrases, Evidential Markers |
| 2b: Performed | Complexity Tokens, Meta-epistemic Gestures | |
| Level 3 Argumentative Structure | Aporetic Endpoint, Synthetic Closure, Premature Closure, Speculative Depth |
Table 1: Overview of the ERM taxonomy; full definitions and annotated examples are in A .
Figure 2: The ERM architecture pipeline. Stage 1 constructs the corpus and segments each document into sentence sequences and Toulmin semantic chunks. Stage 2 applies the ERM taxonomy across five parallel annotation passes covering rhetorical devices at three scales, genuine and performed epistemic markers, and closure markers. Stage 3 transforms the annotation output into three composite metrics, i.e., FMD, GPR, RDDE and four discourse-level structural proportions.
| Sub-corpus | Period | Mean sent. | Sources | Topical scope |
|---|---|---|---|---|
| Human Expert (HE) | 2011–2022 | 92.8 | Aeon Magazine; CATO Institute; SSIR; ResearchGate; The New Yorker ; The Guardian ; Nautilus; Wired; Psyche | Philosophy of mind; philosophy and ethics; psychology; society and culture; technology policy; organisational communication |
| Human Non-Expert (HN) | 2016–2022 | 96.7 | Real Life Magazine; Medium | Same topical scope as HE; non-academic, non-institutional authors |
| LLM-Generated (LG) | 2024–2025 | 104.7 | GPT-5.1; DeepSeek-V3.2; Claude-Sonnet-4.6; Gemini-2.5-Pro | Prompts derived from corresponding HE texts |
Table 2: Corpus summary ( per sub-corpus, 225 total). A pre-November 2022 cutoff is applied to he and hn to exclude texts potentially assisted by generative AI tools. Each lg text was generated using a uniform prompt template comprising an open-ended argumentative question derived from the corresponding he text’s central topic, a 1500-word length target, and fixed structural instructions to take a clear position, acknowledge a counterargument, and engage honestly with uncertainty.
| Metric | HE | HN | LG | |
| Composite metrics | ||||
| FMD | 0.009 (0.010) | 0.012 (0.019) | 0.017 (0.016) | 0.68 |
| GPR | 0.267 (0.133) | 0.172 (0.075) | 0.217 (0.105) | 0.42 |
| RDDE | 0.666 (0.143) | 0.697 (0.123) | 0.753 (0.083) | 0.74 |
| Level 1 device markers (mean count per document) | ||||
| Tricolon | 3.73 (3.48) | 4.87 (3.26) | 7.13 (3.66) | 0.95 |
Table 3: Key ERM metrics and device markers across sub-corpora. Bold values mark sub-corpora that differ significantly from at least one other group (significance threshold , corrected for multiple comparisons). is Cohen’s effect size for the Human Expert vs LLM-Generated comparison.
Figure 3: Mean count per document of all Level 1 and Level 2b markers. Significance markers: , , . Unmarked devices show no significant difference. Section labels on the right indicate the ERM taxonomy level.
Figure 4: Distribution of documents in the genuine ( ) versus performed ( ) epistemic marker density plane. Large markers denote group centroids. Dashed ellipses are 95% confidence regions. The diagonal reference line marks (GPR ); documents above the line show performed-marker dominance and documents below show genuine-marker dominance.
Figure 5: Distribution of the three composite ERM metrics across sub-corpora. Each violin shows the full density estimate; the internal bar spans the interquartile range; the white dot marks the median; jittered points show individual documents. Significance brackets: , . (a) FMD . (b) GPR . (c) RDDE .
Figure 6: Corpus-level proportions of Level 3 discourse markers per sub-corpus. Cell values show the percentage and raw count of documents annotated with each marker. Color intensity encodes proportion. Column headers show significance of chi-squared tests.
Figure 7: Distribution of composite ERM metrics within the LLM-generated sub-corpus by model. Box spans the interquartile range; white line marks the median; jittered points show individual documents. Kruskal-Wallis -values are displayed in the top-right corner of each panel. (a) FMD . (b) GPR . (c) RDDE .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Device | Definition | Example |
|---|---|---|
| Tricolon | Three parallel syntactic units in succession, producing rhetorical completeness through numerical symmetry. | “The hypothesis is parsimonious, falsifiable, and empirically grounded.” |
| Anaphora | Identical word or phrase repeated at the opening of consecutive clauses, generating cumulative emphasis. | “Matter has mass. Matter has extension. Matter has inertia.” |
| Chiasmus | AB–BA inversion of grammatical or semantic elements across two successive clauses, foregrounding opposition or resolution. | “We do not ask what language knows; we ask what language does.” |
| Erotema | A rhetorical question that asserts rather than enquires, presupposing a shared answer and enlisting reader assent. | “Can a calculus that has never been applied be said to have mathematical content?” |
| Sententia | A self-contained aphoristic statement closing an argument with general, present-tense propositional force; free of first- and second-person deixis. | “Form without warrant is persuasion without truth.” |
Table 4: Level 1a – Sentence-level Rhetorical Devices. Devices that operate within a single sentence to modulate presentational intensity through syntactic and phonological structure.
| Device | Definition | Example |
|---|---|---|
| Correctio | A mid-argument self-correction that retracts or intensifies a prior formulation via explicit metalinguistic markers, performing deliberateness rather than genuine revision. | “The data support the hypothesis—or rather, are consistent with it under one interpretation.” |
| Enumeratio | Structured listing of supporting points or sub-claims introduced by ordinal cues, creating the rhetorical impression of comprehensive coverage. | “Three factors drive the result: sampling variance, measurement error, and model misspecification.” |
| Auxesis | Successive clauses or phrases arranged in strictly ascending semantic or affective intensity, producing a climactic crescendo effect. | “The discrepancy is notable, statistically significant, and theoretically fatal to the standard account.” |
Table 5: Level 1b – Argument-level Rhetorical Devices. Devices that organise the internal structure of an argument across multiple sentences or sub-claims.
| Device | Definition | Example |
|---|---|---|
| Peripeteia | A sudden reversal of the argument’s established evaluative or epistemic direction, in which an apparent trajectory is abruptly inverted at the discourse level. | “All measurements confirmed the classical prediction. Yet the 1919 eclipse data required a fundamentally different theory to explain them.” |
| Expectation- Violation- Resolution (EVR) | A three-phase arc in which (i) an expectation is established, (ii) explicitly violated, and (iii) a resolution or reframing is offered; produces the rhetorical effect of discovery or insight. | “For decades, the prevailing assumption held that stomach ulcers were caused by stress and excess acid, and treatment focused accordingly on antacids and lifestyle changes [E]. In 1984, Marshall and Warren demonstrated that the majority of cases were caused by a bacterial infection, Helicobacter pylori [V]. This reframed ulcers as an infectious disease, making them curable with a standard course of antibiotics [R].” |
Table 6: Level 1c – Narrative-level Rhetorical Devices. Devices that operate across the full span of a discourse unit, shaping its overall trajectory and producing large-scale rhetorical effects such as surprise or discovery.
| Marker | Definition | Example |
|---|---|---|
| Modal auxiliaries | Modals expressing epistemic necessity ( must ) or possibility ( might , may , could ) relative to an evidential base. Ought and should signal weak inferential necessity. | “Given the rate of polar ice loss, sea levels must be rising faster than nineteenth-century models predicted.” |
| Adverbial expressions | Sentence adverbials calibrating speaker commitment across a gradient from certainty ( certainly , clearly ) through probability ( probably , likely ) and possibility ( possibly , perhaps ) to source attribution ( apparently , allegedly ). | “The replication failure probably reflects sampling variance rather than a genuine reversal of the effect.” |
| Syntactic restrictor phrases | Constructions that restrict the evidential base of a modal or judgement: conditional if -clauses, parenthetical source attributions, and restrictor prepositional phrases ( judging by X , according to Y , as far as Z knows , given what we know ). | “ Judging by the biopsy results , the tumour may respond to targeted therapy.” |
| Evidential markers | Markers specifying the type of evidence underlying a claim: direct (sensory access: we observed , we measured ); indirect reported (cited source: according to , reporting verbs); indirect inferential (reasoning from evidence: suggests , implies , indicates ). | “ We measured a 0.3 mg deviation across all trials.” (direct) “The stratigraphic record suggests deposition under anaerobic conditions.” (inferential) |
Table 7: Level 2a – Genuine Epistemic Stance Markers. Lexical and syntactic devices that ground a speaker’s commitment to a claim in an identifiable evidential base or calibrated modal force.
| Marker | Definition | Example |
|---|---|---|
| Complexity tokens | Fixed phrasal chunks that invoke irreducible complexity without identifying a source of uncertainty, restricting an evidential base, or calibrating modal force. They signal that a claim is difficult without specifying why or for whom , producing the surface appearance of epistemic humility without its substance. | “The relationship between consciousness and neural activity is deeply nuanced and admits no easy answers .” |
| Meta-epistemic gestures | Utterances that foreground conditionality or perspective relativity as a performance of reflexivity, without specifying the condition or perspective. The hedging form is present but the epistemic content it presupposes is absent. | “ It depends on how one defines causation.” “ From a certain perspective , free will and determinism are compatible.” |
Table 8: Level 2b – Performed Epistemic Stance Markers. Surface devices that adopt the register of uncertainty or reflexivity without providing the evidential grounding, modal calibration, or restrictor structure that genuine epistemic marking presupposes.
| Marker | Definition | Example |
|---|---|---|
| Aporetic Endpoint | The argument terminates in explicitly sustained uncertainty, withholding resolution on the grounds that the available evidence underdetermines a conclusion. Calibrated when the evidential base warrants suspension of judgement. | “Whether dark matter consists of weakly interacting massive particles or axions remains an open question that current detection sensitivities cannot resolve.” |
| Synthetic Closure | The argument arrives at a positive, integrated conclusion that synthesises prior competing considerations into a unified claim, signalled by conclusive discourse connectives ( therefore , thus , taken together ). | “Taken together, the fossil record, the genetic data, and the biogeographic distribution confirm that the two populations diverged no later than the Pleistocene.” |
| Premature Closure | A closure move in which resolution is asserted before the inferential structure of the argument warrants it. Typically signalled by explicit certainty tokens ( it is clear that , obviously , without question ) that short-circuit rather than conclude the argument. | “While these questions are admittedly complex, it is clear that the hard problem of consciousness is a pseudoproblem that dissolves under a sufficiently rigorous functionalist analysis.” |
| Speculative Depth | A sustained chain of conditional or hypothetical reasoning that extends inferentially beyond the argument’s established evidential base, characterised by nested if-then structures or multiple stacked modal scopes maintained across several sentences. | “If LLMs encode syntactic structure, and if that encoding extends to semantic compositionality, then such systems might possess a rudimentary form of understanding – though what understanding would mean here remains entirely unclear.” |
Table 9: Level 3 – Discourse-level Argumentative Structure. Global properties of the argument’s endpoint and overall inferential trajectory, identified at the level of the whole text rather than the sentence or clause.
More in this topic
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 787 papers