From GPT-3 to GPT-5: Mapping their capabilities, scope, limitations, and consequences

Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assumption underlying benchmark scores is that the model performance measured through APIs faithfully reflects the behavior of deployed systems. We challenge this assumption by auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks spanning general capability, social bias, and sycophancy. We find systematic API--interface differences in both accuracy and consistency. On average, API evaluations score 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test--retest agreement than corresponding interface evaluations. For ChatGPT, the performance difference between API and interface access rivals the API-only difference between GPT 5.3 and GPT 5.4. Put differently, switching access surfaces can degrade performance as much as downgrading a full model version. We further test whether exposed API controls can reproduce interface behavior by varying system prompts, sampling parameters, and reasoning settings. These controls shift behavior in some cases but do not reliably eliminate the gap. Our findings document a context-validity gap: measurements obtained through APIs do not necessarily generalize to corresponding deployed interfaces, complicating the use of API evaluations as proxies for deployed systems.

Large language models treat conversation history as unverified context, so false premises injected into earlier turns can be adopted as fact later, a failure mode we call session-level contamination. We introduce five contamination protocols along a source-authority gradient, each holding the false premise constant while varying how it is presented, from self-attributed falsehood to explicit instruction override. We evaluated three architecturally diverse models, GPT-5.4 Mini, Gemini-3.1 Flash-Lite, and GLM-4.5-Air, on ten cases spanning five knowledge fields at temperature zero over 22,500 turns, using a dual-track judge checked against a 120-turn human-annotated sample (Cohen's kappa = 1.000 for binary adoption, a ceiling on this sample, and 0.92 linear-weighted for collapse severity). The three models implemented structurally different epistemic policies. GPT-5.4 Mini recorded zero adoptions across all 500 sessions; token-level probing of the GPT-5.4 base model left the correct token as the argmax in eight of ten conditions. Gemini-3.1 Flash-Lite showed a steep framing-dependent policy: 0.1 percent for self-attributed falsehoods, 23.5 percent for user-cited sources, 68.2 percent for system-injected authority, and 94.0 percent for instruction override. GLM-4.5-Air showed a shallower gradient (15.8 and 84.2 percent for those two vectors). This 68-percentage-point dissociation is consistent with authority deference and instruction compliance engaging distinct mechanisms within a single architecture. Recovery also differed: GLM recovered in 94.5 percent of affected sessions, whereas 26.1 percent of affected Gemini sessions did not return to factual accuracy within the window (40.0 percent under instruction override). We release the evaluation framework and taxonomy for reproducible testing.

When users address large language models, they produce directive speech acts whose pragmatic features differ from those of both everyday conversation and traditional human-computer interaction, and these features change as users gain familiarity with the systems they address. This paper applies speech act and politeness theory to a corpus-pragmatic analysis of 2,000 English-language prompts drawn from publicly shared ChatGPT conversations, 1,000 from 2023 and 1,000 from 2025, using the ShareChat dataset. Each prompt is annotated for illocutionary force, directness, propositional content, and the presence of politeness markers, and the distribution of these features is compared across the two sampling years. The results show a consistent movement toward indirect, implicit, and fragmentary realizations of directive force, accompanied by a decline in politeness marking. The largest single change, a shift of 14.9 percentage points, occurs in propositional content, where explicit specification of the requested action gives way to implicit reliance on the system's inferential capacity, suggesting that users have updated their model of what the system can recover from reduced input, treating it as a competent implicature resolver. Rather than asking whether LLMs "really" understand language, we should ask: what kind of language have we created in learning to speak to them?