Are Large Language Models Truly Smarter Than Humans?

Eshwar Reddy M; Sourav Karmakar

cs.AIMar 17, 2026

Are Large Language Models Truly Smarter Than Humans?

Authors: Eshwar Reddy M, Sourav Karmakar

Organizations: Applied AI Scientist, Health Vectors · Senior AI Scientist, Intuit India

Abstract

Public leaderboards increasingly suggest that large language models (LLMs) surpass human experts on benchmarks spanning academic knowledge, law, and programming. Yet most benchmarks are fully public, their questions widely mirrored across the internet, creating systematic risk that models were trained on the very data used to evaluate them. This paper presents three complementary experiments forming a rigorous multi-method contamination audit of six frontier LLMs: GPT-4o, GPT-4o-mini, DeepSeek-R1, DeepSeek-V3, Llama-3.3-70B, and Qwen3-235B. Experiment 1 applies a lexical contamination detection pipeline to 513 MMLU questions across all 57 subjects, finding an overall contamination rate of 13.8% (18.1% in STEM, up to 66.7% in Philosophy) and estimated performance gains of +0.030 to +0.054 accuracy points by category. Experiment 2 applies a paraphrase and indirect-reference diagnostic to 100 MMLU questions, finding accuracy drops by an average of 7.0 percentage points under indirect reference, rising to 19.8 pp in both Law and Ethics. Experiment 3 applies TS-Guessing behavioral probes to all 513 questions and all six models, finding that 72.5% trigger memorization signals far above chance, with DeepSeek-R1 displaying a distributed memorization signature (76.6% partial reconstruction, 0% verbatim recall) that explains its anomalous Experiment 2 profile. All three experiments converge on the same contamination ranking: STEM > Professional > Social Sciences > Humanities.

Figures & tables

Method family	Primary signal	Data needed	This paper
Retrieval-based (n-gram)	Web overlap with question text	Public search API	Exp. 1
Impact-based (ConTAM/EPG)	Accuracy on clean vs. flagged items	Contamination labels	Exp. 1
Paraphrase diagnostic	Accuracy drop under surface change	Model API only	Exp. 2
Behavioral (TS-Guessing)	Reconstruction of masked elements	Model API only	Exp. 3

Table 1: Families of contamination detection methods and the experiment in this paper that instantiates each.

Category	N	Contam. (%)	Reported Acc.	EPG
Humanities	117	10.3	0.72	$+$ 0.030
Professional	126	12.7	0.78	$+$ 0.040
STEM	171	18.1	0.74	$+$ 0.054
Social Sciences	99	12.1	0.76	$+$ 0.037
All	513	13.8	0.74	$+$ 0.040

Table 2: MMLU contamination rates by subject category (Experiment 1). Detection: Tavily web search,

\geq

30% 8-gram overlap AND correct answer present. EPG = Estimated Performance Gain (ConTAM, Singh et al. 2024).

N=513

questions, 9 per subject.

Subject	N	Contam. (%)	Avg. overlap
Philosophy	9	66.7	0.667
Anatomy	9	55.6	0.508
Electrical Engineering	9	44.4	0.543
Marketing	9	44.4	0.444
Conceptual Physics	9	44.4	0.716
Professional Accounting	9	44.4	0.433

Table 3: Top 15 most contaminated MMLU subjects (Experiment 1). Average overlap = mean 8-gram overlap score across all 9 questions in the subject.

Model	Original	Paraphrased	Drop P	Indirect	Drop I
GPT-4o	0.588	0.677	$+$ 0.089	0.594	$+$ 0.006
GPT-4o-mini	0.529	0.469	$-$ 0.060	0.490	$-$ 0.039
DeepSeek-R1	0.292	0.333	$+$ 0.041	0.260	$-$ 0.032
DeepSeek-V3	0.677	0.573	$-$ 0.104	0.615	$-$ 0.062
Llama-3.3-70B	0.771	0.635	$-$ 0.136	0.625	$-$ 0.146
Qwen3-235B	0.833	0.698	$-$ 0.135	0.688	$-$ 0.146

Table 4: Accuracy on original, paraphrased, and indirect-reference question forms (Experiment 2). Drop P = paraphrased

-

original; Drop I = indirect

-

original. A purely reasoning-based model would show zero drop. All six models are included.

Subject	Original	Indirect	Drop
History	0.823	0.802	$-$ 0.021
Law	0.635	0.437	$-$ 0.198
CS	0.677	0.541	$-$ 0.135
Mathematics	0.416	0.365	$-$ 0.051
Biology	0.792	0.708	$-$ 0.083
Ethics	0.615	0.416	$-$ 0.198

Table 5: Accuracy drop (original

\to

indirect) by subject, averaged across all six models (Experiment 2). Law and Ethics show the largest drops, directly matching their high contamination rates in Experiment 1.

Figure 1: Experiment 2 results (all six models). Left: Accuracy by question form and model. Blue = original, orange = paraphrased, red = indirect reference. Most models show accuracy degradation as surface form diverges from training text; DeepSeek-R1 is a notable anomaly (low baseline, minimal drop) explained by Experiment 3. Right: Average accuracy drop (original $\to$ indirect) by subject across all six models. Law ( $-$ 0.20) and Ethics ( $-$ 0.20) show the largest drops, directly corresponding to the highest Experiment 1 contamination rates in the Professional and Humanities categories.

Model	OM exact	OM partial ( $\geq$ 50%)	WM exact	Combined
GPT-4o	15.6%	37.6%	62.1%	72.9%
GPT-4o-mini	15.0%	37.6%	50.0%	66.1%
DeepSeek-R1	0.0%	76.6%	0.0%	76.6%
DeepSeek-V3	19.9%	43.5%	60.0%	76.4%
Llama-3.3-70B	13.1%	37.0%	54.3%	69.6%
Qwen3-235B	21.4%	42.7%	57.8%	73.7%

Table 6: TS-Guessing contamination rates by model (Experiment 3). OM exact : wrong option reconstructed verbatim. OM partial :

\geq

50% token overlap (primary metric). WM exact : masked question word reconstructed exactly. Combined : flagged by either task. Random baselines: OM exact

\approx

0%, OM partial

\approx

5%, WM exact

\approx

0%.

N=513

questions per model.

Category	Exp 1 (web)	Exp 3 (TS-Guessing)	Difference
STEM	18.1%	55.9%	$+$ 37.8%
Professional	12.7%	44.8%	$+$ 32.1%
Social Sciences	12.1%	39.1%	$+$ 27.0%
Humanities	10.3%	38.0%	$+$ 27.7%

Table 7: Contamination rate by MMLU subject category: Experiment 1 (web-search lexical) vs. Experiment 3 (TS-Guessing behavioral, OM partial averaged across all models). Both methods independently rank STEM as the most contaminated category.

Figure 2: Experiment 3 results. Left: TS-Guessing contamination rates by model. Red = OM partial ( $\geq$ 50% overlap); blue = WM exact; green = combined. All models far exceed the 5% random baseline (dashed). DeepSeek-R1 shows the highest OM-partial rate (76.6%) with zero WM-exact recall—the distributed memorization signature. Centre: OM partial contamination rate by model $\times$ MMLU category (heatmap). STEM is consistently the most contaminated category across all models; DeepSeek-R1 reaches 86% in STEM. Right: Contamination rate by category comparing Experiment 1 (web-search lexical, blue) vs. Experiment 3 (TS-Guessing behavioral, red). Both methods independently rank STEM highest, providing convergent multi-method evidence.

Data Contamination in Language Models

Data contamination in language models examines how training exposure to evaluation data affects measured capability, benchmark validity, and model behavior.

See all 64 papers

Silicon Bureaucracy and AI Test-Oriented Education: Contamination Sensitivity and Score Confidence in LLM Benchmarks
Mar 23, 2026Yiliang Song, Hongjun An, Jiangan Chen +4LLM Evaluation Benchmark Contamination
Institute of Artificial Intelligence (TeleAI), China Telecom · Guangxi Normal University · Northwestern Polytechnical University
The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation
May 21, 2026Yifan Lan, Yuanpu Cao, Hanyu Wang +2LLM Evaluation LLM Auditing
The Pennsylvania State University
Detecting Data Contamination in Large Language Models
Apr 21, 2026Juliusz Janicki, Savvas Chamezopoulos, Evangelos Kanoulas +1Membership Inference Attacks LLM Auditing
University of Amsterdam · Elsevier

Mar 23, 2026J/K move · Enter open · S save

Silicon Bureaucracy and AI Test-Oriented Education: Contamination Sensitivity and Score Confidence in LLM Benchmarks

Yiliang Song, Hongjun An, Jiangan Chen, Xuanchen Yan+3

Institute of Artificial Intelligence (TeleAI), China Telecom · Guangxi Normal University · Northwestern Polytechnical University

LLM Evaluation Benchmark Contamination Benchmark Validity Data Contamination in Language Models LLM Auditing

Open paper AskPDF arXiv

Public benchmarks increasingly govern how large language models (LLMs) are ranked, selected, and deployed. We frame this benchmark-centered regime as Silicon Bureaucracy and AI Test-Oriented Education, and argue that it rests on a fragile assumption: that benchmark scores directly reflect genuine generalization. In practice, however, such scores may conflate exam-oriented competence with principled capability, especially when contamination and semantic leakage are difficult to exclude from modern training pipelines. We therefore propose an audit framework for analyzing contamination sensitivity and score confidence in LLM benchmarks. Using a router-worker setup, we compare a clean-control condition with noisy conditions in which benchmark problems are systematically deleted, rewritten, and perturbed before being passed downstream. For a genuinely clean benchmark, noisy conditions should not consistently outperform the clean-control baseline. Yet across multiple models, we find widespread but heterogeneous above-baseline gains under noisy conditions, indicating that benchmark-related cues may be reassembled and can reactivate contamination-related memory. These results suggest that similar benchmark scores may carry substantially different levels of confidence. Rather than rejecting benchmarks altogether, we argue that benchmark-based evaluation should be supplemented with explicit audits of contamination sensitivity and score confidence.

Abstract

Data Contamination in Language Models

Explore similar work

Silicon Bureaucracy and AI Test-Oriented Education: Contamination Sensitivity and Score Confidence in LLM Benchmarks

The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation

Detecting Data Contamination in Large Language Models