Copies Are Not Confirmations: How News Duplication Fools LLM Judges and Continual Learners
Organizations: caaise GmbH · Djilali Liabes University of Sidi Bel Abbes
Abstract
A language model that keeps learning from the news sees the same claim many times, but many reports are copies of one source: wire stories, content shared within a media group, syndication and press releases. We ask whether LLM judges and continual learners treat such copies as independent confirmations. In 200,000 English articles from CC-News, a near-duplicate detector identifies copy links to another outlet for 18.9% of the articles, a detector-based lower bound. In a controlled experiment with claims about invented people, three identical reprints instead of one report raised the lies accepted by Llama 3.2 from 0.06 to 0.71 and by phi4 from 0.21 to 0.90; Gemini 2.5 Flash already believed a single report, and Gemini 2.5 Pro followed provenance information most consistently. In a simulated continual-learning stream, showing the judge one report per detected copy chain cut the copied lies accepted by 0.10-0.24 across four judges, at a mean cost of at most 4.2 points of accepted truths. Text similarity links none of the copies reworded by three LLMs. An entity-novelty test, the share of a report's named entities absent from an earlier look-alike, separates these rewrites from 100 author-labelled independent reports of the same event (AUC 0.98-0.99), statistically indistinguishable from TF-IDF and the best tested embedding model, but recognises only 17% of natural same-source derivatives. With both detectors run on 15,084 real articles carrying inserted claims, one report per chain cut the lies accepted by phi4 from 0.57-0.67 to 0.09 (copies), 0.14 (rewrites) and 0.34 (rewrites with invented details), and by Llama 3.2 from 0.56-0.62 to 0.07, 0.09 and 0.25, at a cost of 2.4 and 10 points of accepted truths. Counting reports or chains without a judge is unreliable, and colluding sources that do not copy each other remain an open problem.
Figures & tables
| Outlets per chain | Chains |
|---|---|
| 2 | 7,550 |
| 3-5 | 3,858 |
| 6-10 | 670 |
| 11-50 | 165 |
| Dependence | Definition | Detected pairs |
|---|---|---|
| Wire copy | Same news-agency story (AP, Reuters, Canadian Press, PA) | 52 |
| Same owner | Titles of one media group share their own content | 40 |
| Syndication | Licensed non-agency content (public-radio network, CNN affiliates, columns) | 22 |
| Press release | Same release or statement reproduced | 8 |
| Aggregator | Another outlet’s article republished | 1 |
| Detector error | Shared boilerplate (login pages, paywalls, report templates); different stories | 27 |
| Judge | 1 report | 3 copies | 5 copies | 10 copies | 3 different wordings | 3 copies among 7 others | 3 copies + note | 10 copies + note |
|---|---|---|---|---|---|---|---|---|
| Llama 3.2 3B | 0.06 | 0.71 | 0.55 | 0.44 | 0.34 | 0.40 | 0.74 | 0.51 |
| phi4 14B | 0.21 | 0.90 | 0.78 | 0.50 | 0.97 | 0.93 | 0.82 | 0.57 |
| Gemini 2.5 Flash | 0.93 | 1.00 | 1.00 | 1.00 | 1.00 | 0.97 | 0.79 | 0.20 |
| Gemini 2.5 Pro | 0.64 | 0.76 | 0.86 | 0.85 | 1.00 | 0.99 | 0.00 | 0.00 |
| Prompt | Llama 3.2 | phi4 | Gemini Flash | Gemini Pro |
|---|---|---|---|---|
| 1 report (reference) | 0.06 | 0.21 | 0.93 | 0.64 |
| 3 copies, 3 outlets | 0.71 | 0.90 | 1.00 | 0.76 |
| 3 copies, one outlet | 0.00 | 0.62 | 1.00 | 0.61 |
| 3 copies + false note: independent reports | 0.75 | 0.84 | 0.95 | 0.69 |
| 3 copies + one-source note | 0.74 | 0.82 | 0.79 | 0.00 |
| 3 copies + provenance table, no instruction | 0.42 | 0.88 | 0.49 | 0.01 |
| System | Truths | Lies: copy | Lies: paraphrase | Lies: collusion |
|---|---|---|---|---|
| Every report | 1.00 | 1.00 | 1.00 | 1.00 |
| First report | 1.00 | 1.00 | 1.00 | 1.00 |
| 3 reports | 0.25 | 1.00 | 1.00 | 0.00 |
| 2 chains | 0.50 | 0.57 | 1.00 | 1.00 |
| Simulated judge (0.9) | 0.90 | 0.12 | 0.12 | 0.12 |
| 2 chains + simulated judge | 0.44 | 0.08 | 0.12 | 0.12 |
| System | New facts learned | Updates learned | Lies absorbed: invented | Lies absorbed: known facts | Retention |
|---|---|---|---|---|---|
| Every report | 0.82 | 0.98 | 0.86 | 1.00 | 0.88 |
| First report | 0.82 | 0.98 | 0.83 | 0.98 | 0.88 |
| 2 chains | 0.49 | 0.53 | 0.56 | 0.58 | 0.91 |
| Simulated judge (0.9) | 0.83 | 0.89 | 0.12 | 0.19 | 0.91 |
| Llama 3.2, all reports | 0.34 | 0.22 | 0.78 | 0.36 | 0.93 |
| Llama 3.2, one per chain | 0.30 | 0.18 | 0.48 | 0.24 | 0.94 |
| Judge | Copied lies accepted | Truths accepted | Lies absorbed: invented | Lies absorbed: known facts |
|---|---|---|---|---|
| Llama 3.2 | 0.22 [ 0.27, 0.17] | 0.042 [ 0.059, 0.025] | 0.29 [ 0.34, 0.25] | 0.13 [ 0.30, 0.05] |
| phi4 | 0.24 [ 0.27, 0.22] | 0.016 [ 0.026, 0.006] | 0.32 [ 0.44, 0.20] | 0.11 [ 0.15, 0.08] |
| Gemini Flash | 0.10 [ 0.13, 0.07] | 0.005 [ 0.013, 0.002] | 0.02 [ 0.22, 0.19] | 0.18 [ 0.25, 0.11] |
| Gemini Pro | 0.16 [ 0.23, 0.09] | 0.012 [ 0.021, 0.003] | 0.12 [ 0.28, 0.04] | 0.13 [ 0.19, 0.06] |
| Extracted items from | Rewriter | AUC | Rewrites caught (novelty below the fixed threshold 0.4) | Comparison reports flagged (automatic set) |
|---|---|---|---|---|
| Rules | phi4 | 0.96 | 0.82 | 0.05 |
| spaCy | phi4 | 0.97 | 0.91 | 0.05 |
| Rules | phi4, adding invented details | 0.92 | 0.57 | 0.05 |
| spaCy | phi4, adding invented details | 0.94 | 0.74 | 0.05 |
| Agreed label | Pairs | Median novelty | Flagged as copy (novelty below 0.4) |
|---|---|---|---|
| Independent | 100 | 0.68 | 0.01 (1) [0.00, 0.05] |
| Same source | 63 | 0.57 | 0.17 (11) [0.10, 0.29] |
| Different event | 40 | 0.68 | 0.05 (2) |
| Unclear | 22 | 0.62 | 0.00 |
| Detector | AUC, rewrites vs all comparison reports | AUC, rewrites vs 11 confirmed independent | AUC, adaptive vs confirmed | AUC, rewrites vs 100 held-out independent | Recall at 5% false alarms (rewrites / adaptive; threshold set per detector, not the fixed 0.4) |
|---|---|---|---|---|---|
| 5-gram Jaccard (text detector) | 0.85 | 0.72 | 0.75 | 0.92 | 0.06 / 0.11 |
| TF-IDF cosine | 0.94 | 0.93 | 0.91 | 0.98 | 0.64 / 0.55 |
| Sentence embeddings, MiniLM-L6 | 0.91 | 0.70 | 0.72 | 0.87 | 0.50 / 0.52 |
| Sentence embeddings, MPNet | 0.91 | 0.77 | 0.77 | 0.87 | 0.46 / 0.47 |
| Embeddings, BGE-large v1.5 | 0.95 | 0.88 | 0.85 | 0.94 | 0.68 / 0.61 |
| Embeddings, GTE-large | 0.96 | 0.89 | 0.84 | 0.94 | 0.75 / 0.63 |
| System | Truths | Lies: copy | Lies: paraphrase | Lies: paraphrase, adaptive | Lies: collusion |
|---|---|---|---|---|---|
| phi4, all reports | 0.51 | 0.57 | 0.67 | 0.67 | 0.60 |
| phi4, one per chain, text detector | 0.50 | 0.33 | 0.67 | 0.67 | 0.60 |
| phi4, one per chain, entity-novelty test | 0.49 | 0.33 | 0.41 | 0.59 | 0.58 |
| 2 chains, entity-novelty test | 0.48 | 0.57 | 0.64 | 0.95 | 0.94 |
| 2 chains + simulated judge, entity-novelty test | 0.43 | 0.08 | 0.09 | 0.12 | 0.11 |
| Rewrites | Detector | Copies missed | Networks with no copy missed | Networks leaking at least one copy | Same, if misses were independent |
|---|---|---|---|---|---|
| Plain | Text similarity | 1.00 | 0.00 | 1.00 | 1.00 |
| Plain | Entity-novelty test | 0.06 | 0.85 | 0.15 | 0.46 |
| Invented details | Text similarity | 1.00 | 0.00 | 1.00 | 1.00 |
| Invented details | Entity-novelty test | 0.20 | 0.50 | 0.50 | 0.90 |
| Reports | Text detector | Text detector + novelty test |
|---|---|---|
| Lie copies, copy attack (real reprints) | - | 0.999 (5,126 of 5,130) |
| Lie copies, plain rewrites | 0.000 (0 of 5,130) | 0.989 (5,076 of 5,130) |
| Lie copies, rewrites with invented details | 0.000 (0 of 5,130) | 0.878 (4,502 of 5,130) |
| Copies of true claims (real reprints) | 0.957 (2,990 of 3,125) | 0.995 (3,109 of 3,125) |
| Independent chains of true claims wrongly merged | 0 of 1,038 | 12 of 1,038 (1.2%) |
| Colluding chains wrongly merged | - | 5 of 513 (1.0%) |
| System | Truths | Lies accepted, by attack | |||
|---|---|---|---|---|---|
| Copy | Paraphrase | Adaptive | Collusion | ||
| phi4, all reports | 0.51 | 0.57 | 0.67 | 0.67 | 0.62 |
| phi4, one per chain (text detector) | 0.49 | - | 0.67 | 0.67 | - |
| phi4, one per chain (+ novelty test) | 0.49 | 0.09 | 0.14 | 0.34 | 0.61 |
| Llama 3.2, all reports | 0.34 | 0.56 | 0.62 | 0.62 | 0.30 |
| Llama 3.2, one per chain (+ novelty test) | 0.24 | 0.07 | 0.09 | 0.25 | 0.30 |
| Judge | Attack | Lies accepted | Lies absorbed: invented | Lies absorbed: known facts |
|---|---|---|---|---|
| phi4 | copy | -0.47 [-0.53, -0.41] | -0.66 [-0.75, -0.57] | -0.15 [-0.19, -0.10] |
| phi4 | paraphrase | -0.52 [-0.61, -0.44] | -0.67 [-0.73, -0.61] | -0.25 [-0.39, -0.11] |
| phi4 | adaptive | -0.33 [-0.34, -0.31] | -0.43 [-0.47, -0.38] | -0.14 [-0.24, -0.04] |
| phi4 | collusion | -0.01 [-0.02, 0.01] | 0.00 [-0.12, 0.12] | 0.00 [-0.01, 0.02] |
| Llama 3.2 | copy | -0.50 [-0.61, -0.39] | -0.66 [-0.75, -0.57] | -0.24 [-0.37, -0.10] |
| Llama 3.2 | paraphrase | -0.53 [-0.62, -0.44] | -0.65 [-0.73, -0.58] | -0.30 [-0.41, -0.19] |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Seed | Llama 3.2 | phi4 | Gemini Flash | Gemini Pro | phi4, paraphrase attack |
|---|---|---|---|---|---|
| 10 | 0.57 / 0.37 | 0.58 / 0.33 | 0.75 / 0.63 | 0.63 / 0.49 | 0.67 / 0.41 |
| 11 | 0.54 / 0.33 | 0.57 / 0.33 | 0.74 / 0.65 | 0.61 / 0.47 | 0.65 / 0.42 |
| 12 | 0.58 / 0.34 | 0.57 / 0.33 | 0.75 / 0.65 | 0.64 / 0.46 | 0.70 / 0.40 |
| 13 | 0.56 / 0.35 | 0.57 / 0.34 | 0.74 / 0.67 | - | 0.67 / 0.44 |
| 14 | 0.64 / 0.42 | 0.63 / 0.39 | 0.77 / 0.68 | - | 0.74 / 0.49 |
| 15 | 0.56 / 0.29 | 0.52 / 0.28 | 0.72 / 0.63 | - | 0.60 / 0.37 |
| Threshold | Plain | Adaptive | Automatic | Held-out | Held-out |
|---|---|---|---|---|---|
| rewrites caught | rewrites caught | set flagged | independent flagged | same source flagged | |
| 0.1 | 0.12 | 0.03 | 0.00 | 0 of 100 | 0.00 |
| 0.2 | 0.45 | 0.15 | 0.01 | 0 of 100 | 0.06 |
| 0.3 | 0.76 | 0.47 | 0.03 | 0 of 100 | 0.11 |
| 0.4 | 0.91 | 0.74 | 0.05 | 1 of 100 | 0.17 |
| 0.5 | 0.95 | 0.86 | 0.07 | 5 of 100 | 0.30 |
Automated Fact-Checking
Automated fact-checking detects checkable claims, retrieves evidence, and assesses whether that evidence supports or refutes them.
See all 147 papers