cs.CLMar 23, 2026

Cross-Context Verification: Hierarchical Detection of Benchmark Contamination through Session-Isolated Analysis

Authors: Tae-Eun Song

Organizations: Daejeon Jungang Cheonggua Co., Ltd.

Abstract

LLM coding benchmarks face a credibility crisis: widespread solution leakage and test quality issues undermine SWE-bench Verified, while existing detection methods--paraphrase consistency, n-gram overlap, perplexity analysis--never directly observe whether a model reasons or recalls. Meanwhile, simply repeating verification degrades accuracy: multi-turn review generates false positives faster than it discovers true errors, suggesting that structural approaches are needed. We introduce Cross-Context Verification (CCV), a black-box method that solves the same benchmark problem in N independent sessions and measures solution diversity, combined with the Hierarchical Cross-Context Architecture (HCCA), a multi-agent analysis framework that prevents confirmation bias through intentional information restriction across specialized analytical roles. On 9 SWE-bench Verified problems (45 trials, Claude Opus 4.6, temperature 0), CCV achieves perfect separation between contaminated and genuine reasoning (Mann-Whitney U=0, p approx 0.012, r = 1.0). Key findings: (1) contamination is binary--models either recall perfectly or not at all; (2) reasoning absence is a perfect discriminator; (3) 33% of prior contamination labels are false positives; (4) HCCA's independent analysis structure discovers contamination-flaw composite cases that single-analyst approaches miss. A pilot experiment extending HCCA to multi-stage verification (Worker to Verifier to Director) yields a negative result--100% sycophantic confirmation--providing further evidence that information restriction, not structural complexity, is the key mechanism. We release all code and data.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility

    Aug 10, 2026Siyang Wu, Yibo Jiang, Bryon AragamPrompt SensitivityLLM Evaluation

  2. SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?

    Sep 18, 2026George Ma, Benjamin Mikek, Haoyu Li +9Formal VerificationAgentic Code Generation

  3. Automated Benchmark Auditing for AI Agents and Large Language Models

    May 25, 2026Junlin Wang, Federico Bianchi, Shang Zhu +4LLM EvaluationBenchmark Auditing