Benchmark Contamination: Why Boards Should Demand Overlap Audits

This guide explains why shared training data between pretraining corpora and benchmark test sets inflates scores and misleads boards.

It gives a reader rule: demand a contamination audit quantifying overlap, and treat any score above 5% overlap as unverified pending a held-out evaluation.

How Shared Data Inflates Scores

Every large language model begins with pretraining on web-scale datasets, and the scale is enormous. Llama-3, for example, was pretrained on 15 trillion tokens, a figure Unsloth cites in its continued pretraining documentation. Those corpora are scraped from the open web, which means public benchmark test sets — the very questions used to grade models — are frequently swept up alongside everything else.

This is the core mechanism of contamination: when a benchmark's test items appear in the pretraining corpus, the model can memorize the answers rather than learn the underlying skill. A high score then reflects recall of material it has already seen, not generalization to genuinely new problems. The distinction matters because boards and procurement teams treat benchmark numbers as evidence of capability, when they may only be evidence of exposure.

To make this auditable, define contamination precisely: the percentage of test items that were seen during training. If a benchmark reports a 60% overlap, that means 60% of its questions were present in the training data — and the reported score is inflated by an unknown margin, because there is no way to separate memorized answers from genuine reasoning within that score. Any overlap above zero degrades the score's meaning; the larger the overlap, the less the number tells you about how the model will perform on problems it has never encountered.

The practical check for a board reviewing a pilot proposal is straightforward. Ask the vendor for the training corpus provenance for each model under consideration, and ask whether the benchmark test sets in question are known to circulate on the public web. If the vendor cannot document that its pretraining data excluded or deduplicated against a given benchmark, treat that benchmark's score as unverified for that model.

When overlap figures are unavailable, the fallback is a held-out evaluation: test the model on questions you can confirm were not in any published corpus before the model's training cutoff. This is the only evaluation that is immune to contamination by construction, because you control the exposure. It costs more effort than reading a leaderboard, but it measures generalization directly instead of inferring it from a number that memorization may have inflated.

The rule to carry forward: before accepting any benchmark score, demand a contamination audit that quantifies training-test overlap. If the reported overlap exceeds 5%, treat the score as unverified and require a held-out evaluation before the score is used in any approval decision.

Evidence: Overlap and Score Gaps

The most-cited efficiency numbers in continued pretraining come from Unsloth's benchmarks, and they are worth reading carefully because they measure speed and memory, not contamination. Unsloth reports that training on all linear layers, including the embedding and output head, with a rank of 256 on an L4 GPU, delivers 2x speed and a 52% VRAM reduction for Llama-3 8B relative to Hugging Face with Flash Attention 2 QLoRA. For Llama-3 70B on an 80GB card, the same configuration yields 1.9x speed and a 49% VRAM reduction. Those figures tell you what hardware you need to run continued pretraining; they say nothing about whether the benchmark you plan to cite afterward shares items with the data you trained on. A board reviewing a pilot proposal should treat them as capacity planning inputs, not as evidence of generalization.

The cost side is equally instructive. The tokenmonster pretraining study reports that pretraining all 16 models took a cumulative 147 days on 8x GPUs, which works out to 1,176 GPU days, at a cost of $8,000. Divide 1,176 by 8 and you get 147, confirming the arithmetic. That total is small enough that a motivated team can assemble a pretraining-scale corpus without a major grant, which means the barrier to accidentally or deliberately including benchmark-adjacent text is low. When a vendor cites a strong score, ask what corpus was used and whether any evaluation set was filtered out of it.

Here is the gap that matters for approval decisions: none of the sources above reports a contamination rate. Unsloth's benchmarks are training efficiency metrics. The tokenmonster study is a cost and duration accounting. Neither publishes the percentage of test items that appeared in training data, and no source in this grounding supplies that overlap figure for any model. That absence is the finding. A score presented without an overlap percentage is an unverified score, regardless of how impressive the speed or cost numbers look beside it.

Use a simple intake table when a multi-model pilot comes to the board. For each model, require the vendor to state the training corpus cutoff, the benchmark version evaluated, and the measured overlap percentage between the two. If the overlap exceeds 5%, mark the score unverified and require a held-out evaluation on data the vendor confirms was excluded from training. If the vendor cannot produce an overlap figure at all, treat that as a failure to report rather than a clean result.

The practical rule for the board is narrow and enforceable: no overlap percentage, no approval. Efficiency benchmarks like Unsloth's and cost studies like tokenmonster's are useful for scoping a pilot, but they cannot substitute for a contamination audit, and a score that arrives without one should be logged as unverified until a held-out evaluation replaces it.

Options for Contamination Detection

Once you accept that a benchmark score is only as good as its separation from training data, the practical question becomes: how do you actually detect contamination? Three methods dominate, and they differ sharply in speed, sensitivity, and defensibility. Boards evaluating multi-model pilots should understand what each one can and cannot prove before signing off on an evaluation plan.

The first method is n-gram overlap. You split benchmark test items into short word sequences and check whether those exact sequences appear in the training corpus. It is fast, cheap, and runs on commodity hardware, which makes it a reasonable first-pass screen across large corpora. Its weakness is equally clear: it only catches verbatim reuse. A test question paraphrased with different wording, reordered clauses, or translated into a near-equivalent phrasing will slip past an n-gram check entirely. Treat it as an exact-match detector, not a contamination audit.

The second method is embedding similarity. Instead of matching literal strings, you encode both test items and candidate training passages into vector space and flag pairs whose semantic distance falls below a tuned threshold. This catches paraphrased and semantically equivalent leakage that n-gram screening misses. The trade-offs are real, though: you need a reference corpus to compare against, you must tune the similarity threshold, and a threshold set too aggressively will flag legitimate topical overlap as contamination, producing false positives that inflate your overlap estimate. Embedding similarity is a strong second pass, but its findings depend on choices a reviewer can contest.

The third method is held-out evaluation, and it is the only one that produces a definitive answer. You reserve a private test set that was never published, never scraped, and never available during any model's training window, then measure performance on it. Because no training process could have seen the items, contamination is ruled out by construction rather than inferred from statistical signals. The constraint is access: you must control a test set that no vendor has touched, which is exactly why this method belongs in a board-level audit requirement rather than a vendor self-attestation.

The practical rule for a governance review: use n-gram and embedding screening as cheap triage to estimate overlap, but treat any reported score as unverified until a held-out evaluation confirms it. A vendor who can only offer the first two methods has shown you a screening result, not an audit. Ask the vendor which method produced its overlap figure and on what corpus; if the answer is screening only, require the held-out run before the score is used in any approval decision.

Costs and Numbers That Matter

The price tag on trustworthy numbers is the first thing a board should see. A rigorous leaderboard run is expensive, which is why independent validation of headline scores is rare: most vendors never re-run their own results under neutral conditions, and few buyers can afford to do it for them. Treat any benchmark figure presented without a third-party replication as a marketing claim, not evidence of capability. Ask the vendor to state who ran the evaluation and under what conditions; if the answer is the vendor itself, the score is unverified.

Contamination, meanwhile, is cheap to introduce. The tokenmonster pretraining study reports that pretraining all 16 of its models took a cumulative 147 days on 8x GPUs — 1,176 GPU days — and cost $8,000 in compute. That is the entire budget of a mid-level pilot, spent on a phase where web-scale corpora flow directly into model weights. If a team can produce 16 pretrained models for $8,000, a board should assume that uncontrolled data pipelines, not careful curation, are the default — and that any benchmark score from such a pipeline is unverified until an audit says otherwise.

The barrier to continued pretraining keeps dropping, which widens the exposure. Unsloth's continued pretraining method trains 2x faster and cuts VRAM use by 52% on Llama-3 8B, meaning a team can steer a base model toward a new domain on a single commodity GPU. Every such run is another opportunity for benchmark test material to slip into the training mix. When you evaluate a vendor's fine-tuned or continually pretrained model, ask specifically what corpus was used and whether any benchmark-derived text was filtered out; a vendor who cannot answer has effectively answered.

Put the two cost curves side by side and the governance gap is obvious. Producing a potentially contaminated model costs thousands of dollars; verifying it costs tens of thousands per leaderboard run. That asymmetry guarantees a supply of impressive-looking scores and a shortage of verified ones. Your procurement process should not pretend otherwise.

What the Evidence Does NOT Establish

Before this section goes further, it is worth being precise about what the evidence assembled so far does not establish. Nothing in the sources reviewed here provides a direct measurement of benchmark contamination for any specific model or benchmark. No source reports a specific overlap percentage — say, the share of test items a given model saw during pretraining — for any benchmark. Such figures do not exist in the literature we can cite; they must be measured per model and per benchmark, which is exactly why a contamination audit has to generate them rather than look them up.

The relationship between overlap and score inflation is also unquantified here. We can define contamination as the percentage of test items seen during training, but the available sources contains no data mapping that percentage to a score delta. The inflation from a given overlap presumably varies by task type, benchmark construction, and model, but none of the sources reviewed measures that variation. So while the mechanism — test items appearing in web-scale pretraining corpora, as DZone's overview of pretraining on web-scale datasets makes plausible — is sound, the magnitude is unknown. A board should treat any claimed inflation figure as unverified unless the audit that produced it is shown.

Mitigation is similarly unevidenced. Techniques such as data deduplication are commonly proposed, but the available sources offers no measurement of how effective any mitigation is at reducing overlap or restoring honest scores. A vendor that says "we deduplicated" has said nothing verifiable without before-and-after overlap measurements on the same benchmark.

Note what the existing evidence actually covers, because the gap is specific. Unsloth's continued pretraining benchmarks report speed and VRAM — 2x speed and 52% VRAM reduction for Llama-3 8B, 1.9x and 49% for Llama-3 70B — and tokenmonster's pretraining study reports cost and duration, 1,176 GPU days at $8,000 for 16 models. Those are efficiency and cost figures, not contamination figures. NVIDIA's MLPerf results measure training performance per GPU. None of these sources measures training-test overlap or adjusted scores, which is precisely the hole an audit must fill.

The practical rule for a board follows from the gap. Require any pilot proposal to include an audit that reports two numbers per benchmark: the measured overlap percentage and the score adjusted for it. If a proposal cannot produce those numbers, the benchmark scores it cites are unverified regardless of how impressive they look, and a held-out evaluation on data the model has demonstrably not seen is the only acceptable substitute.

Auditing a Model

Conducting a contamination audit is a bounded exercise, not an open-ended research project. The goal is a single deliverable: a report stating what percentage of each benchmark's test items appear in the model's training data, and what the benchmark scores look like once contaminated items are removed or replaced. Boards should treat that report as a gate before approving any multi-model pilot, because a score without an overlap figure is an unverifiable number.

Step one is obtaining the pretraining corpus, or a representative sample of it, directly from the model provider. Llama-3, for example, was pretrained on roughly 15 trillion tokens, per Unsloth's continued pretraining documentation, so a full corpus review is rarely practical — a stratified sample is acceptable. If the provider will not share the corpus or a sample, use a proxy such as Common Crawl, which underlies most web-scale pretraining mixes, and document that the audit is a proxy-based estimate rather than a direct measurement.

Step two is the overlap computation itself. For each benchmark item, compute n-gram overlap against the corpus — a 13-gram match is a common and defensible choice, since a sequence that long is unlikely to occur by chance. Flag any test item whose longest matching n-gram exceeds 80% of the item's length, meaning the item is substantially present in training data. Run this per benchmark, not across the suite as a whole, because contamination is highly uneven: a coding benchmark and a trivia benchmark drawn from the same web crawl can have very different exposure profiles.

Step three is the decision rule. If flagged items exceed 5% of a benchmark's test set, treat that benchmark's reported score as unverified and commission a held-out evaluation — fresh test items written or curated after the model's training cutoff, never published online. A rigorous leaderboard run is costly, covering item design, human review, and inference across the candidate models, but that cost is small relative to the cost of pretraining itself; the tokenmonster pretraining study reports the pretraining phase for 16 models took a cumulative 1,176 GPU days at a cost of $8,000, so a contamination audit is a rounding error against any pilot it protects.

The audit report the board receives should contain three numbers per benchmark: the percentage of test items flagged, the reported score, and the adjusted score on uncontaminated or held-out items. If a vendor resists providing corpus access or a sample, that refusal is itself a finding — require the held-out evaluation as the default path in that case, and do not let an unaudited benchmark score anchor the pilot's model selection.

Worked Example: Run the Numbers

Here is a worked example you can hand to a vendor or an internal team. Scenario: a mid-size insurer is evaluating two models for a claims-classification pilot. Vendor A reports 92% on a 400-item benchmark suite; Vendor B reports 90.5% on the same suite. The board asks for a contamination audit before approving a multi-model pilot. The item counts and scores below are constructed for arithmetic demonstration, not drawn from a published benchmark.

Step one: quantify overlap. The audit samples the 400 test items and matches 60 of them against Vendor A's disclosed training corpus, so contamination is 60 ÷ 400 = 15%. Under the 5% rule, that score is unverified, and a held-out evaluation is required. Vendor B's audit finds 8 matching items: 8 ÷ 400 = 2%, which passes the threshold, so its 90.5% stands as reported.

Step two: do not attempt to compute an adjusted score from the reported benchmark alone. Vendor A's 92% on 400 items means 0.92 × 400 = 368 correct answers, and the audit flagged 60 of those 400 items as contaminated. Simply removing the 60 flagged items yields 368 − 60 = 308 correct out of 400 − 60 = 340 clean items, or 308 ÷ 340 ≈ 90.6% — but that arithmetic assumes every flagged item was answered correctly, which the audit does not establish. The honest adjustment requires knowing how the model performs on items it demonstrably has not seen, so until a held-out run exists, the 92% is simply unverified, not adjusted.

Step three: run the held-out evaluation on a fresh, privately written item set and report both scores side by side. The comparison the board approves on is the held-out number, not the published benchmark. If the vendor refuses a held-out run, that refusal is itself the finding.

Step four: identify the break-even trigger. Suppose the held-out set shows Vendor A answering 280 of the 340 unseen items correctly, for a clean-item rate of 280 ÷ 340 ≈ 82.4%, versus Vendor B's 90.5% — Vendor B wins this example, and the gap came entirely from items Vendor A had seen. The trigger to watch: if a vendor's clean-item rate falls below the competitor's reported score, the pilot flips, no matter how impressive the headline benchmark looked.

The general rule this example encodes: report overlap percentage, report the held-out score, and declare the winner on held-out numbers only. Any vendor summary that gives you a benchmark percentage without an overlap figure has given you half an audit.

Decision Rules for Governance

Governance works when it converts a principle into rules a committee can apply without debate. The canonical rule is simple: before accepting any benchmark score, demand a contamination audit that quantifies training-test overlap. The if/then rules below operationalize that rule for boards evaluating multi-model pilots, and they are the only decision rules in this piece.

Rule one: if a vendor reports a benchmark score without a contamination audit, then reject the score outright and require an independent audit. A score without an audit is not a weak claim; it is an unverified claim. Boards should treat the absence of an overlap report the same way they treat an unaudited financial statement — as something that cannot be relied upon until someone independent checks it. The audit request should be written into the vendor contract before any pilot discussion begins, not after.

Rule two: if the audit shows overlap exceeding 5%, then demand a held-out evaluation before approving the pilot. The 5% line is the board's tripwire: below it, the reported score stands with a disclosed caveat; above it, the score is unverified and only a fresh evaluation on data the model has provably never seen can restore trust. The held-out set should be assembled by the buyer, not the vendor, and drawn from the buyer's own operational domain wherever possible.

Rule three: if a held-out evaluation is not feasible — because labeled data is scarce, timelines are tight, or the domain is too narrow to build a clean test set — then cap the model's claimed performance at the level of a random baseline for the contaminated portion of the benchmark. This is the conservative floor. It does not accuse the vendor of anything; it simply says that any portion of a score that could reflect memorized test items earns zero credit until proven otherwise. A vendor confident in genuine generalization will accept this cap and work to beat it; a vendor who resists it has told the board something useful.

These three rules chain together, and the chain matters more than any single link. A board that accepts unaudited scores has no trigger for rule two; a board that waives held-out evaluations under time pressure has no floor under rule three. Apply all three in sequence, and the multi-model pilot decision rests on scores the board can actually defend — to auditors, to regulators, and to itself.

What to do next

Frequently Asked Questions

What exactly should a contamination audit deliver before we approve a multi-model pilot?

Boards should require contamination audits reporting overlap percentages and adjusted scores before approving multi-model pilots.

If a vendor reports high benchmark scores but no audit, how should we treat that score?

Treat any benchmark score as unverified when training-test overlap exceeds the stated threshold, and require a held-out evaluation before trusting it.

Where does contamination actually enter the model?

Contamination enters during pretraining, the language modeling phase where a model learns from web-scale datasets scraped from the open web.

Why does web-scale pretraining make benchmark contamination likely in the first place?

Because pretraining corpora are scraped from the open web, public benchmark test sets end up in the training data, which inflates scores and overstates generalization.

Can you give a concrete example of the scale of pretraining data involved?

Llama-3 was pretrained on 15 trillion tokens, a figure Unsloth cites in its continued pretraining documentation.

What is the core risk to a board that relies on benchmark scores without an audit?

Benchmark scores inflated by shared training data overstate model generalization, which is why the guide treats heavy overlap as misleading to boards.

Quick answers

Also worth reading: Why the next phase of enterprise AI requires a complete rethink of data strategy: Why the next phase of · Excel to slides reporting: 19 of 68 pilots passed Deloitte 2026 benchmark: Excel to slides reporting: 19 · Driving superior enterprise AI performance with optimization algorithms: Driving superior enterprise AI performance