This guide explains why baseline choice acts as a hidden regression gate in multi-model AI pilots and how un-re-baselined swaps distort measured failure rates.
It sets out the 2026 rule: freeze a baseline suite, re-run it on every candidate model, and block any swap whose failure-rate delta exceeds the pre-registered gate.
How baseline choice changes failure rates
A pilot baseline is not the model you happen to be running today. It is three separate artifacts bundled together: the frozen input distribution, the scoring harness, and the reference model version. Change any one of those three and the measured failure rate moves, even when the deployed model is byte-for-byte identical. That is the trap. Teams treat "baseline" as a synonym for "the model we started with," then swap the model and keep reporting the old failure rate as if the denominator never moved. It did. The baseline is the reference distribution and the scoring harness, not the model — and swapping models without re-baselining silently redefines the denominator underneath every number you report. To check whether your own baseline is actually frozen, confirm you can emit the identical input distribution and scoring harness from stored artifacts rather than from live traffic; if you cannot, you do not have a baseline, you have a trend line.
Regression gates only work by comparison. A candidate model's failure rate is measured against the baseline's failure rate on the same frozen suite, and the gate fires when the delta crosses the pre-registered threshold. That comparison is valid only if the baseline is re-run at swap time, on the same suite, under the same harness. The method is unglamorous: before the swap, pre-register the suite hash and the baseline run ID; after the swap, re-run the frozen suite on the new model and diff the two runs. If the suite hash does not match, you are not comparing anything — you are comparing two different tests and calling the difference a regression. The check before you trust any delta: confirm the suite hash and harness version in the run record match the pre-registered values; if either is missing, the number is not comparable to anything.
Scope selection compounds the problem. ScreenshotNeo's overview of regression testing types lays out the two scope-selection approaches — retest-all over the whole relevant suite, and selective regression over a risk-weighted subset — and the choice between them changes what the baseline even measures. A selective suite that was tuned to the old model's weak spots will under-sample the new model's failure modes, so the delta you compute is an artifact of suite composition rather than model quality. Freeze the suite first, then swap. If you must change scope, version the suite and re-baseline both models on the new scope before you compare. The check before you commit to a selective gate: confirm the risk-based selection criteria are pre-registered in writing, because an unjustified exclusion is just an unmeasured regression.
The practical rule for 2026 pilots: lock the baseline, and re-run it on every swap. Pre-register the gate before you see the candidate's numbers — a delta threshold expressed in percentage points on the frozen suite — and block the swap when the candidate exceeds it. The threshold itself belongs to your pre-registration, not to this section; what matters here is that the gate is meaningless without a baseline run executed at the same moment as the candidate run. The check before you trust any failure-rate figure: ask for the frozen input distribution and scoring harness version; if either is missing, treat the number as a local measurement, not a comparable statistic.
| Baseline artifact | What changes if you swap it | Check before the swap |
|---|---|---|
| Frozen input distribution | Denominator of every failure rate | Suite hash matches the pre-registered hash |
| Scoring harness | What counts as a failure at all | Harness version pinned in the run record |
| Reference model version | The comparison arm of the delta | Baseline run ID re-executed at swap time |
None of this requires new tooling. It requires refusing to treat the baseline as a historical fact. Re-run it, diff it, and let the gate decide. The check before you adopt any baseline strategy: confirm you can emit the identical input distribution and scoring harness from stored artifacts rather than from live traffic — that is the precondition for a delta you can defend.
The evidence: what regression gates catch
The uncomfortable finding from the sources behind this piece: none of them reports an AI pilot failure-rate figure. Bug0.com discusses regression test cost in terms of fully loaded engineer time to write and maintain tests when the UI changes. ScreenshotNeo.com catalogs regression testing types and scope-selection approaches without publishing failure-rate benchmarks. Ornstein, Hays and Franzese (2022, Economics and Politics 34(3):429–443, DOI 10.1111/ecpo.12204) is a regression-discontinuity study of left-government interest premiums on government bonds — econometrics, not software QA. Hoerl and Kennard (1970) introduced ridge regression as biased estimation for multicollinear data. So when someone tells you a model swap "only moved failure rates by a point or two," ask where that number came from. The check: request the frozen input distribution and scoring harness version behind the figure; if either is missing, the number is a local measurement from someone's specific harness, not an industry statistic.
What the sources do establish is the economics of the gate itself. Bug0.com's core point is that regression test cost is dominated not by tool licenses but by fully loaded engineer time — writing tests, then maintaining them every time the UI changes. That has a direct implication for pilot design: a gate that re-runs the entire suite on every model swap is economically self-defeating, because you are paying the full maintenance burden of retest-all on a cadence (every swap) that multiplies it. The check before you commit to a gate: estimate the engineer-hours per full re-run, multiply by your expected swap frequency over the pilot, and see whether the gate costs more than the regressions it catches.
ScreenshotNeo.com names the two canonical scope-selection approaches: retest-all (the whole relevant suite) and selective regression (a risk-based subset). Your gate design must declare, in writing, which one it uses — before the first swap, not after the first dispute. A selective gate is cheaper but only defensible if the risk-based selection criteria are pre-registered; otherwise "we only ran the tests we thought were affected" becomes a post-hoc excuse for a missed regression.
The econometrics literature adds a second, subtler warning. Ornstein, Hays and Franzese's RDD study is valuable here precisely because it shows how sensitive a measured effect is to specification choices — their partisan effects appear only under specific conditions and vanish elsewhere. Hoerl and Kennard's ridge regression exists for the same underlying reason: naive estimation on poorly conditioned data produces unstable, misleading coefficients. A pilot failure-rate delta is a statistic, and statistics computed on a changed reference distribution are not comparable to the baseline. That is the mechanism the gate exists to police.
The practical rule that falls out: before any model swap, re-run the frozen baseline suite on the new model, and block the swap if the failure-rate delta exceeds your pre-registered gate. Declare retest-all versus selective up front, cost the re-run in engineer-hours per bug0.com's framing, and treat any failure-rate figure that lacks a frozen-baseline re-run behind it as unmeasured — because no source in the available sources set gives you one to lean on.
Options compared: baseline strategies
The comparison below asks one question of every option: can you re-run it, unchanged, on a new model? A baseline that cannot be replayed is a historical statistic, not a gate. Before adopting any strategy, confirm you can emit the identical input distribution and scoring harness from stored artifacts rather than from live traffic — that is the precondition for a delta you can defend.
Option A, no baseline, compares the new model against last week's dashboard. It is the cheapest option because it needs no fixtures and no maintenance. It also confounds the model swap with data drift, which fails the gate's validity condition: the reference moves at the same time the candidate does, so a rising failure rate cannot be attributed to either. Check before trusting it: if you cannot reproduce last week's failure rate from stored inputs, you do not have a baseline — you have a trend line.
Option B, retest-all, runs the whole relevant suite. Fidelity is highest, and so is fully loaded cost. Bug0.com frames regression test cost as engineer time to write tests and maintain them when the interface changes, not as license spend — a framing that makes retest-all viable only for small suites, and increasingly painful as the suite grows. Check: if a full pass takes longer than the interval between your model swaps, retest-all is a backlog item, not a gate.
Option C, selective regression, runs the risk-selected slice, following the scope-selection approach screenshotneo.com describes. It wins on cost and speed when the swap touches a bounded intent slice, because the slice is where the risk sits. The failure mode is leakage: if you re-choose the slice after seeing the new model's failures, you are scoring on the test set. Freeze the selection at pre-registration.
| Strategy | What it compares | Failure mode | Use when |
|---|---|---|---|
| A — No baseline (last week's dashboard) | New model vs. prior live traffic | Swap confounded with data drift; gate invalid | Not viable as a gate; smoke check only |
| B — Retest-all | New model vs. frozen full suite | Fully loaded maintenance cost and runtime | Suite re-runs inside the swap window |
| C — Selective regression | New model vs. frozen risk-selected slice | Slice re-chosen after results | Swap touches a bounded intent slice |
| Frozen re-run baseline (winner) | New model vs. frozen slice and harness, re-run each swap | Requires artifact discipline | Default for every gated swap |
The winner is the frozen re-run baseline: option C's scoping discipline plus option B's reproducibility. For 2026 pilots, lock the slice and harness before the swap, pre-register the gate (for example, block the swap when the failure-rate delta exceeds 0.5 percentage points), then re-run the identical suite on the new model and compare the two numbers. If the suite cannot be emitted from stored inputs, the delta is uninterpretable and the swap should wait.
Costs and numbers that matter
The cost basis for a baseline re-run is fully loaded engineer time, not tool licensing. Using the cost accounting method described at bug0.com, the per-test figure is (engineer hours to write the test + engineer hours to maintain it per interface change) × the loaded hourly rate for that engineer. Maintenance enters as a per-change multiplier rather than a one-time charge, so a test over a frequently changing interface carries more cost than one over a stable schema. If your 2026 pilot budget sheet lists only the vendor subscription, you do not have a cost model, you have an invoice. Rebuild the ledger before anyone argues about whether re-baselining is affordable.
Multiply that per-test figure by the number of tests in the frozen slice to get the baseline re-run cost: (suite size on the affected slice) × (per-test fully loaded cost). The affected slice, not the whole suite, is the multiplier — a swap that cannot reach an interface does not obligate you to re-run tests over it. That product is the number that belongs in the swap request. Report it as a range keyed to your write-time and maintenance-hour assumptions, and state those assumptions so a reviewer can check the arithmetic rather than trust it.
The gate threshold is a pre-registered method parameter, not a finding. Write it into the pilot plan before the first candidate runs — for example, block the swap if the candidate's failure rate on the frozen slice exceeds the baseline by more than 0.5 percentage points. That number is a local policy choice made by your team; it is not a sourced statistic and should not be defended as one. Defend it against what a shipped regression costs you, and pre-register it so the threshold cannot be renegotiated after the delta is visible.
Compute the decision as a subtraction: candidate failure rate on the frozen slice minus baseline failure rate on that same slice. Block the swap when that delta exceeds the pre-registered gate. Then set the two costs side by side — the regression you would ship by skipping the re-run versus the re-run cost from the formula above — and record both in the swap log so the next swap inherits the ledger instead of rebuilding it.
| Ledger line | How to compute |
|---|---|
| Per-test fully loaded cost | (write hours + maintenance hours per interface change) × loaded hourly rate |
| Baseline re-run cost | tests on the affected slice × per-test fully loaded cost |
| Gate threshold | Pre-registered local policy, e.g., block if candidate exceeds baseline by more than 0.5 percentage points |
| Swap decision | candidate rate − baseline rate on the frozen slice; block if delta > gate |
Two checks keep the numbers honest. Hold the slice fixed across baseline and candidate runs, and recompute per-test cost only when interface change frequency shifts. When a swap is blocked, record whether the failure sat in the model output or in the scoring harness; a harness fix changes the baseline number and therefore invalidates the prior gate comparison.
What the evidence does NOT establish
Before you quote a failure rate from any 2026 pilot, ask where the number came from. None of the sources behind this piece — bug0.com, screenshotneo.com, the regression-testing design guides, the economics papers — reports an AI pilot failure-rate figure. Bug0.com discusses regression test cost in terms of fully loaded engineer time; screenshotneo.com catalogs scope-selection approaches like retest-all versus selective regression. That means any "X% of pilots fail" claim you encounter is a local measurement from someone's specific harness, input distribution, and model version. Treat it as such: label it, name the harness, and never cite it as an industry statistic.
The same discipline applies to the academic citations. Ornstein, Hays, and Franzese (2022) is a regression-discontinuity study of left government and bond-yield premia in Economics and Politics — political economy, not AI evaluation. Its heterogeneity finding is genuinely useful as a principle: effects were statistically discernible only under sharp governing alternatives, in certain eras (around the 1950s–1970s), and over a short horizon of about one year. But that is a lesson about conditional effects, not a numeric estimate you can port into a model-swap gate. If someone cites it as justification for a specific percentage-point threshold in an AI pilot, the citation does not support that use.
Honestly, Hoerl and Kennard (1970) has the same problem. It is a statistics methods paper introducing ridge regression — biased estimation for improved prediction stability, published in Technometrics. What transfers is the principle: prefer a stable estimator over an unstable one even at some cost in raw fit. That is a defensible argument for freezing a baseline and re-running it on every model swap. What does not transfer is any figure. The paper contains no AI pilot data, no failure-rate deltas, no gate thresholds.
So what can you actually check? Three things. First, when a vendor or internal team reports a pilot failure rate, ask for the frozen input distribution and scoring harness version — if either is missing, the number is not comparable to anything. Second, when a study is cited to justify a regression gate, verify the domain: an RDD result about government bonds or a Heckman selection table about audit committees (as in the ResearchGate earnings-management paper behind this piece) establishes methodology, not AI failure rates. Third, when a threshold like a half-percentage-point gate is proposed, treat it as a pre-registered local choice — defensible, but an engineering decision your team owns, not an empirical finding from the literature.
The practical rule for 2026 pilots: separate what the evidence establishes from what it does not. The sources establish that regression testing has real maintenance costs, that scope selection matters, and that effect estimates can be highly conditional. They do not establish any universal failure-rate figure or any universal gate. Your baseline lock and your delta threshold are commitments you make and document yourself — and the honesty of that labeling is what makes the gate credible when a model swap is on the table.
A 3-slice swap gate
A swap gate only works if you can price it before you run it. Take an illustrative pilot with three intent slices: billing (120 tests), auth (80 tests), and search (200 tests) — 400 tests total. The inputs are a loaded engineer rate of $95/hr, 0.25 hr to write each test, and 0.1 hr to maintain each test per interface change. Those two time figures come from the bug0.com cost basis for regression tests, which prices tests in fully loaded engineer time rather than tool licenses.
Checkpoint 1 is cost per test. Writing: 0.25 hr × $95 = $23.75. Maintaining per interface change: 0.1 hr × $95 = $9.50. Fully loaded, each test costs $33.25 per change. Multiply that by your slice count and you know what any swap will cost you in gate time before you approve it.
Checkpoint 2 is selective execution. Suppose the swap only touches billing behavior. The gate runs the billing slice twice — once as the frozen baseline, once on the candidate — so the cost is 120 × $33.25 × 2 = $7,980. Retest-all, by contrast, would run the full 400-test suite on both sides: 400 × $33.25 × 2 = $26,600. Selective slicing saves $18,620 on this swap, which is why slice selection belongs in the gate worksheet, not in an afterthought.
| Scope | Tests per run | Runs (baseline + candidate) | Cost |
|---|---|---|---|
| Billing slice only | 120 | 2 | $7,980 |
| Retest-all | 400 | 2 | $26,600 |
Checkpoint 3 is the decision rule itself. Copy this worksheet into your pilot doc: (1) list the slices and test counts; (2) mark which slices the swap can plausibly affect — if you cannot argue a slice is untouched, include it; (3) compute gate cost as affected tests × $33.25 × 2; (4) pre-register the failure-rate delta that blocks the swap — the reader rule in this piece uses 0.5 percentage points, but the number must be fixed before the first swap, not negotiated after; (5) run baseline and candidate on the same frozen inputs and harness; (6) block the swap if the delta exceeds the gate, and log the run either way.
Two cautions keep the gate honest. First, selective regression is a recognized scope-selection approach — screenshotneo.com contrasts retest-all with selective, risk-based selection — but "risk-based" means you justify the slice boundary in writing, because an unjustified exclusion is just an unmeasured regression. Second, the gate measures deltas, not absolutes: if the baseline run itself drifts between executions, your harness is unstable and no delta is interpretable until you fix it.
Worked Example: Run the Numbers
The numbers below are a worked illustration, not a measured result — reproduce the arithmetic rather than citing it. Scenario: a 40-agent service desk piloting a triage classifier. Decision party: the pilot's selection group, made up of the service-desk business sponsor and the platform lead, meeting April 3, 2026. The scoring suite was frozen on January 6, 2026 and covers the January 6–March 27, 2026 window: 1,600 labeled cases. Pre-registered gate: block a swap if the candidate's failure-rate delta exceeds 0.5 percentage points.
| Run (illustration) | Failures of 1,600 | Failure rate | Delta vs. A1 | Decision |
|---|---|---|---|---|
| A1, incumbent first run | 64 | 4.00% | — | Reference |
| A2, incumbent repeat run | 67 | 4.1875% | +0.1875 pp | Sets the run-to-run floor |
| B, candidate one | 71 | 4.4375% | +0.4375 pp | Pass |
| C, candidate two | 79 | 4.9375% | +0.9375 pp | Block |
| Gate (0.5 pp) | 8 cases | — | Last passing count: 72; first blocking count: 73 | — |
Step one: convert the gate into cases. 0.5% of 1,600 is 8, so a candidate may fail at most 8 cases more than the incumbent's first run. Step two: run the incumbent twice. A1 fails 64 cases (4.00%) and the repeat A2 fails 67 (4.1875%), a spread of 0.1875 percentage points. That spread is the floor for this suite — a candidate delta smaller than the incumbent's own re-run spread is not a signal. Step three: score the candidates on the same frozen suite. B fails 71 cases (4.4375%), seven cases above A1 and one case inside the eight-case gate. C fails 79 (4.9375%), fifteen cases above A1 and seven cases past the gate.
Winner for this example: B. C is blocked at the gate, so it never reaches a head-to-head comparison, however good its other metrics look. B's break-even trigger is 73 failures: any later re-run of the frozen suite that lands B at 72 failures (4.50%, exactly +0.50 points) still passes, while 73 (4.5625%, +0.5625 points) exceeds the gate and reopens the swap decision.
Two rules carry over. First, re-run the incumbent in the same session as the candidate, never against a stored pass; the incumbent's re-run spread is the only honest floor you have. Second, size the suite to the gate. At 1,600 cases one flipped case is 0.0625 points, so the 0.5-point gate spans eight cases. At 200 cases one flipped case is 0.5 points, and a single case decides the swap. If your suite cannot span the gate, fix the suite before arguing about the model.
Decision rules for 2026 pilots
Rule one: if a model swap touches more than one intent slice, then re-run the baseline on each touched slice separately. Never average across slices. A pooled delta can hide a two-point regression on one slice behind a half-point improvement on another, and the average will pass the gate while the slice your customers actually hit fails. The check is mechanical: list every slice the swap's prompt or routing changes touch, run the frozen suite per slice, and record one delta per slice — no composite numbers in the swap log.
Rule two: if the candidate's failure-rate delta on any frozen slice exceeds the pre-registered gate — for example, a gate set at 0.5 percentage points — then block the swap and re-baseline before retrying. Blocking is not a judgment call made in the moment; it is the gate you registered when the pilot started, applied without exception. After a block, the sequence matters: re-baseline first, then re-evaluate the candidate against the new baseline. Retrying against the old baseline after a block just re-measures the same mismatch.
Rule three: if the frozen input distribution no longer matches production traffic, then re-freeze the baseline before interpreting any delta. A baseline suite that was representative at pilot start can drift out of alignment as traffic shifts, and a delta measured against a stale distribution tells you about the stale distribution, not about your users. The check is a distribution comparison — slice mix, input lengths, edge-case frequency — run before every swap decision, not after a failed one.
Rule four: if the swap changes the scoring harness as well as the model, then treat it as a re-baseline event, not a swap. The baseline is the reference distribution, the harness, and the reference version together; changing two of the three at once makes the delta uninterpretable, because you cannot attribute the movement to the model. Sequence harness changes separately from model changes, with a clean baseline run in between.
One operational note on cost, drawn from the regression-testing literature: bug0.com frames selective regression testing — running only the tests affected by the change — as the way to control the fully loaded engineer time of writing and maintaining suites when the system changes. Apply the same selectivity here: re-run the frozen baseline on touched slices, not the entire suite, but never let selectivity become averaging. The gate is per slice, or it is not a gate.
What to do next
| Step | Action | Why it matters |
|---|---|---|
| 1 | Lock the frozen baseline suite before the multi-model pilot begins, and record the incumbent model's failure rate from that run as the reference number for the pilot. | Baseline choice is a regression gate, not a setup detail. Without a frozen reference run, every later failure-rate delta is unmeasured and no swap can be judged. |
| 2 | Write the pre-registered failure-rate gate into the pilot plan before any candidate model is loaded, using the gate figure given above. | A gate set after the swap is post-hoc rationalization. Setting it before the pilot starts is what makes the block defensible to whoever owns the rollout. |
| 3 | Re-run the frozen baseline suite on each candidate model — the same suite, unchanged — rather than relying on that candidate's own tests. | A candidate's own tests change the denominator and answer a different question. Only the frozen suite produces a delta comparable to the original baseline run. |
| 4 | Compute the failure-rate delta between the candidate's baseline-suite result and the original baseline run, then compare it against the pre-registered gate. | This comparison is the entire decision. Treating baseline choice as a setup detail hides that swapping a model without re-baselining can move measured failure rates by more than the model itself. |
| 5 | Block the swap if the delta exceeds the gate, and log the run alongside the gate value so the block is auditable. | The gate only works if exceeding it stops the swap. A blocked swap with a logged run is the evidence that the pilot caught a regression rather than absorbing it. |
| 6 | If the delta is within the gate, approve the swap only after the frozen suite run is on file for the new model; treat any swap without that run as unmeasured. | An unmeasured swap cannot be distinguished from a regression later. Keeping the frozen suite as the fixed reference makes the next model swap a repeat of the same gate, not a fresh guess. |
Frequently Asked Questions
What three artifacts make up a pilot baseline besides the model you happen to be running today?
A pilot baseline is three separate artifacts bundled together: the frozen input distribution, the scoring harness, and the reference model version.
Can the measured failure rate move even if the deployed model is byte-for-byte identical?
Yes, changing any one of those three moves the measured failure rate even when the deployed model is byte-for-byte identical.
How can I tell whether my own baseline is actually frozen?
Confirm you can emit the identical input distribution and scoring harness from stored artifacts rather than from live traffic; if you cannot, you do not have a baseline, you have a trend line.
What does the 2026 rule require before a model swap is allowed?
Freeze a baseline suite, re-run it on every candidate model, and block any swap whose failure-rate delta exceeds the pre-registered gate.
Why do regression gates break when teams swap the model but keep reporting the old failure rate?
Because swapping models without re-baselining silently redefines the denominator underneath every number you report.
What common misconception do teams have about what a baseline is?
Teams treat "baseline" as a synonym for "the model we started with," then swap the model and keep reporting the old failure rate as if the denominator never moved.
Quick answers
| What three separate artifacts bundled together make up a pilot baseline? | It is three separate artifacts bundled together: the frozen input distribution, the scoring harness, and the reference model version. |
| What happens to the measured failure rate when any one of the three baseline artifacts is changed? | Change any one of those three and the measured failure rate moves, even when the deployed model is byte-for-byte identical. |
| What does swapping models without re-baselining silently redefine underneath every number you report? | Swapping models without re-baselining silently redefines the denominator underneath every number you report. |
| How can you confirm whether your own baseline is actually frozen? | To check whether your own baseline is actually frozen, confirm you can emit the identical input distribution and scoring harness from stored artifacts rather than from live traffic. |
| What is the 2026 rule for handling a model swap? | It sets out the 2026 rule: freeze a baseline suite, re-run it on every candidate model, and block any swap whose failure-rate delta exceeds the pre-registered gate. |
Also worth reading: Excel to slides reporting: 19 of 68 pilots passed Deloitte 2026 benchmark: Excel to slides reporting: 19 · AI Code Generation in PHP and Python: Capabilities and Realities: AI Code Generation in PHP · Practical Logistic Regression: Plotting and Interpreting Results in PHP and Python: Practical Logistic Regression: Plotting and