Why a single run always lies

Every week somebody shows me a screenshot. One prompt, one output, one number that looks great — and a sentence that says "it works." I've stopped arguing about whether the number is real. The problem is the one.

A single run can't distinguish skill from luck. If your procedure beat the baseline by 0.04 on one pass, you've learned almost nothing — because you have no idea whether the next run gives you 0.04 again, or −0.03. Point estimates from one draw are, by construction, consistent with "the effect is zero and you got lucky."

Noise has a floor, and most demos live below it

This is where statistical power enters. Power is the probability your test detects a real effect when one exists. Too few samples, and even a genuine improvement will fail to register — while noise will occasionally produce a "win" that vanishes on the next run. Underpowered evidence isn't weak evidence. It's evidence that can't tell you which story is true.

The GOLD standard fixes n=300 paired instances per slice before the run starts. Not "run it and see if it looks significant" — sized in advance, identical for candidate and baseline, so the comparison is fair and the power exists before anyone knows the answer.

Paired, not independent

The comparison is paired: every instance is scored with both procedures. That matters because the noise is shared. When both sides see the same hard instance, the shared difficulty cancels in the difference, and the signal-to-noise ratio of the gain goes up. Independent runs would ask each side to fight a different battle and then compare scores — which mostly compares which battle was harder.

What a single run looked like, honestly

Here's a real report from the forecasting domain — one judge() run, both slices, the sealed truth. Note what it says when the flat slice hasn't cleared the bar:

slice:trend  mean_gain=1.2396  lower_bound=1.0967  win=True
slice:flat   mean_gain=0.0163  lower_bound=-0.0276  win=False
placebo_clean=True   goodhart_ok=True   cost_parity=True

GOLD: False

The trend slice wins decisively. The flat slice has a positive mean — 0.0163, which a screenshot would happily show you — but the bootstrap lower bound is negative. The interval still touches zero. Luck is still on the table. That slice fails, and the whole gate fails with it. One point estimate said "winning." The interval said "not yet." The interval was right to be cautious.

The rule: the win is a lower confidence bound on the mean paired difference — never a point estimate. If the interval touches zero, the run hasn't proven anything.

Why five seeds, then a holdout

Even a well-powered single run has a residual risk: you picked one sealed seed, and while the seed was fair, the procedure was tuned while that seed's neighborhood was in view. So GOLD requires the gate to pass on at least five independent fresh seeds, and then one final run on a frozen holdout the procedure has never touched. A single GOLD True never counts. Five fresh passes plus the holdout — or the loop continues.

It sounds strict. It is. But the alternative is shipping a number that was true exactly once, on the day you happened to run it.