The verification standard for AI work. A procedure only earns GOLD when its gains are proven by a signal it can't game — and it still beats a strong, fair baseline. No mocked runs. Real outcomes only.
Free · Open standard · Runs anywhere Python runs
"It works" is the most expensive sentence in software. Three failure modes the standard kills on contact:
One cherry-picked run, one lucky seed, declared victory. Single runs prove nothing — variance hides in the noise floor.
Optimize the scoreboard, not the outcome. Goodhart's law ships to production and the real metric never moves.
"We beat the naive approach." Against a weak baseline, anything looks like genius. A fair fight needs a tuned opponent.
The judge() runs every candidate through all six — automatically, before anything is allowed to be called done.
Both procedures are fingerprinted and committed before the sealed test seed is drawn. No post-hoc tuning, no moving goalposts.
Candidate and baseline see identical instances with identical compute budgets. random.Random(1) for both. No resource cheating.
Scored against mismatched hidden truths. If the gain survives a placebo, it's just the prior — not skill.
Bootstrap lower confidence bound on the paired difference must clear zero. Bonferroni-corrected across slices. Luck doesn't pass.
If the optimizable proxy improves faster than the sealed truth signal, the procedure is gaming its own scoreboard. Flagged, not shipped.
Candidate runtime is held against a slack multiple of the baseline. Victory by brute force is a cost failure, not a win.
From a live benchmark of an iterative refinement loop against a tuned single-call baseline — 30 prompt × spec pairs, every outcome an actual API response scored by a frozen rubric.
Every gate in GOLD is a named, published pattern from statistics and AI evaluation. The standard's job is not novelty — it's making sure none of them get skipped under deadline pressure. Sources are linked; follow them.
When a measure becomes a target, it ceases to be a good measure. GOLD checks that gain on the optimizable proxy never outruns gain on the sealed truth signal.
wikipedia.org/wiki/Goodhart's_lawDeepMind's catalogue of agents achieving the letter of the objective while violating its spirit. The placebo gate is designed to catch exactly this: gain against a mismatched truth is prior, not skill.
deepmind.google — Krakovna et al.Proving that black-box models have memorized their evaluation sets. GOLD's answer: five fresh sealed seeds plus one frozen holdout the procedure never touched — a single lucky pass never counts.
arxiv.org/abs/2310.17623The win is a lower confidence bound on the mean paired difference — not a point estimate. If the interval touches zero, luck is still on the table, and the gate fails.
wikipedia.org/wiki/BootstrappingEvery extra slice tested multiplies the chance of a false win. α is divided across slices so "it won somewhere" can't quietly become "it won."
wikipedia.org/wiki/Bonferroni_correctionThe winner's curse: run enough comparisons and noise wins one of them. Leaderboards built on many unreported comparisons are mostly reporting their luck.
wikipedia.org/wiki/Multiple_comparisons_problemSearching the data until something looks significant. The commit gate closes this structurally: procedures are fingerprinted before the sealed seed exists.
wikipedia.org/wiki/Data_dredgingToo few samples can't distinguish skill from noise, however real the effect. GOLD fixes n=300 paired instances per slice before the run — sized, not improvised.
wikipedia.org/wiki/Statistical_powerWinning on the data you tuned against and losing on everything else. The frozen holdout exists solely to catch a procedure that learned the test instead of the task.
wikipedia.org/wiki/OverfittingGOLD is a standard, not another harness. These are the real, verifiable projects that built the ground it stands on — linked so you can check our work against theirs.
The standard is a loop, not a report. spec → plan → test → build → verify → review → commit — and verify has teeth.
Both procedures are fingerprinted. The sealed seed is derived from the commit hash plus fresh entropy — drawn after the code is frozen. Tuning after the fact is structurally impossible.
Every truth() score comes from an outcome that actually happened. Synthetic inputs, if any, are labeled. Mocked runs fail by definition — the placebo gate catches them.
GOLD must hold on five independent seeds — a single lucky pass never counts. Then one final run on a holdout set the procedure has never touched. All six, or the loop continues.
The full report — wins, losses, flags, bounds, cost — ships with the commit. The receipt is the deliverable. "Trust me" is not a deliverable.
The open loop: you've seen the gates, the patterns, the live telemetry — but you haven't seen your domain pass the gate. That gap is exactly what the harness closes.
Every day you ship without a receipt, a claim ships that can't survive one. That's the loss — not a missed discount. The standard is free. The cost of skipping it compounds.
Get the judge() harness, the six-gate checklist, and the seed protocol. One file. Standard library only. Runs on your phone. Free forever — no account, no upsell.
Free · No spam · The standard belongs to everyone
Done is a claim. GOLD is a receipt.
Get the Standard →