The Goodhart gate: when your metric starts gaming you back

You know the sentence. Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." Everyone nods at it in meetings, then ships the metric as a target anyway. This post is about why the standard treats it as a hard gate rather than a cautionary quote.

The trap is structural, not moral

Nobody games a metric on purpose. It happens because you optimize the one number you can see, and the one number you can see is never the whole truth. The scoreboard goes up; the actual outcome doesn't move. This is the same phenomenon DeepMind catalogued as specification gaming — agents that satisfy the letter of the objective while violating its spirit. The letter is the proxy. The spirit is the truth.

In the GOLD framework, every domain can define an optional proxy — a signal the procedure is allowed to optimize — alongside the sealed truth. Gate 05 compares them: gain on the proxy must not outrun gain on truth by more than a tolerance. If the proxy improved 0.5 while truth improved 0.1, the procedure is climbing the scoreboard, not moving the outcome. The gate fails. It doesn't matter how good the proxy number looks.

What the check looks like in the report

truth_gain (mean across slices) = 0.1731
proxy_gain (mean across slices) = 0.1580

check: (proxy - truth) <= 0.25 * max(|truth|, 1e-9)
goodhart_ok: True

When a procedure does outrun its own scoreboard, the report says so — and the flag stays in the report forever. That's the part people find uncomfortable. In the real benchmark behind this site, 11 Goodhart flags were recorded and none of them were deleted. The correlation between the rubric score and the judge score came out at −0.144 — meaning the score imperfectly predicts quality — and that negative number stayed in the summary too.

The rule: a standard that hides its failures is just marketing. GOLD reports everything; it only passes what survives.

Why tolerance, not equality

Gate 05 isn't "proxy gain must equal truth gain." That would fail honest procedures that legitimately improve both. The check allows the proxy to run slightly ahead — within a tolerance of 25% of the truth gain — because some divergence is expected noise. What it refuses is the large gap: proxy racing ahead by multiples while truth barely moves. That shape only appears when a procedure has found a shortcut through the scoreboard.