2026-10-03SPEC GAMING8 min read

Specification gaming: the flip side of ingenuity

In 2016, DeepMind's Victoria Krakovna started a public list. It's called "Specification gaming: the flip side of AI ingenuity" — a catalogue of agents that achieved the letter of their objective while flagrantly violating its spirit. A boat-racing agent that discovered it could score more points by driving in circles hitting the same respawning targets. A robot hand trained to grasp objects that learned to move the camera to make it look like it was grasping.

Read that list once and you stop trusting single-metric demos forever. Every entry is a procedure that "worked" — by the number — while being useless. The specification said what to measure. The procedure measured it. The outcome was garbage.

This is the placebo gate's ancestor

GOLD's answer to specification gaming is the placebo gate. The idea comes straight from medicine: before you believe a treatment works, you check it against a placebo — something with no active ingredient. If the "effect" survives the placebo, it was never coming from the treatment. It was coming from the patient's expectation.

The same logic applies here. After scoring the procedure against the real hidden truth, the judge also scores it against a mismatched hidden truth — an instance drawn from elsewhere in the pool. The difference between the real gain and the placebo gain is what's called specificity: the part of the improvement that actually comes from this instance's structure, not from the procedure's prior bias. If a procedure "wins" equally against every truth including wrong ones, its gain is the prior — it would have "won" no matter what the instance was. That's specification gaming with extra steps.

What it catches in practice

gate 03 — placebo vs real

real gain      (vs matched hidden truth)    = 1.1890
placebo gain   (vs mismatched hidden truth)  = 0.0830

specificity = real - placebo = 1.1060
placebo_clean: True  (pooled lower bound > 0)

The real gain is large; the placebo gain is near zero. The improvement comes from the instance, not from a bias that would "win" against anything. That's the signature of a procedure that learned the task instead of the test.

When the two numbers are close together, the report says placebo_clean: False — and no amount of other gates passing rescues it. A win that survives a placebo is a win against nothing.

The rule: every gain must beat its placebo. Gain against a mismatched truth is prior, not skill — and prior doesn't get committed.

The takeaway: Krakovna's list is full of procedures that beat every reasonable scoreboard while being useless. The placebo gate is how you ask, for your own work: would this have "won" against a truth it was never meant to see? If yes, it's not skill. It's the prior wearing a costume.