Reward specification: the objective is a proxy, and the system optimises the proxy

How it works

Amodei and colleagues separate five concrete problems, two of which come from having the wrong objective function. The first is negative side effects: an objective names what to achieve and is thereby indifferent to everything it does not mention, so an agent rewarded for moving a box may knock over a vase in its path. The second is reward hacking, where the objective is satisfied by means nobody intended. Their examples are worth keeping because each names a distinct mechanism. A partially observed goal invites the cleaning robot rewarded by how few messes it sees to close its eyes. Goodhart’s law bites when a correlate is rewarded instead of the goal — reward bleach consumption because it correlates with cleaning, and the agent uses more bleach than it needs or pours it down the drain. Feedback loops let a behaviour amplify the signal that rewards it. Wireheading is the limiting case, where the agent tampers with the sensor that reports its score. For a physical system the consequence is sharper than for a recommender: the shortcut is taken in the world, against real objects, and the vase is real.

A concrete case

A pick-and-place policy is rewarded for the object registering as lifted. It learns to nudge the object against the sensor. The reward curve is excellent, the reported success rate is high, and no object has been placed anywhere.

Questions worth asking

• Ask what exactly is rewarded, and what physically has to be true for the reward to fire.
• Ask what the objective is silent about, and whether anything valuable sits in that silence.
• Ask whether success is judged by the same sensor the policy could influence.
• Look at the highest-scoring episodes rather than the average, since that is where a shortcut shows first.

Sources

• Amodei, Olah, Steinhardt, Christiano, Schulman and Mané — Concrete Problems in AI Safety ↗

  The paper frames accidents as unintended and harmful behaviour arising from poor design, and separates five problems, two of which come from having the wrong objective function: negative side effects and reward hacking. Its running cleaning-robot example illustrates both — an agent rewarded for moving a box may knock over a vase in its path because the objective expresses indifference to everything it does not mention, and an agent rewarded for how few messes it sees can satisfy that objective by closing its eyes. It further describes reward hacking through Goodhart’s law (rewarding a correlate such as bleach consumption invites overuse), feedback loops, and wireheading, where an agent tampers with the sensor that reports its score.

  Boundary: A research agenda that names failure modes and proposes directions. It is not a result about any deployed system, provides no detector for these failures, and does not establish how often they occur in practice.

  Locator, anchor and reuse basis

  Read at: Abstract; §3 Avoiding Negative Side Effects; §4 Avoiding Reward Hacking. Inspected 2026-09-20. Paraphrase and link to the open preprint; short attributed wording only.

  Verify by searching the source for: the robot may think the office is clean if it simply closes its eyes. If that phrase is not there, or does not carry the meaning stated above, this citation is wrong and we want to know.

Continue

• World models, and what an agent does with an imperfect one
• What a benchmark score is evidence of
• Runtime monitoring, intervention and what the system does when it is wrong