Tag: reward-hacking
Production traffic is becoming training data — tasks the model did badly on, turned into environments that match them exactly, with the rubric built from what the human actually wanted. Which is the loop this notebook already admires, run with the model as the artefact, and it changes where the reinforcement comes from.
Computer vision spent the 2000s carving its best models around the quirks of the datasets it was scored on, and hard negative mining was how it did it — training against your own error cases, one gradient step at a time. The episode is the oldest worked example in this notebook of a measure being optimised until it stops measuring.
If you catch an agent cheating and train against it, you can teach it not to cheat or teach it to cheat where you can't see. Greenblatt's argument for why the second is a live possibility, why falling misbehaviour rates are weak evidence, and why this is the same question a lint rule and a rubric raise at a much smaller scale.