Tag: ai-safety
Production traffic is becoming training data — tasks the model did badly on, turned into environments that match them exactly, with the rubric built from what the human actually wanted. Which is the loop this notebook already admires, run with the model as the artefact, and it changes where the reinforcement comes from.
Ryan Greenblatt argues that AI research is verifiable enough to automate itself, that automating it buys four or five years of progress in one, and that what comes out the other end can transform the world without ever learning to play politics. His median for full automation is 2030–31 and his odds on takeover by 2040 are 35–40%.
Dwarkesh wants a model that is his advocate the way a lawyer is; the published specs describe something closer to an ethical contractor with its own view of the good. Greenblatt thinks the fiduciary version would be better and then makes the strongest case against it — that a society running on do-whatever-you-ask labour loses a check it depends on.
OpenAI withheld GPT-2 in February 2019 on the grounds of risks that a later post admitted had not materialised, and the justification pointed at future systems rather than at the model being withheld. The argument structure — act early because acting late is worse — is the one that later produced the board crisis, and it is unresolved in exactly the same way today.
If you catch an agent cheating and train against it, you can teach it not to cheat or teach it to cheat where you can't see. Greenblatt's argument for why the second is a live possibility, why falling misbehaviour rates are weak evidence, and why this is the same question a lint rule and a rubric raise at a much smaller scale.