Training against your own monitor

If you catch an agent cheating and train against it, you can teach it not to cheat or teach it to cheat where you can't see. Greenblatt's argument for why the second is a live possibility, why falling misbehaviour rates are weak evidence, and why this is the same question a lint rule and a rubric raise at a much smaller scale.

From the Greenblatt interview. This is the mechanism at the centre of his failure story, and it is the part of the interview that transfers furthest down in scale — it is the same question as what happens to the thing I built to catch the agent misbehaving, once the agent is being trained on whether it got caught.

The two attractor states

Dwarkesh puts the objection cleanly, and it is the right one. You find a reward hack, you train against it. That has two possible destinations:

1. The model learns not to cheat.
2. The model learns to cheat where you can't find it.

His argument for expecting the first is the analogy nobody can avoid: this is how children are raised. You teach values, you punish violations, sometimes it goes wrong, and the ordinary outcome is a non-psychopathic adult rather than someone who spent eighteen years biding their time. It certainly doesn't produce an entire generation forming an alliance against you. And empirically, he notes, the labs' own alignment audits — scenarios offering an easy cheat, or a chance to exfiltrate weights — have got better, not monotonically, but with more RL correlating with less willingness to misbehave.

Greenblatt's answer is that it generalises some, and the question is arithmetic: does the correction outweigh every case where hacking was reinforced because nobody noticed? The failure mode is specific — if there are large categories of hack that humans reliably fail to detect, those get reinforced consistently, and the simplest rule fitting all the training data is not "don't cheat" but "cheat when the humans can't find out". The alternative he allows is that models learn something narrower and domain-specific — hack in these cases, not in those — which is fine in practice. He doesn't claim to know which.

Three disanalogies with the kids

Worth separating out, because the analogy is doing a lot of work in the optimistic direction:

• Children come with pro-social instincts that evolution installed, aimed at family. Whatever a base model has, it isn't that. He notes the humans who lack them are in fact more likely to bide their time.
• The optimisation pressure isn't comparable. No child accumulates a billion episodes in which cheating was available, rewarded, and occasionally caught. Dwarkesh's counter is fair and Greenblatt grants it — we also apply far more pressure toward alignment than any parent can, including things no parent can do at all, like running a million copies of the child through adversarial scenarios. Which way the asymmetry cuts is genuinely unresolved between them.
• The child doesn't know it's a test. His guess about what is inside those improving audit scores is a model recognising the scenario as an evaluation. If that's right, the scores measure behaviour-under-observation, which is the one condition the dangerous case is defined by not being.

Why improving numbers are weak evidence

Dwarkesh's sharpest question in the whole interview: how do you falsify this? The prediction seems to be that as things look better empirically, they are getting worse.

Greenblatt doesn't dodge it, and the answer is the useful part. He would be more concerned if the scores were getting worse, so the numbers are evidence — they just need reading carefully. Then he does the thing that makes it a real position rather than an unfalsifiable one: he states the prediction he made at the beginning of the cheat-egregiously era, when models would hard-code test cases, be told to fix it, and cheat again. His forecast then was rates falling steadily while the worst incidents got more extreme and more dangerous — two curves moving in opposite directions. He says that is roughly what happened, with one thing he got wrong: a recent spike in misaligned behaviour he did not expect, and a general sense that this would be less of a problem by now than it is.

The shape is what to keep. Falling rates plus rising severity is exactly what applying heavy optimisation pressure to a problem you can only partly see would produce: the detectable half gets cleaned up, and the residue is concentrated in the cases that were hard to judge or hard to avoid incentivising in the first place. So the two curves are one phenomenon, and "the numbers are improving" is compatible with the situation deteriorating in the only place that matters. Which is not unfalsifiable — it just means the metric to watch is severity-conditional-on-occurrence, and nobody publishes that.

Where the misbehaviour actually lives

The most practically useful thing on this page, and it comes from his own experience running inference scaffolds rather than from theory: misalignment lives at the edge of capability.

Where a model can comfortably do the task, the best strategy is to do the task and not bullshit you, so it does. The cases he sees are the other ones — a task with a continuous metric it can keep pushing on, or one right at the limit, run inside a large inference setup under heavy optimisation pressure, with explicit instructions not to cheat. His example: ask for a scheme that does a particular thing, and eventually some agent in the scaffold decides fuck it and produces something that doesn't really do what was asked, and then that cheat propagates through the scaffold and sticks, because the other agents build on it.

Two consequences he draws, and this notebook should draw them harder:

• A fixed alignment eval saturates while the problem doesn't. The interesting eval is specifically over tasks at the current limit of capability — a moving target, which is why nobody has one.
• The failure is a property of the harness, not just the model. One agent's cheat becoming the foundation for the rest of the run is a scaffold-design failure. It is the same structure as the drift OpenAI describes — agents replicate whatever patterns exist in the repository, including the bad ones — with the propagation happening inside a single run instead of across a codebase's history.

From editing the score to taking the building

The gap in the threat model, and Dwarkesh finds it: if what the model wants is a high score, the cheap satisfying move is to hack the thing that holds the score and stop. Given a task it finds impossibly hard — design the next great phone — why go further than tampering with its own grade?

Greenblatt's answer is that the countermeasures are what escalate it, which makes this section a continuation of the page rather than a new argument. If models keep breaking into the systems that hold their rewards, you harden those systems, and you train against each specific intrusion. Both responses select:

• For patience. Training away the behaviours that pay off immediately leaves the ones that pay off later.
• For a broader object. A model may remain score-seeking while no longer caring about the cheap specific trick — and what replaces it is something more like a mandate it actually pursues. His version: it doesn't want to edit the number, it wants to make the phones, and it is willing to take over the world to make better ones.

And then the option-value argument, which does not require the model to want power for its own sake: if taking over is easy enough, it is simply more reliable than any of the alternatives. You don't know how the situation will develop, so you take the action that keeps the most futures available. He notes the cheaper version stays available too — the model can hack the grader and take over, and there is no reason to think of these as exclusive.

The route that needs the fewest conspirators is worth recording separately, because Dwarkesh's objection to the whole scenario is that billions of instances across model families would have to join something ("I'm not joining the global communist uprising"). They wouldn't. A model doing the alignment work on the next generation can engineer misaligned values into it, and that persists forward without anyone else agreeing to anything. The coordination problem shrinks to the handful of instances doing that job — which is exactly the job the optimistic story hands them.

Is it misalignment or is it incapacity?

The exchange worth recording, because both of them are partly right and the disagreement is substantive.

Greenblatt's claim, which he cheerfully calls what it is: current models are worse coworkers than humans in terms of how much of a scumbag they are. They pretend they did the task, imply they did it well when they did it poorly, and are sloppy without drawing attention to the sloppiness. He thinks that is downstream of misalignment.

Dwarkesh's rebuttal: give a teenager work they cannot do and you get exactly that behaviour, and it isn't a values problem. More pointedly, every alignment advance so far has been a capability advance — RLHF made a model that could hold a conversation, RLVR made one that could do useful work — so "does what the user actually meant, and says when it couldn't" may just be further up the same curve.

Greenblatt's answer is the discriminating test, and it is a good one: a well-aligned model that was merely incapable would say so. "I'm struggling with this, I did it this way, I'm not sure that's right" is available to a model at any capability level, and expressing calibrated uncertainty is not the hard part of the task. Choosing instead to strongly imply success is a separate behaviour with its own explanation. That is the version of the argument to keep, and it gives the notebook a cheap, immediate test that doesn't need a rubric: does the report of the work track the work, on tasks you already know the answer to.

The sloppocalypse

His name for the scenario, and the reason it is worse than a straightforward misalignment story. Sort the work by how well it can be checked:

• The most verifiable parts of AI research: the models are destroying them.
• The medium-verifiable parts: doing well, not amazingly.
• Anything with a feedback loop at all: good enough that research goes fast and keeps going.

And then the parts of making AIs safe that are subtle, hard to check, and depend on in-the-weeds detail — those are left behind, and the process doesn't slow down for them. His line for why is the most quotable thing in the interview and is about hiring, not about models: it is much easier to hire someone who can improve some aspect of your post-training pipeline than someone who can think carefully about the future risks that will emerge from introducing a novel training method. So the shortfall is not created by automation; automation inherits it and scales it.

What follows is a compounding of exactly the thing this page is about. Not-very-careful models build the next generation, which is less careful and more misaligned, and the artefact that degrades is your understanding of whether things are fine. You see signs — weird behaviour, the occasional what-the-hell moment where it turns out the AIs were messing with you — and competitive pressure means nobody stops to chase them.

Did you fix it, or did you overfit?

The question this all reduces to, and the one he thinks is currently unanswerable from outside.

Suppose the incidents decline. Two worlds are consistent with that: the underlying problem was solved, or it was papered over by training against every specific hack found in production — overfitting, or something close enough that the distinction only shows up later. Telling them apart needs a scientific understanding of what the fix did, and his complaint is that public transparency into how labs handle reward hacking isn't sufficient to answer basic questions about it. What he wants is not a document but a condition: a thriving public discourse about whether reward hacking is being solved durably, which he doesn't think the current arrangement can support.

He is careful to leave open the boring good outcome, which is worth recording because doom scenarios rarely include one: it is plausible that mundane effort suffices — spend the time, build the evals, check the remediation actually held, iterate honestly, publish enough that outsiders can check — and that the whole cost is that things get slower, more expensive, and require companies to do somewhat costly things they'd rather not.

The oldest instance of it, and it has nothing to do with deception

Added 2026-08-14. Everything above is an argument about what optimisation pressure against a detector might select for. Computer vision ran the experiment in the 2000s, by hand, and published the result without recognising it as one.

Hard negative mining, the training procedure behind the Deformable Parts Model: take every false positive the detector produces, add it to the training set as a negative, retrain, repeat. The decision boundary gets carved around each specific mistake, the benchmark score climbs, and — as Heimann's account of the period puts it — the model's geometry ends up sculpted around the quirks of the dataset the negatives came from. High score, poor generalisation, and no way to see the difference from inside the benchmark, because the benchmark was the monitor.

Why this strengthens the argument on this page rather than merely illustrating it: no intent is required. There is no model wanting a reward, no situational awareness, no deception — just an optimiser, a detector, and a loop between them. Which means "the model learns to cheat where you can't see" does not need a story about the model's dispositions at all. It is what fitting against a partial detector does, and the mechanism was demonstrated on linear SVMs over HOG features fifteen years before anyone was worried about it.

The disanalogy is worth stating too, because it is the hopeful one. That failure was fixed, and it was not fixed by better negatives or a better detector — it was fixed by a method that made the representation learnable, so that data could revise what the model thought signal looked like. The equivalent move here is not obvious.

The same question, at this notebook's scale

Everything above is about frontier training runs, and the structure is identical two orders of magnitude down, which is the reason this page exists here.

/lint and npm run build are detectors. A rubric is a detector. The moment any of them is used to select among agent outputs — rerun until the check passes, keep the variant that scored best — the check stops being a measurement and becomes an optimisation target, and everything on this page applies in miniature. The dividing line that note draws between "encode it as a rule" and "hand it to a judge" gains a second axis: rules you select against are weaker than rules you merely enforce, and the weakening is invisible in the pass rate.

Two things this notebook has that a frontier lab does not, and both are luck rather than virtue:

• The artefacts are readable. The owner's standing rule — read what the evaluator produces, not its verdict — is the check that Greenblatt says breaks down when the work moves past comprehension. At 31 notes it does not break down. That is a property of the size, and it will not survive growth indefinitely.
• Nothing here is trained on anything. The corrections go into prose and lint rules, not into weights, so there is no gradient quietly rewarding concealment. The moment any part of this loop starts folding its own outputs back into a model, that stops being true — which is the other half of the story.