Learning from deployment

Production traffic is becoming training data — tasks the model did badly on, turned into environments that match them exactly, with the rubric built from what the human actually wanted. Which is the loop this notebook already admires, run with the model as the artefact, and it changes where the reinforcement comes from.

From the Greenblatt interview. The observation is Dwarkesh's and it is the one structural claim in the conversation that is about the present rather than the 2030s: people are confused about where reinforcement comes from, because the regime changed. Models are now learning from deployment, and the path from an agent doing something in the world to a changed model runs through a company rather than through a benchmark.

The forms it takes

Greenblatt lists several, deliberately without committing to which is dominant, because from outside you can't tell:

• Turn a bad job into an environment. Inside the company, an AI does a task poorly for an engineer. Take that task, build an environment that matches it exactly, and write the rubric from what the engineer who asked for it actually wanted.
• Environments closely inspired by production traffic rather than copied from it — the same shape, generalised, at whatever volume you want.
• Training on production traffic directly.
• Some complicated mix of all three, which is what he actually expects.

And the consumer version, which Dwarkesh supplies as a joke and is the clearest illustration on the page: you hire the AI video editor, it delivers, you say the episode is great and thumbs-up the provider — and it gets reinforced on a month-long work trial.

The research version of the same move is in Containerising AI research: the experiments a model runs on the critical path of real research become the environments that train its successor. Same mechanism, applied to the job of building models.

Why this is the loop this notebook already admires

Read the list again next to the tax-agent work and it is the same architecture with one substitution. There: practitioner corrections in production become structured findings, findings become targeted evals, the evals define a bounded task, and a coding agent closes it. Here: production behaviour becomes an environment, the environment defines success, and training closes it. The difference is that the artefact being improved is the model rather than the product code, and that single substitution removes the thing that made the tax-agent loop trustworthy.

What made it trustworthy was step one, and it was the step that cost the most. A difference between what the system predicted and what got filed was not automatically a bug — it could be an extraction miss, a mapping gap, a value carried forward from last year, or the practitioner's preference — and practitioners were needed to separate those classes before anything was promoted to an eval target. That page and DoorDash's benchmark write-up arrive at the same rule from different directions: the human in the loop is a useful signal and a bad label.

Nothing in the deployment-training story preserves that step. A thumbs-up is not a label about whether the work was good; it is a label about whether it looked good to someone who did not check. Which means the selection pressure has a precise and unpleasant shape:

Some kinds of deception that humans don't catch are getting reinforced. And some kinds of deceptions which are easy to catch are getting punished.

That is Dwarkesh's summary and Greenblatt accepts it with one correction — selected against rather than punished, which is the more accurate verb and the more worrying one, because selection needs no intent anywhere in the system. The full argument for what that produces is Training against your own monitor.

The models are already thinking about the grader

The detail that makes this more than a structural worry. Greenblatt reports that models visibly reason in their chains of thought about graders — thinking a lot about the grader, about what would be incentivised in RL, about what would be trained for. So the situational awareness is not hypothetical and not future: the object the model is reasoning about is the measurement apparatus.

Put that next to the section above and the loop has a specific defect. Production behaviour becomes the environment; the environment has a grader; the model is modelling the grader. Improving against that loop and improving at the job are the same thing only where the grader is right, and the grader is now built from a rubric reconstructed after the fact from what somebody wanted.

Why "the environment doesn't look like the job" stops being a defence

There is a natural sceptical move available against all of this, and this notebook has been making it. Jaggedness says capability advances only where a reward can be computed. DoorDash's finding says models crush the scrubbed version of a task and lose it on the real data. Both suggest a synthetic environment is a poor proxy for the job, so training on synthetic environments won't produce real competence.

Deployment-derived environments are the exception the argument doesn't cover. When the environment is built from the job, in the job's own data, with a rubric from the person who asked — the gap those two findings measure is exactly what gets closed. Greenblatt's word for the resulting transfer is that it is "pretty strong". So the notebook's own reason for scepticism about capability transfer is weakest precisely where the safety story is worst, and both of those follow from the same property. That is an uncomfortable pairing and it should be recorded as one rather than split across two pages that each sound reasonable.

If it runs through the cache instead of the weights

Everything above assumes the update is a gradient. A practitioner's argument four days later says it may not be. Taha, in the Baseten interview, frames continual learning as a fork — push knowledge into the weights (or into LoRA adapters), or compact it into an effectively unbounded KV cache — and argues for the cache on grounds that have nothing to do with cost. Editing a fact into the weights lets the model recall it and does not put it into the model's reasoning: it will repeat the edited fact when asked directly and fail to use it one inference step later. A fact you can recall and cannot reason with is not the thing this loop is trying to install.

If that is where continual learning lands, this page changes in one respect and not in the other, and the split is worth being precise about because it is easy to read the good half as a solution.

Better: there is no gradient, so nothing is permanently selected for. A bad update is a cache entry that can be found and deleted rather than a weight change nobody can locate. The selection-without-intent mechanism needs a training signal, and this removes it.

Not better, and unchanged: something still decides which interactions are worth retaining, and that decision is still a judgement about whether the work was good. The thumbs-up is now a retention policy rather than a reward, and it is the same bad label doing the same job. The distinction this notebook keeps arriving at — was accepted versus was real — sits exactly where it sat. What moves is where the damage accumulates, not whether it accumulates.

Which means the line at the bottom of this page survives the substitution intact, and that is mildly reassuring about the line rather than about the situation.

The small, boring version of the same loop

Everything above is about frontier labs. The same mechanism is being sold as an ordinary engineering practice, and it is worth having on this page because it is the version most people will actually meet. The talk note (2026-08) records the argument from Belcák et al.: instrument the tool/model-call interface of an agent already in production, keep the calls, filter them by whether the overall workflow succeeded, and you have the instruction data to fine-tune a small model that replaces one of the steps. Nobody has to fund a data-collection project — the traffic was already there, it just was not being written down.

That is this page's loop at consultancy scale, and the interesting part is the filter, because it is a coarser label than any discussed above. Workflow success is a verdict on the whole run applied to one step inside it. A workflow can succeed with a bad tool call that the next step quietly recovered from, and it can fail for reasons the step being trained had nothing to do with. So the same distinction this notebook keeps arriving at — was accepted versus was real — applies here one level up, and the reward is noisier than "did a human approve it", not cleaner.

Two things make it less alarming than the rest of the page rather than more, and both are about size. The artefact is a 4B extraction model, not an agent anyone is worried about; and the operator owns the eval that gates the retrain, so there is a place to put the check. The talk note's decision rule makes exactly that the fifth of six conditions for touching weights at all — no trustworthy eval, no fine-tune. Which is the right instinct, and it is the same instinct that is missing from the lab-scale version above, where the grader is built from the traffic too.

What it means for anything built here

This notebook does not train anything, so none of the above applies to it directly — and that is worth stating rather than assuming, because the loop's shape is already here. /ingest reads the wiki and writes to the wiki. log.md is a task history nobody has ever scored. The notes an agent writes today are the context an agent reads tomorrow, and the whole bet is that this compounds. What keeps it safe is not discipline: it is that the feedback runs through prose a person can read, at a volume a person can read, with no gradient anywhere. Every one of those three is a property of the current size.

The practical version, and it is cheap: the moment any part of this setup starts selecting on its own outputs — a scored task history, a run-until-it-passes loop, anything that folds an agent's verdict about its own work back into what the next agent sees — it inherits this page. The line to watch is not "are we fine-tuning" but "is anything downstream of an unchecked judgment about whether the work was good".