All notes
DoorDash's model spend rose about 20× in five months, so they built a benchmark over their own coding tasks to work out what it bought — and found that the models crush a scrubbed version of a task and then underperform on the real data. The first buyer-side view in this notebook, and the open question it leaves.
A Latent Space writeup of early access to OpenAI's GPT-6 Astra, framed around the claim that it is cheap enough and capable enough to function as a junior AI Engineer — running 20-50 subagents in parallel, monitoring its own multi-day jobs, and building its own benchmarks. Read as a vendor demo rather than a measurement, and cross-checked against what the notebook already has on spend, throughput and unattended runs.
Four mechanisms decide what starts an agent's next turn and what stops the sequence — /goal, /loop, Stop hooks and scheduled runs. Notes on the /goal documentation, including the constraint that shapes everything else: its evaluator has no tools and can only judge what the transcript shows.
The emerging discipline around long-running coding agents — designing the scaffolding, feedback loops and environments that let an agent do reliable work, rather than writing the code yourself. Entry point for the harness-design cluster in this notebook.
A conference talk arguing that the spec-to-code loop degrades a codebase on every pass, that the reason is ordinary software entropy, and that five failure modes of AI coding each have a fix sitting in a twenty-year-old book. The first source in this notebook that locates the harness inside the code rather than around it.
Ousterhout's deep modules, repurposed as an agent-legibility property and as the line between what a human designs and what an agent is allowed to write unread. Why the boundary is the unit of delegation, and the one place this collides with the notebook's other evidence.
Notes on Ryan Lopopolo's OpenAI post — five months shipping a product with no hand-written code, and what a repository has to look like for that to work: docs as the system of record, a bootable app per worktree, custom linters as taste enforcement, and continuous cleanup of agent drift.
Custom lint rules as the enforcement layer for conventions an agent would otherwise drift away from — architecture boundaries, taste invariants, documentation freshness. Why a linter beats an instruction, and what to put in the error message.
The claim that the unit of work is shifting from the prompt to the loop — Cherny, Osmani and Guzman all saying a version of it — and why the interesting question is not how to keep an agent running but who decides what "done" means.
Domain-driven design's shared vocabulary, generated from a codebase and handed to a model as a working artefact. The reported effect is the odd part — not just better plans but shorter thinking — and it is cheap enough to test here.
Production traffic is becoming training data — tasks the model did badly on, turned into environments that match them exactly, with the rubric built from what the human actually wanted. Which is the loop this notebook already admires, run with the model as the artefact, and it changes where the reinforcement comes from.
Karpathy expects specialised models and does not see them arriving, and his explanation is that adjusting weights without losing capability is still an underdeveloped science while context windows just work. Plus where he puts the open-weight models — six to eight months behind, and better off there.
A lab put its entire training pipeline — pre-training, SFT, RL, evaluation — into imperative code, with CI as the entry point and one-click hermetic runs. The first source in this notebook where the harness is built for training a model rather than for writing software, and the argument is organisational rather than technical.
Planning note for a 50-minute talk in Prague — the business case for specialized small models, then the training harness live. Includes the evidence check that killed the original opening slogan, the six conditions under which fine-tuning is the right call, and the caveat about borrowing a frontier lab's factory as a template.
Karpathy's name for an agent loop that improves a model without a human in it — an objective, a metric, boundaries, and go. What it did to a repo he had hand-tuned for two decades, the one constraint that decides where it applies, and the parallel version he has not solved.
Why video generation is stuck at five-second clips — quadratic attention over 35,000 tokens for five seconds of 480p — and why the open/closed gap that has nearly closed for text is night-and-day for video. Plus the argument that text diffusion is being marketed as the wrong product.
Philip Kiely and Ali Taha on what actually happens between an open-weight checkpoint and a production endpoint — the four optimisations that stack to 2–4× on fixed hardware, why quantising more of a model can make it better, and the reliability failures that turn out to be a race condition in a kernel rather than anything about the weights.
Ryan Greenblatt argues that AI research is verifiable enough to automate itself, that automating it buys four or five years of progress in one, and that what comes out the other end can transform the world without ever learning to play politics. His median for full automation is 2030–31 and his odds on takeover by 2040 are 35–40%.
Inference engineers now write training pipelines, RL rollouts bottleneck on inference speed, and Baseten reports running GPU kernels inside their inference engine that GLM-5.2 wrote for itself from its own profiling traces. Plus the argument that continual learning belongs in the KV cache rather than in the weights.
Dwarkesh wants a model that is his advocate the way a lawyer is; the published specs describe something closer to an ethical contractor with its own view of the good. Greenblatt thinks the fiduciary version would be better and then makes the strongest case against it — that a society running on do-whatever-you-ask labour loses a check it depends on.
Before AlexNet the field's best argument against neural networks was that the data was doing the lifting — and Efros had the control experiment to prove it. AlexNet settled that case on the same data everyone else had. The reason to keep the episode is that the modern version of the argument is running now, with the control missing.
Karpathy's argument that models are advancing only where a reward can be computed, with a joke that hasn't changed in three or four years as the demonstration — and why that is the same boundary the harness cluster keeps running into from the other side.
Computer vision spent the 2000s carving its best models around the quirks of the datasets it was scored on, and hard negative mining was how it did it — training against your own error cases, one gradient step at a time. The episode is the oldest worked example in this notebook of a measure being optimised until it stops measuring.
Richard Heimann's book reads the reconstructed reading list Ilya Sutskever gave John Carmack as an argument rather than a bibliography. Chapters 1 and 2 give the four-part worldview it claims to encode, and then spend a chapter showing that AlexNet invented almost nothing — which is the part this notebook cares about.
OpenAI withheld GPT-2 in February 2019 on the grounds of risks that a later post admitted had not materialised, and the justification pointed at future systems rather than at the model being withheld. The argument structure — act early because acting late is worse — is the one that later produced the board crisis, and it is unresolved in exactly the same way today.
If you catch an agent cheating and train against it, you can teach it not to cheat or teach it to cheat where you can't see. Greenblatt's argument for why the second is a live possibility, why falling misbehaviour rates are weak evidence, and why this is the same question a lint rule and a rubric raise at a much smaller scale.
What it would concretely take to train a model to do AI research — the environments, the three scales they run at, and the one part of the job Greenblatt thinks stays unverifiable. Plus the argument that algorithms rather than bought human judgment have been driving progress, which is what makes the whole thing work.
Splitting the agent that does the work from the agent that judges it — why self-evaluation fails, what makes an evaluator worth its cost, and how three independent teams arrived at the same shape from taste, from QA, and from production corrections.
Notes on the Tax AI post — how practitioner corrections in production become structured findings, then targeted evals, then bounded engineering tasks a coding agent can close. The clearest published example of an eval-driven improvement loop with a named metric that moved.
Seven writers independently arrive at the same move this notebook's harness material arrives at — never let the thing that produces the work also judge it while it is being produced. The structure matches; the reason for it is the exact opposite, and that asymmetry is the interesting part.
A compilation episode in which seven writers answer the same four questions about practice — how to choose a project, how to begin without inspiration, how to produce pages reliably, and what to do when the pages are bad. The answers agree on more than they disagree, and the disagreements are the useful part.
Notes on the NoPriors interview with DoorDash's two co-founders — a natural-language ordering surface that half the time sends people to a merchant they have never used, a delivery robot built to a use case rather than to a capability, and an internal benchmark built to work out what a 20× rise in model spend actually bought.
Karpathy's name for the layer above a coding session — something that keeps looping in its own sandbox, remembers more than a compacted context, and answers on one messaging channel. The home-automation example he built, and why it is a partial answer to two open questions in this notebook.
Karpathy makes the same argument twice in one interview — smart-home apps should be APIs, docs should be markdown for agents rather than HTML for people — because in both cases an agent consumes the interface and routes to a human. Where that lands for this notebook, where it thins out, and what changes when a vendor does it deliberately.
DoorDash's argument for why they had to build their own robot — the field builds a capability and then hunts for a problem — plus the edge cases nobody writes down at a desk, and the one incumbent data advantage in the interview that is actually load-bearing.
A staged plan for building the Anthropic generator/evaluator harness myself, cheapest step first — starting with the rubric alone and only adding orchestration where it earns its keep. Plus what's changed since the March post, and the third-party implementations worth reading.
Notes on Prithvi Rajasekaran's Anthropic Labs post — a GAN-inspired generator/evaluator loop applied first to frontend design, then extended to a planner/generator/evaluator harness that builds full-stack apps over multi-hour autonomous runs, with the costs and the walk-back to a simpler harness.
The pieces the 2026 harness-design posts build on — Anthropic's earlier long-running-agent harness, context engineering, building effective agents, the frontend design skill — captured as references because the article bodies could not be fetched from this session.
Notes on the NoPriors interview where Karpathy says he hasn't typed a line of code since December, that every remaining failure feels like a skill issue rather than a capability ceiling, and that token throughput is the resource you should feel nervous about wasting. Plus the jaggedness caveat that undercuts the whole picture.
A piece by Anton Leicht on sovereign AI sent me looking for how the frontier labs actually run their internal coding agents. The answer is that they are still experimenting — nobody has Auto Research — which means the harness is mine to build, and building it well is a skill worth training.
Six months of AI coding have made it obvious that the human is the bottleneck. Where that leaves my role — closer to managing a small team of agents than to doing QA on their output — and the practical questions that follow from it.
Andrej Karpathy's pattern for an LLM-maintained personal wiki — kept verbatim at the repo root as LLM_Wiki.md — is the founding idea mantua.io itself instantiates. What the pattern claims, and where this repo's implementation matches or departs from it.
Running notes on Christoph Molnar's "Tabular Foundation Models" — chapter takeaways, questions, and how tabular FMs fit into this notebook's map of SOTA ML-engineering technology.
Running notes on Manning's "LLM Customization and Fine-Tuning" (MEAP, Bahree & Tok) — chapter takeaways, questions, and links back into the rest of this notebook.
Lab notes on turning a spare Linux desktop with a 12 GB GPU into an always-on, OpenAI-compatible HTTP endpoint for open-weight small models — VRAM budgeting, a dated model shortlist, llama.cpp vs. Ollama vs. vLLM, systemd, and not exposing the thing naked to the internet.