Tag: harness-design
DoorDash's model spend rose about 20× in five months, so they built a benchmark over their own coding tasks to work out what it bought — and found that the models crush a scrubbed version of a task and then underperform on the real data. The first buyer-side view in this notebook, and the open question it leaves.
A Latent Space writeup of early access to OpenAI's GPT-6 Astra, framed around the claim that it is cheap enough and capable enough to function as a junior AI Engineer — running 20-50 subagents in parallel, monitoring its own multi-day jobs, and building its own benchmarks. Read as a vendor demo rather than a measurement, and cross-checked against what the notebook already has on spend, throughput and unattended runs.
Four mechanisms decide what starts an agent's next turn and what stops the sequence — /goal, /loop, Stop hooks and scheduled runs. Notes on the /goal documentation, including the constraint that shapes everything else: its evaluator has no tools and can only judge what the transcript shows.
The emerging discipline around long-running coding agents — designing the scaffolding, feedback loops and environments that let an agent do reliable work, rather than writing the code yourself. Entry point for the harness-design cluster in this notebook.
A conference talk arguing that the spec-to-code loop degrades a codebase on every pass, that the reason is ordinary software entropy, and that five failure modes of AI coding each have a fix sitting in a twenty-year-old book. The first source in this notebook that locates the harness inside the code rather than around it.
Ousterhout's deep modules, repurposed as an agent-legibility property and as the line between what a human designs and what an agent is allowed to write unread. Why the boundary is the unit of delegation, and the one place this collides with the notebook's other evidence.
Notes on Ryan Lopopolo's OpenAI post — five months shipping a product with no hand-written code, and what a repository has to look like for that to work: docs as the system of record, a bootable app per worktree, custom linters as taste enforcement, and continuous cleanup of agent drift.
Custom lint rules as the enforcement layer for conventions an agent would otherwise drift away from — architecture boundaries, taste invariants, documentation freshness. Why a linter beats an instruction, and what to put in the error message.
The claim that the unit of work is shifting from the prompt to the loop — Cherny, Osmani and Guzman all saying a version of it — and why the interesting question is not how to keep an agent running but who decides what "done" means.
Domain-driven design's shared vocabulary, generated from a codebase and handed to a model as a working artefact. The reported effect is the odd part — not just better plans but shorter thinking — and it is cheap enough to test here.
A lab put its entire training pipeline — pre-training, SFT, RL, evaluation — into imperative code, with CI as the entry point and one-click hermetic runs. The first source in this notebook where the harness is built for training a model rather than for writing software, and the argument is organisational rather than technical.
Planning note for a 50-minute talk in Prague — the business case for specialized small models, then the training harness live. Includes the evidence check that killed the original opening slogan, the six conditions under which fine-tuning is the right call, and the caveat about borrowing a frontier lab's factory as a template.
Karpathy's name for an agent loop that improves a model without a human in it — an objective, a metric, boundaries, and go. What it did to a repo he had hand-tuned for two decades, the one constraint that decides where it applies, and the parallel version he has not solved.
If you catch an agent cheating and train against it, you can teach it not to cheat or teach it to cheat where you can't see. Greenblatt's argument for why the second is a live possibility, why falling misbehaviour rates are weak evidence, and why this is the same question a lint rule and a rubric raise at a much smaller scale.
What it would concretely take to train a model to do AI research — the environments, the three scales they run at, and the one part of the job Greenblatt thinks stays unverifiable. Plus the argument that algorithms rather than bought human judgment have been driving progress, which is what makes the whole thing work.
Splitting the agent that does the work from the agent that judges it — why self-evaluation fails, what makes an evaluator worth its cost, and how three independent teams arrived at the same shape from taste, from QA, and from production corrections.
Notes on the Tax AI post — how practitioner corrections in production become structured findings, then targeted evals, then bounded engineering tasks a coding agent can close. The clearest published example of an eval-driven improvement loop with a named metric that moved.
Karpathy's name for the layer above a coding session — something that keeps looping in its own sandbox, remembers more than a compacted context, and answers on one messaging channel. The home-automation example he built, and why it is a partial answer to two open questions in this notebook.
Karpathy makes the same argument twice in one interview — smart-home apps should be APIs, docs should be markdown for agents rather than HTML for people — because in both cases an agent consumes the interface and routes to a human. Where that lands for this notebook, where it thins out, and what changes when a vendor does it deliberately.
A staged plan for building the Anthropic generator/evaluator harness myself, cheapest step first — starting with the rubric alone and only adding orchestration where it earns its keep. Plus what's changed since the March post, and the third-party implementations worth reading.
Notes on Prithvi Rajasekaran's Anthropic Labs post — a GAN-inspired generator/evaluator loop applied first to frontend design, then extended to a planner/generator/evaluator harness that builds full-stack apps over multi-hour autonomous runs, with the costs and the walk-back to a simpler harness.
The pieces the 2026 harness-design posts build on — Anthropic's earlier long-running-agent harness, context engineering, building effective agents, the frontend design skill — captured as references because the article bodies could not be fetched from this session.
Notes on the NoPriors interview where Karpathy says he hasn't typed a line of code since December, that every remaining failure feels like a skill issue rather than a capability ceiling, and that token throughput is the resource you should feel nervous about wasting. Plus the jaggedness caveat that undercuts the whole picture.
A piece by Anton Leicht on sovereign AI sent me looking for how the frontier labs actually run their internal coding agents. The answer is that they are still experimenting — nobody has Auto Research — which means the harness is mine to build, and building it well is a skill worth training.
Six months of AI coding have made it obvious that the human is the bottleneck. Where that leaves my role — closer to managing a small team of agents than to doing QA on their output — and the practical questions that follow from it.