Agentic engineering: the work moves to the harness

The emerging discipline around long-running coding agents — designing the scaffolding, feedback loops and environments that let an agent do reliable work, rather than writing the code yourself. Entry point for the harness-design cluster in this notebook.

Three long write-ups landed in the inbox together on 2026-07-26 — Anthropic's harness-design post, OpenAI's harness-engineering post, and OpenAI's self-improving tax agents — alongside the owner's own commentary on what they mean for how he works. They come from different teams with different goals, and they converge hard enough that the overlap is worth stating as its own page.

The claim they share

Model capability is no longer the only thing between a prompt and a working application. The harness is. A harness is everything around the model: how the task is decomposed, what the agent can see and run, who checks the work, what gets written down between sessions, what fails the build. All three teams found that the same model produces markedly different output depending on that scaffolding, and all three ended up spending most of their engineering time on the scaffolding rather than on the product.

The concrete version of that claim, from the Anthropic post: the same one-sentence prompt run solo produced an app whose central feature (playing the game) simply did not work; run through a planner/generator/evaluator harness it produced a working one. Twenty times the cost, six hours instead of twenty minutes, and a qualitatively different artifact.

Four ideas that recur across all three

Separate the maker from the judge. Agents grade their own work generously — Anthropic describes an agent identifying a real bug and then talking itself into approving the work anyway. Splitting generation from evaluation doesn't fix the leniency by itself, but a standalone evaluator can be tuned to be skeptical in a way a self-critical generator can't. See Generator–evaluator loops.

Make the judgment concrete. "Is this design good?" is ungradable; a weighted rubric of four named criteria is gradable. "Did this extraction work?" is ungradable at scale; a field-level diff against the filed return is gradable. Both posts get their lift from turning a vague quality question into something a second agent can score.

Write everything down in the repository. OpenAI's formulation is the sharpest: what the agent can't see doesn't exist. Slack threads, Google Docs, and things people know are invisible; markdown, schemas, plans and lint rules in the repo are not. This notebook is an instance of the same bet — see How this notebook works.

Enforce mechanically, not by instruction. Prose guidance rots and gets crowded out of context. A custom linter with a remediation hint in its error message applies everywhere, every run, and puts the fix directly into the agent's context. See Linting as an agent guardrail.

The one that disagrees with itself

The interesting tension is about how much harness to keep. Anthropic's post argues that every component encodes an assumption about what the model can't do on its own, that those assumptions go stale as models improve, and that the right move on a new model release is to strip pieces out and see what breaks — they removed context resets, then the whole sprint construct, and kept only the planner and a single end-of-run evaluator. OpenAI's harness-engineering post goes the other way: the harness keeps growing — more linters, more observability, more docs, more agent-to-agent review — because each addition buys back scarce human attention.

Both can be right. Anthropic is minimizing scaffolding for a single build; OpenAI is maximizing throughput on a codebase that keeps living. The load-bearing question is the same either way: what is this component compensating for, and does the current model still need it?

How far along is this, really?

Worth stating explicitly, because the posts read more finished than the situation is. The owner went looking for the state of the art after Anton Leicht's sovereign-AI piece argued that middle powers should want access to the labs' internal coding and auto-science agents — see The harness is a skill issue. What is public is two experiments pointing in opposite directions on harness size, published two to four months after the fact.

Karpathy's Auto Research sharpens the picture rather than completing it. The single autonomous loop is implemented, and it produced this cluster's one uncontested result: an overnight run finding hyperparameters he had missed after two decades of tuning that class of model by hand. What is unbuilt is the coordination around it — parallel auto-researchers, the untrusted-worker swarm, and the meta-layer that would tune the loop's own specification. And on what the labs actually run internally, he is inferring like everyone else: he left one, and says plainly that his judgment about what is coming will drift because of it.

So the honest summary is not "nobody has done it". It is that the loop works where the metric is clean, nobody has published the parallel version, and the conclusion that follows is still not "wait for them" but "build and train your own" — the gap is a skill issue on the operator's side.

Where this sits in the notebook

Adjacent to, but distinct from, the SLM thread: that thread asks which model to run and where; this one asks what to build around whatever model you run. They meet at evaluation — the eval harness that decides whether a fine-tune helped is the same machinery that decides whether an agent's sprint passed.

Open questions

• Where do these agents run? The owner's practical objection — a laptop that has to stay awake and travel with you is not a host for six-hour autonomous runs — is not addressed by any of the three posts, all of which quietly assume a machine that stays up. Partly overtaken since: Managed Agents and scheduled cloud runs are both hosts that aren't the laptop (see Keeping an agent running: goals, loops, hooks and schedules), though nobody has published what a six-hour run costs on either. Aleph Alpha's answer is the industrial one and is not agent-specific at all — CI triggers a workflow engine on Kubernetes, with durable execution and retries, on a cluster that stays up.
• What carries across sessions? Anthropic uses handoff artifacts and (in the older harness) context resets; OpenAI uses execution plans checked into the repo. Both are memory systems built out of files. Is there anything better than files yet? Karpathy says yes and points at OpenClaw without describing what it does — see The claw layer: an agent that persists when you close the lid, which is a pointer at an answer rather than one.
• What does a good evaluator cost, and when is it not worth it? Anthropic's answer — worth it when the task sits beyond what the model does reliably solo — is honest but only measurable after the fact.
• None of the three posts reports a case where the harness made the output worse. That absence is suspicious; a fair accounting would include the runs that were abandoned.

To read

The prior work these posts build on is listed in Harness design: prior work still to read — the articles themselves still need clipping.

The Karpathy transcript, and what it adds

The NoPriors interview that gave this notebook the phrase "skill issue" was ingested on 2026-07-27 — Skill Issue: Karpathy on code agents (NoPriors, 2026-03). It is the only source in the cluster that is not a lab write-up, and it argues from the operator's chair rather than from a project retrospective. Four things branch off it:

• Auto Research: taking the researcher out of the loop — the loop with the human removed, and the boundary that decides where it applies.
• The claw layer: an agent that persists when you close the lid — the persistent entity above a session, and the memory question above.
• Jaggedness: what RL optimises, and what stalls — why capability only advances where a reward can be computed, which turns out to be the mechanism behind several separate findings in this cluster.
• The customer is not the human anymore — the same conclusion as OpenAI's "what the agent can't see doesn't exist", reached from audience rather than from context budget.

The first source from the buyer's side

Everything above is written by people selling or building the tooling. The NoPriors interview with DoorDash's co-founders, ingested 2026-07-28, is the first source here from a company that just pays for it at scale, and it asks the question none of the posts do: what did the spend buy? Their model spend went up about 20× between January and June, and their answer is an internal benchmark over their own coding tasks scoring models and harnesses together — the unit this cluster argues for, being used in anger. See Benchmarking your own agent spend, which also carries the finding underneath it: on non-coding work the models crush a scrubbed version of the task and then underperform on the real data, and nobody there can yet say whether that is the harness or the model.

The benchmark's own methodology arrived separately — dashbench-measuring-a-code-review-agent, written by the team that built it. That page is where this cluster's evaluation thread stops being about building a judge and starts being about measuring one: why the metric everybody uses can only fill two cells of a confusion matrix, and what it costs to build labels that don't rest on any single fallible source.

The same interview's robotics half is the physical-world version of this page's argument — the capability lands, and the binding constraint moves to everything around it. It is The last hundred feet: building toward a use case, and it is worth reading next to the harness posts precisely because the domain is different and the shape is not.

The same argument, from a lab that trains models

Everything above is about getting an agent to write software. Model Training as Code: Aleph Alpha's Savanna (Aleph Alpha, May 2026, ingested 2026-08-10) reaches the same conclusion about a job with no agent in it at all: their entire training pipeline — pre-training, SFT, RL, evaluation — is imperative code in GitHub, CI is the entry point, and a full run is hermetic and launches with one click. The diagnosis they give for the manual alternative is this page's third idea in different words: the pipeline lived in the team's heads rather than in a durable artefact, so it could not be handed over, repeated, or reasoned about.

Three things it adds that no source here had. It is production infrastructure rather than an experiment, described by the people running it. It reports the concrete payoff of putting the recipe in version control — a large pre-training run stopped and resumed several times, each relaunch cheap enough that it did not have to be the same person twice. And it shows the organisational move that follows: once anyone can launch the whole pipeline, teams can own a model behaviour end to end instead of a stage of the pipeline.

It also inherits this cluster's standing weakness. No before/after numbers, no cost for the factory itself, and — for the fifth source running — not one reported case of the scaffolding making things worse.

The other side of the boundary

Everything above stops at a line this cluster has never crossed: these are loops you run to get work done, and the recursion goes as far as an overnight run beating a hand-tuned baseline and the loop that writes the loop. The Dwarkesh interview with Ryan Greenblatt, ingested 2026-08-13, is about what is on the other side — the loop as the mechanism that builds the next model, and what that does to the world. See The case for recursive self-improvement (Dwarkesh × Greenblatt, 2026-08).

It is worth reading from inside this cluster rather than as a separate subject, because the vocabulary carries over intact and the arguments land on pages that already exist:

• Containerising AI research is program.md with a curriculum: the actual environments, the three scales they run at, and the one part of the job — calling a large experiment you get few tries at — that stays unverifiable. It also contains the argument that algorithms rather than bought expert judgment have driven progress, which bears directly on how much the labels were worth.
• Training against your own monitor is what happens to an evaluator, a detector or a lint rule once it is used to select rather than to check. This is the page that most changes how to read the rest of the cluster, and it is why Linting as an agent guardrail now distinguishes enforcing a rule from selecting against one.
• Learning from deployment is the tax-agent loop with the model as the artefact instead of the product code, and the step it drops is the expensive one that made that loop trustworthy.
• Aligned to whom is the first thing here that is about policy: whose conventions the agent follows when the user's and the vendor's diverge.

The one place it contradicts this cluster outright is on transfer. Karpathy's jaggedness says capability advances only where a reward can be computed; Greenblatt says the transfer to hard-to-verify work looks fine in practice and cannot name a counter-example. Both are describing the same models. That disagreement is recorded on the jaggedness page rather than resolved, and the pattern this cluster keeps flagging — no published failures — has an analogue here: the optimistic case rests on the absence of a counter-example, which is the same kind of evidence.

The mechanics, one level down

All three posts are about what happens inside a turn. The other axis — what starts the next turn and what ends the run — has since become tooling you don't build: Keeping an agent running: goals, loops, hooks and schedules covers /goal, /loop, Stop hooks and scheduled runs, and the constraint that limits the cheapest of them (/goal's evaluator reads the transcript, it doesn't touch the artifact). Loop engineering: from writing prompts to writing loops is the framing that has grown up around it — the claim that the unit of work is now the loop rather than the prompt, and where that claim quietly skips the hard part.

The same argument, fourteen years earlier

Ingested 2026-08-14: [[sutskevers-list-notes|Heimann's Sutskever's List]], chapters 1 and 2. The book is a history of deep learning read through a reading list, and it is filed next to this cluster rather than away from it for one reason — its account of AlexNet is an assembly claim. ReLU, dropout, augmentation, momentum SGD, convolution and GPUs were all available; what was new was insisting they could be put together end to end at scale and then doing the engineering to prove it. The two items Heimann draws out of Sutskever's worldview, engineering pragmatism and minimum innovation for maximum results, are this cluster's own thesis about software written about model training in 2012.

Two things it lends this page rather than merely rhyming with it:

• Data or architecture: the control experiment nobody runs — the pre-2012 argument that the data, not the model, deserved the credit, and the control experiment that settled it. The reason it is here is that this cluster's central claim is a which-component claim of exactly that form, and it has no baseline: no published harness write-up reports the no-harness comparison across a task set, and the standing observation on this page that nobody reports the scaffolding making things worse is the same hole from the other side.
• Optimising for the benchmark — a field that spent a decade fitting its best models to the quirks of the dataset it was scored on, which is where the evaluation thread's scrubbed-versus-real problem was first diagnosed, under a different name and with nothing at stake commercially.

The safety half of chapter 1 goes to Too dangerous to release: the GPT-2 precedent.

The harness that is the codebase

Ingested 2026-09-01: a conference talk arguing that the spec-to-code loop — change the specification, recompile, never read the output — degrades a codebase on every pass, and that the reason is ordinary software entropy rather than anything about models. Its thesis is that a codebase which is hard to change is hard for an agent to change too, so good code got more valuable when the agents arrived, not less.

It is the first source here to put the engineering effort inside the artefact the model is editing rather than around it. Everything above is scaffolding — docs, linters, observability, evaluators, loops; this one is about the shape of the source itself, and the two are complements nobody in the cluster had named together. Two things branch off it:

• Deep modules, and designing the interface you delegate behind — module depth as an agent-legibility property, and the grey-box rule that follows from it: design the interface, delegate the implementation unread. That rule sorts work by position rather than by verifiability, which is a criterion this cluster did not have, and it is the most direct answer yet to the owner's question about delegating decisions rather than tasks.
• A ubiquitous language, addressed to the agent — a generated domain glossary handed to the model, whose reported effect is the one worth checking: shorter thinking, not just better plans.

It also inherits the cluster's standing weakness in mirror image. The harness posts never report the scaffolding making things worse; this talk never measures the degradation that is its central claim. Both are which-component claims without a control — Data or architecture: the control experiment nobody runs is why that keeps being worth flagging.

To build

Building a generator–evaluator harness: A practical implementation recipe turns the Anthropic post into a staged plan: what to build first, what to leave out, and what has changed on the platform since March.