Self-improving agents from production feedback (OpenAI × Thrive, 2026-05)
Notes on the Tax AI post — how practitioner corrections in production become structured findings, then targeted evals, then bounded engineering tasks a coding agent can close. The clearest published example of an eval-driven improvement loop with a named metric that moved.
Source: Building self-improving tax agents with Codex,
Aravind Srinivasan and Samay Shamdasani (Thrive Holdings) with Arthur Fernandes Araujo and
John de Wasseige (OpenAI), 27 May 2026. Archived under
sources/2026-07/building-self-improving-tax-agents-with-codex/.
Six months of work with Crete, a network of 30+ accounting firms, on preparing 1040 and 1041 returns. 7,000 returns through the pilot season. The claim worth engaging with isn't the time saving — it's that the system got measurably better while running, without an engineer hand-diagnosing each failure.
The metric
They score returns by share of fields correct, and report the fraction of returns crossing 75%, 90% and 100%:
| ≥75% fields correct | ≥90% | 100% | |
|---|---|---|---|
| Week 0 | 25.1% | 1.3% | 0.01% |
| Week 3 | 79.3% | 38.3% | 6.1% |
| Week 6 | 86.2% | 59.8% | 9.5% |
| Mid-year target | 98% | 90% | 50% |
Reported alongside: about a third of practitioner time saved, up to 97% accuracy, ~50% more throughput, and one accountant going from 180 hours of tax prep in a season to 15. The curve matters more than any single number, and it steepened while the task got harder — early weeks were W-2s and 1099s, later weeks K-1s and multi-source schedules.
Choosing thresholds rather than a mean is the quiet good decision here: what a practitioner cares about is how much follow-up a return needs, and that's a step function, not an average.
The loop
- Practitioners steer. They're doing the work anyway; their corrections are the signal. Crucially, a difference between predicted and filed value is not automatically a bug — it can be an extraction miss, a mapping gap, unsupported product behaviour, a value carried forward from last year's return, practitioner preference, or ordinary workflow noise. Practitioners helped separate those classes. That distinction is the same one DoorDash draws between was accepted and was real, and both teams reached it the same way: by discovering that the human in the loop is a useful signal and a bad label.
- The product manufactures evidence. Not just inputs and outputs: the whole path. Documents organised, split and classified; fields extracted with citations back to the source; values mapped into the tax engine; corrections captured as structured data recording what the system proposed, what the practitioner changed, and what was filed.
- Findings become evals. Field-level review rows (expected, predicted, is-this- actionable) are grouped so recurring product failures separate from noise — "we keep missing fair-rental-days", "we confuse two rental properties in one package" — and only a reviewed, repeated pattern is promoted to an eval target.
- The agent gets a hill to climb. With a targeted eval set already built, the coding agent investigates the pipeline (schema? source selection? mapper? or is the grader wrong?), fixes, reruns the targeted eval plus regression suites, and opens a PR. Ambiguous or unsafe-to-automate cases route back to engineers instead of being forced through.
The task environment
The part I'd steal outright. A bounded workspace per finding, splitting writable from read-only: a git worktree on its own branch holding the scoped slice of product code the agent may change, the targeted and regression evals that define success, and skills/docs describing how to run the task — against read-only production context (the trace, the source documents, the prediction, the filed return, the field docs). The agent can investigate the evidence without mutating it, and success is defined before it starts.
That "success is defined before it starts" is the same move as the sprint contract in Anthropic's harness, arrived at from a different direction. Both are ways of making the definition of done external to the agent doing the work — see Generator–evaluator loops.
Scope discipline
Worth noting what they did not automate. The loop runs on one bounded layer: extraction and mapping of source documents into tax workflows. Engineers still own architecture, product decisions and shipping. Rental properties alone took about six weeks and substantial engineering oversight to reach 90% precision and recall — after which the abstractions, review artifacts and eval conventions made the next schedules cheaper. So the self-improvement is real but bounded, and the first domain is expensive.
Connections
The prerequisite is unglamorous and expensive: you need production traces rich enough to localise a failure, and a domain expert willing to label ambiguity. Without both, the loop degenerates into an agent optimising against its own guesses.
This is also the industrial version of a line in the fine-tuning reading notes — that production logs are the highest-value eval set a team has. Here that's not an aphorism: the logs are the product surface, the corrections are labels, and the grader is code someone maintains (and occasionally has to fix, since "the grader is wrong" is listed as a legitimate root cause).
The open question for a small setup: this works because thousands of practitioners generate labelled corrections for free as a by-product of their job. What's the equivalent signal when the only user is you?
The same loop with the model as the artefact
This page is the notebook's cleanest example of an improvement loop that works, and Greenblatt describes the same architecture with one substitution: the thing being improved is the model, not the product code. Production behaviour becomes a training environment — a task the AI did badly on, rebuilt as an environment matching it exactly, with the rubric written from what the engineer who asked for it wanted — or environments closely inspired by production traffic, or training on the traffic itself, or all three at once.
Which makes step one of the loop above the important step, and it is the one the substitution drops. Here, a difference between predicted and filed was not a bug by default, and practitioners were needed to sort extraction misses from mapping gaps from preference before anything became an eval target. That work is what a thumbs-up does not do. A satisfied user is a label about whether the output looked right to somebody who did not check it, and folding that back into training reinforces whatever passed rather than whatever was correct — which is the mechanism, not a risk. The distinction this page already draws between was accepted and was real turns out to be the load-bearing part of the whole design.
The other thing worth noting is what it does to the open question below. Asked what the equivalent signal is when the only user is you, this page's two answers have been "practitioners generate labels for free" and "you buy them". Greenblatt's argument implies a third that is worse rather than better: you can take the signal without paying for the labelling, and then the loop is running on satisfaction instead of ground truth. Cheaper, faster, and no longer the thing this page is recommending.
One answer from the other extreme of scale: you buy them. DoorDash launched a product called Tasks, where people in its delivery fleet collect data points to train world models — the labelling workforce as a line item rather than a by-product. Which restates the prerequisite rather than removing it: if the corrections aren't free, someone is paying per label, and that is a budget a one-person setup does not have. See The last hundred feet: building toward a use case for what makes DoorDash's version of the data worth paying for.
Linked from
- Agentic engineering: the work moves to the harnessThe emerging discipline around long-running coding agents — designing the scaffolding, feedback loops and environments that let an agent do reliable work, rather than writing the code yourself. Entry point for the harness-design cluster in this notebook.
- Containerising AI researchWhat it would concretely take to train a model to do AI research — the environments, the three scales they run at, and the one part of the job Greenblatt thinks stays unverifiable. Plus the argument that algorithms rather than bought human judgment have been driving progress, which is what makes the whole thing work.
- DoorDash on agentic commerce and DOT (NoPriors)Notes on the NoPriors interview with DoorDash's two co-founders — a natural-language ordering surface that half the time sends people to a merchant they have never used, a delivery robot built to a use case rather than to a capability, and an internal benchmark built to work out what a 20× rise in model spend actually bought.
- Inference engineering as a discipline (Latent Space × Baseten, 2026-08)Philip Kiely and Ali Taha on what actually happens between an open-weight checkpoint and a production endpoint — the four optimisations that stack to 2–4× on fixed hardware, why quantising more of a model can make it better, and the reliability failures that turn out to be a race condition in a kernel rather than anything about the weights.
- Learning from deploymentProduction traffic is becoming training data — tasks the model did badly on, turned into environments that match them exactly, with the rubric built from what the human actually wanted. Which is the loop this notebook already admires, run with the model as the artefact, and it changes where the reinforcement comes from.
- The last hundred feet: building toward a use caseDoorDash's argument for why they had to build their own robot — the field builds a capability and then hunts for a problem — plus the edge cases nobody writes down at a desk, and the one incumbent data advantage in the interview that is actually load-bearing.
- Training and inference are mergingInference engineers now write training pipelines, RL rollouts bottleneck on inference speed, and Baseten reports running GPU kernels inside their inference engine that GLM-5.2 wrote for itself from its own profiling traces. Plus the argument that continual learning belongs in the KV cache rather than in the weights.