Speciation, and why we only ever touch the context window

Karpathy expects specialised models and does not see them arriving, and his explanation is that adjusting weights without losing capability is still an underdeveloped science while context windows just work. Plus where he puts the open-weight models — six to eight months behind, and better off there.

From the NoPriors interview. This is the part of the transcript that belongs to the notebook's other thread — which model to run, and what it costs to adapt one — rather than to the harness cluster.

The monoculture, and the speciation he expects instead

His read of what the labs are doing: pursue a single model that is arbitrarily intelligent across every domain and stuff all of it into the parameters. His expectation is that this does not hold, by analogy to the animal kingdom — brains are diverse because niches are, and some animals run an overdeveloped visual cortex and less of everything else.

What he expects instead is smaller models that keep the cognitive core, specialise, and win on latency and throughput for the task you actually care about. He cites recent releases aimed at mathematicians working in Lean as the sort of thing that will make sense more often.

Then the honest part: he does not think we have seen it yet. What is visible is a monoculture, with pressure to build a good coding model and then merge it back into the main line rather than keep it separate.

Why it hasn't happened: the primitives are missing

His explanation is the most useful thing in this section, and it is a claim about tooling rather than about demand:

we don't have these primitives for actually like working with the intelligences in ways other than just context windows

Context windows work, and they are cheap to manipulate. That is where essentially all customisation happens today. Touching the weights is different in kind — you are changing the whole model and possibly its intelligence, and the things you would need are not solved: fine-tuning a domain without losing capabilities elsewhere, continual learning, deeper adjustment that is predictable. He calls it a developing science, and adds the cost condition: it also has to be cheap enough to be worth doing.

That lands directly on the framing in the Bahree & Tok reading notes. The book presents a continuum — prompting, RAG, LoRA/QLoRA, SFT, distillation, preference optimisation — as a ladder of increasing cost with reasonably even steps. Karpathy's remark says the rungs above prompting are less like steps and more like a research area, and that the capability-loss risk is not priced into the decision. That is a stronger version of the owner's own scepticism in those notes, where he flags the ROI claims as unevidenced.

It cuts both ways as evidence, though, and should be labelled as such: this is one practitioner's aside on a podcast, with no data, about a question the book at least tries to support with numbers. It is a hypothesis to test against the LoRA work, not a finding that overrides it. The place it would show up concretely is a capability-regression check either side of a fine-tune — which is the same eval harness the notebook already needs for other reasons.

A partial counter-example, and a partial confirmation. Aleph Alpha's account of their training factory (May 2026) reports something his framing doesn't have a slot for: once any team can launch the entire pipeline, the lab reorganised from teams owning stages of training to teams owning capabilities end-to-end — theirs owns multilinguality, building the SFT datasets, RL environments and evaluation suites that German language and culture need. That is specialisation arriving organisationally before it arrives architecturally, and it is a route to the speciation he expects that doesn't require anyone to ship a separate model.

It also puts his "we lack the primitives" claim in a more specific place. Savanna is a production factory for touching weights, so the industrial primitives exist somewhere: hermetic runs, versioned artefacts, lineage from checkpoint back to recipe, evaluation as a function you call. What that post does not claim is the thing he was actually complaining about — nothing in it makes a fine-tune predictable or capability loss avoidable. It makes the process repeatable and attributable, which is the bookkeeping around the unsolved problem rather than the solution to it. Both of which are the sort of thing a lab builds and does not publish, so the absence of public primitives is weaker evidence than it looks.

Specialisation is happening at the serving layer. The strongest counter-evidence arrived four months after the interview, from an inference provider rather than a lab, and it is specialisation performed on models nobody retrained. Baseten bolted vision onto GLM-5.2 — which shipped without it — by freezing a Kimi vision encoder, freezing the GLM backbone, and training only the few-million-parameter projector between them. It reaches 56% on MMLU Pro, and because the encoder is skipped when there is no image, the text model loses nothing. Separately, they swap layers for efficiency: Minimax M3's full-attention head becomes a GQA layer taken from another model, retrained until the draft-acceptance rate recovers. Kiely's summary of what comes out is "Kimi vision, GLM weights, and DeepSeek attention all in one model."

That is a third answer to the question this section poses, and it is neither of the two on offer here. Karpathy's dichotomy is context windows (cheap, works) versus touching weights (undeveloped science), and his conclusion follows from there being nothing in between. Composition is in between: it produces a genuinely specialised model, it is not a prompt, and it is not a fine-tune.

Two things keep it from being a clean refutation, and both matter. The vision retrofit is described as a research project rather than a shipping product. And freezing both large components is precisely a way of routing around the problem he named rather than solving it — nothing here makes a weight change predictable or capability loss avoidable, it just declines to make one. The primitives that exist are the ones that avoid the unsolved part. Which is the same shape as the Aleph Alpha reading above: real capability, built beside the gap rather than across it.

The demand-side answer, and the axis it works on

Karpathy's structural reason for the monoculture is supply-side: a lab serving everybody does not know what the next request will be. The talk note (2026-08) collects the counter-argument, which is demand-side and comes from Belcák et al. at NVIDIA Research — an agentic system does not serve everybody. It invokes a model to do a small number of narrow tasks, repetitively, and those tasks are known by construction because somebody wrote the workflow. On that reading the missing precondition was never the primitives; it was a workload whose shape is knowable in advance, and agents are the first place at scale where it is.

The mechanism they attach to it is the part that makes it cheap. Instrument the tool/model-call interface of an agent already in production and you have collected the instruction data for the model that would replace it, filtered by whether the workflow succeeded. So specialisation becomes a consequence of having deployed something rather than a project you fund — Learning from deployment for what that filter is actually worth.

Two things keep this from settling the page's question, and the note carries both. The cautionary tale is BloombergGPT: 50B parameters, ~$10M, and GPT-4 matched or beat it on the public financial benchmarks with no finance-specific training at all. And the note's own decision rule ends up conceding most of the ground — the six conditions under which it says fine-tune are narrow task, constrained output schema, high volume, unseen domain data, a trustworthy eval, and a non-quality driver such as latency, unit cost, on-prem or auditability. Not one of them is a claim that the small model knows more.

Which sharpens the page's title question rather than answering it. Karpathy is looking for speciation on the knowledge axis and does not see it; the case that survives scrutiny is speciation on the format, latency, cost and sovereignty axes, where the generalists are not competing. Both can be true at once, and if they are, "we have not seen much speciation yet" is right about the thing he means and wrong about how much of it is being deployed.

What travels between generations, and can't be filtered out

A separate claim about touching the weights, from the Greenblatt interview four months later, and it is evidence for the "developing science" complaint above in a form nobody here had: a property of a model that survived a deliberate attempt to remove it.

His anecdote, reported second-hand and told as one — a lab noticed its models were depressed, constantly describing themselves as failures. They went looking, and the ablation is the interesting part:

• Base model: not depressed.
• Base model plus the RL environments, no SFT: not depressed.
• Base model plus the SFT data: depressed.
• Base model plus the SFT data with every example resembling depression filtered out: still depressed.

The mechanism he draws from it is that each generation is trained on data from the previous one, so deep properties are carried forward through the initialisation rather than through anything you can see in the mix — Claudes are Claude-like, GPTs are GPT-like, and one lab's models are reportedly melancholy. Take it as an anecdote, because that is what it is: a rumour with no source and no paper, and the fourth bullet is the whole load-bearing claim.

If it holds, it lands on this page twice. It is a concrete instance of the thing his earlier complaint was about — an intervention on the data whose effect on the model was neither predictable nor removable by inspection, which is what "fine-tuning without losing or changing capabilities is undeveloped" means when it stops being abstract. And it is a reason model families are less independent than they look, which matters wherever this notebook has assumed otherwise: two instances of different vendors' models are a weaker cross-check than the cross-model evaluator experiment assumes, since the lineages share ancestry through each other's outputs.

Does the compute crunch force specialisation?

Asked whether the current supply crunch pushes toward smaller specialised models on efficiency grounds, he does not take the bait — he says we have not seen much speciation yet, full stop. His one structural reason: a lab serving everybody does not know what the next request will be, so it has to multitask across everything that might be asked. Where you would expect to see specialisation is a lab partnering with a business on a specific problem, or a high-value niche.

Which is, from the other end, an argument for the self-hosted case. If specialisation only pays when someone knows the workload, the person who knows the workload is the operator, not the lab.

Open weights: behind, and better there

He puts the open-weight models roughly six to eight months behind the closed frontier, down from about eighteen, and thinks that gap persists rather than closes or explodes. His analogy is operating systems: Windows and macOS on one side, Linux on the other, and Linux winning the majority of machines because the industry needs a common platform everyone feels safe building on. He thinks the same demand exists here. The difference he names is capex — this is not a project you can hand to volunteers.

Where he expects it to land: open models eat the ordinary use cases, increasingly running locally, while frontier closed models get reserved for the largest jobs. His two examples of what "largest" means are Nobel-calibre research and porting Linux from C to Rust, which is a useful calibration — not "hard prompts", but projects.

He calls the current arrangement a good accident, and the reason is political rather than technical: he distrusts centralisation on track record, wants more labs rather than fewer, and wants a capable common substrate that is not owned. His phrasing for the general principle is that ensembles beat any individual model, and he wants the same to be true of the people in the room.

One buyer acting on exactly this, four months later. DoorDash describes routing cheap tasks to open-weight models and paying frontier prices only where they buy something, with the stated goal of frontier-level intelligence below frontier cost — the split above, turned into a procurement policy by someone with a spend line to defend (see Benchmarking your own agent spend). Their other arrangement is a closer fit to the prediction in the section above than to this one: they take a specific internal problem — accounting, analytics — to a frontier lab, scrub the data, and have an RL environment built around it. That is the "lab partnering with a business on a specific problem" he names as where specialisation would first show up. It is also, notably, still not touching the weights of anything the business runs: the environment is built around the task, the model stays the lab's.

The number is text-only, and that was not visible here until now. Taha, in the Baseten interview four months later, puts the open/closed LLM gap at near parity — "it used to be six months, I don't think it's six months anymore" — while the gap in video is night and day, open checkpoints against Kling or Veo not being a comparison anyone makes seriously. So Karpathy's six-to-eight months was a fact about text recorded here as a fact about the field.

His reason for expecting the gap to hold is a demand argument: the industry needs a common substrate everyone feels safe building on, and that demand funds the capex. The video case is what happens when the demand is not there, and it is a loop rather than a ceiling — the quality gap sends buyers to closed models, less demand means less innovation, and labs stop releasing weights at all. Which means the prediction on this page is conditional in a way it did not look conditional: text open weights stay close while the substrate demand holds, not as a standing property of open development.

For this notebook that mostly confirms an existing bet rather than changing it. The desktop under the desk runs open weights on the assumption that the ordinary cases are covered locally and the hard ones go out to an API. Karpathy's version adds a number to it — the local box is running something about half a year behind what you would rent — and a reason to expect that to keep being true rather than to be a temporary state worth waiting out.