Tag: fine-tuning
Karpathy expects specialised models and does not see them arriving, and his explanation is that adjusting weights without losing capability is still an underdeveloped science while context windows just work. Plus where he puts the open-weight models — six to eight months behind, and better off there.
A lab put its entire training pipeline — pre-training, SFT, RL, evaluation — into imperative code, with CI as the entry point and one-click hermetic runs. The first source in this notebook where the harness is built for training a model rather than for writing software, and the argument is organisational rather than technical.
Planning note for a 50-minute talk in Prague — the business case for specialized small models, then the training harness live. Includes the evidence check that killed the original opening slogan, the six conditions under which fine-tuning is the right call, and the caveat about borrowing a frontier lab's factory as a template.
Running notes on Manning's "LLM Customization and Fine-Tuning" (MEAP, Bahree & Tok) — chapter takeaways, questions, and links back into the rest of this notebook.