Tag: inference
Karpathy expects specialised models and does not see them arriving, and his explanation is that adjusting weights without losing capability is still an underdeveloped science while context windows just work. Plus where he puts the open-weight models — six to eight months behind, and better off there.
Why video generation is stuck at five-second clips — quadratic attention over 35,000 tokens for five seconds of 480p — and why the open/closed gap that has nearly closed for text is night-and-day for video. Plus the argument that text diffusion is being marketed as the wrong product.
Philip Kiely and Ali Taha on what actually happens between an open-weight checkpoint and a production endpoint — the four optimisations that stack to 2–4× on fixed hardware, why quantising more of a model can make it better, and the reliability failures that turn out to be a race condition in a kernel rather than anything about the weights.
Inference engineers now write training pipelines, RL rollouts bottleneck on inference speed, and Baseten reports running GPU kernels inside their inference engine that GLM-5.2 wrote for itself from its own profiling traces. Plus the argument that continual learning belongs in the KV cache rather than in the weights.
Lab notes on turning a spare Linux desktop with a 12 GB GPU into an always-on, OpenAI-compatible HTTP endpoint for open-weight small models — VRAM budgeting, a dated model shortlist, llama.cpp vs. Ollama vs. vLLM, systemd, and not exposing the thing naked to the internet.