From Model to Reality
Three things I got wrong taking small models from working-on-my-machine to running-as-a-service: a PVC that deadlocked against its own pod, an eval where my agent lost to the model it wraps, and a serving benchmark that was measuring a command-line flag.
Training is the part everyone shows you. I've built the small end of this twice — a trainer that writes model.json to a volume, an inference service that mounts the same volume and answers GET /predict?x=. Both times the model worked inside an afternoon. Both times the pipeline around it stayed broken for considerably longer, and never because the maths was wrong.
The pipeline that wouldn't schedule
The first failure was a PVC stuck on Pending.
That isn't a bug. local-path storage classes ship with volumeBindingMode: WaitForFirstConsumer, which delays binding and provisioning until a Pod using the claim is created so the volume lands somewhere the pod can run. Sitting Pending indefinitely is the intended behaviour, which makes the claim's status the one status that can't answer your question.
The deadlock arrives when the consumer can't schedule either. The volume is node-local, so the trainer CronJob and the inference Deployment both need pinning to the same node with a nodeSelector, or training writes a model to a disk the serving pod can't see. Pin them and the storage decision and the scheduling decision become the same decision. Whichever half is wrong the symptom is identical — pod Pending, claim Pending, each waiting on the other — and the reason sits in kubectl describe on the pod's scheduler events, nowhere near the claim.
There's a sharper version that isn't mine: the local-path provisioner can bind a PV to a node before the pod is scheduled, nailing the workload to that node for the life of the claim — including when that node lacks the memory to run it and three others have plenty.
I haven't fixed the consequence either. Pinning leaves the serving path with no failover: lose the node, predictions stop until I move things by hand. Distributed storage is the answer and it's a bigger build than the lab, so the single point of failure is documented rather than accidental.
The eval that measured the wrong thing
The second failure was worse, because it looked like a result.
I built a 14-case eval harness for a local coding agent I'd written — real tasks, hard network block, Apple-silicon hardware. The agent scored 78.6%. The raw model it wraps, called directly with nothing around it, scored 92.9%.
The instinct is to go and fix the agent. The right move was to convict the eval. Every case was single-turn: one prompt, one generation, score the output. Single-turn work never exercises the agent loop — no tool call, no reading a file, no reacting to a failed command and trying again — so the wrapper can only add tokens between the model and the answer. Overhead is what I measured.
That trap is documented. τ-bench was built because earlier tool-use benchmarks scored isolated calls, and single-turn estimation overstates dialogue ability; it adds a user simulator, policies the agent has to obey, and stateful scenarios where one action constrains the next. None of that survives compression into a single turn.
The suite is being rebuilt around multi-turn infrastructure tasks. The 78.6% stays in the repo.
The benchmark that was measuring a flag
Third one, July 2026. Two local serving stacks on the same Apple-silicon box: mlx-lm sustained 187 tok/s at 8-way concurrency where vllm-mlx degraded to 91.
The gap is real; my first version of it wasn't. vllm-mlx defaults to a simple mode — one request at a time, no batching overhead — and continuous batching is opt-in behind --continuous-batching. Serve it the default way, throw concurrent requests at it, and you don't get slow numbers, you get 503s. I was comparing one engine's concurrency path against another's absence of one.
So the number characterises a configuration. The flag separating those two readings is one line in a doc that nothing in the failure output points you towards.
What carries across
Version the artefact so "which model is that, and can I rebuild it" has a one-command answer — when serving misbehaves, the first thing to rule out is that the file on disk isn't what you think. Keep the failed runs. Write the constraint that forced the pinning next to the manifest that does it, so nobody deletes the nodeSelector as clutter.
What I don't have is drift monitoring, and the reason matters: this pipeline has no ground truth arriving after the fact. Nothing tells me a prediction was wrong an hour later, and drift charts without a feedback signal are a dashboard nobody can act on.
Current state: the eval rebuild targets multi-turn tasks, the agent loop's measured overhead over the raw model is about 15% wall-clock, and the serving benchmark re-runs unchanged on bigger hardware when I get it.