Field notes from running LLM products in production: evaluation gates, provider fallback, retrieval that refuses to guess, and the incidents that taught me each one.
SWE-chat publishes real coding-agent sessions, most of them Claude Code, with the commits they produced. I pulled every commit where the agent changed a test and the code together, ranked them, ran seven end to end against pr-witness, and found the one thing the cross-run could not see.
The most-downloaded function-calling datasets, rendered through the model's own tokenizer and chat template. At max_seq_len=2048, 44% of the rows in hermes-function-calling-v1's two func_calling configs are cut before the answer. Plus two things met on the way: what a known Ollama bug costs (the model gets a Go struct dump instead of the tool schema), and a pre-fix template in Qwen's official GGUFs.
An agent breaks a behaviour, edits the test to agree, and CI goes green. pr-witness runs the base branch's tests against the pull request's code, checks the description's claims against the real run, and signs the result. Built measuring-stick first; every number here was measured, including the ones that embarrassed me.
Every installed skill injects its description into the system prompt of every turn, used or not. I measured mine: 64 skills, 7,155 tokens per request, half from one source, and one skill installed twice. Here is how to measure yours.
A document-translation service on GPT-4 and Claude: the architecture, the reliability work nobody demos, and the engineering decisions that decide whether an AI product survives contact with paying customers.