Notes on shipping LLM systems — Vitalii Bogachev https://talhayme.github.io/blog/ Field notes from running LLM products in production: evaluation gates, provider fallback, retrieval that refuses to guess, and the incidents that taught me each one. en 3,355 times a coding agent edited a test https://talhayme.github.io/blog/3355-agent-test-edits https://talhayme.github.io/blog/3355-agent-test-edits SWE-chat publishes real coding-agent sessions, most of them Claude Code, with the commits they produced. I pulled every commit where the agent changed a test and the code together, ranked them, ran seven end to end against pr-witness, and found the one thing the cross-run could not see. Sun, 11 Oct 2026 00:00:00 GMT What the trainer does to six tool-calling datasets https://talhayme.github.io/blog/six-tool-calling-datasets https://talhayme.github.io/blog/six-tool-calling-datasets The most-downloaded function-calling datasets, rendered through the model's own tokenizer and chat template. At max_seq_len=2048, 44% of the rows in hermes-function-calling-v1's two func_calling configs are cut before the answer. Plus two things met on the way: what a known Ollama bug costs (the model gets a Go struct dump instead of the tool schema), and a pre-fix template in Qwen's official GGUFs. Sat, 10 Oct 2026 00:00:00 GMT CI only ever runs the new tests against the new code https://talhayme.github.io/blog/ci-only-runs-new-tests https://talhayme.github.io/blog/ci-only-runs-new-tests An agent breaks a behaviour, edits the test to agree, and CI goes green. pr-witness runs the base branch's tests against the pull request's code, checks the description's claims against the real run, and signs the result. Built measuring-stick first; every number here was measured, including the ones that embarrassed me. Sat, 10 Oct 2026 00:00:00 GMT Your Claude Code skills cost 7,155 tokens on every request https://talhayme.github.io/blog/skills-cost-context https://talhayme.github.io/blog/skills-cost-context Every installed skill injects its description into the system prompt of every turn, used or not. I measured mine: 64 skills, 7,155 tokens per request, half from one source, and one skill installed twice. Here is how to measure yours. Fri, 09 Oct 2026 00:00:00 GMT What it actually takes to run an LLM product on your own https://talhayme.github.io/blog/shipping-llm-saas-solo https://talhayme.github.io/blog/shipping-llm-saas-solo A document-translation service on GPT-4 and Claude: the architecture, the reliability work nobody demos, and the engineering decisions that decide whether an AI product survives contact with paying customers. Thu, 08 Oct 2026 00:00:00 GMT