Vitalii Bogachev

CI only ever runs the new tests against the new code. pr-witness runs the base branch's tests against the pull request's code, shows each rewritten test with its old and new assertion, checks the PR description's claims ("412 passed") against the real run, and signs the result with GitHub attestations. Evaluated on commits from real Claude Code sessions with the tool frozen and the labels committed first: the cross-run agreed with the label in all 11 runs it completed; the overall verdict in 7 of 12, every miss traced to its cause and fixed in the next release.

Regression checks for small fine-tuned models on tool calling: a byte-level diff of training vs runtime prompts (it measured what a known Ollama bug costs: the tool schema reaches the model as a Go struct dump), a dataset linter with the model's own tokenizer, and tool-call evals base vs tuned vs quantised. Six public datasets surveyed.

Golden sets, groundedness and hallucination metrics, absolute thresholds and per-case regression against a baseline. Exits non-zero so CI blocks a change that makes quality worse — and CI asserts the gate itself rejects a known-bad run.

Strict JSON schemas so the model calls correctly first time, errors written for the model to act on rather than stack traces, and a calculator sandboxed against code execution through four independent layers.

Calibrated confidence floors, hybrid dense + lexical retrieval, and citations carrying source and heading. Its real weaknesses are pinned as tests so the README cannot quietly become untrue.