You spend a week tweaking the LLM somewhere in your stack. A new prompt, different retrieved context, a tool call you rewired. You ship the PR, and you have a quiet, nagging suspicion that none of it did anything.
That feeling has been bothering me for the better part of two years.
Normal software gives you ways to earn confidence: unit tests, type systems, static analysis, CI, observability. AI systems don’t fit that world. They’re software, but they’re also behavior, and behavior shifts with the prompt, the model, the retrieved context, the tools, the context, and plain randomness. You can’t enumerate every input the system will see, and you can’t responsibly ship on a hunch. So most teams ship on a hunch (or “vibes” as the kids say) anyway.
It usually looks like this: A PM runs a few examples, an engineer runs a few more, someone pastes in a query that looks like a real customer, the new version feels more polished, and it goes out. Sometimes that’s fine like if you’re bolting the latest model onto a low-stakes feature, ship it and move on. But the moment the system touches real users, money, support, compliance, or financial analysis, “feels better” stops being a release gate. The only question worth answering is whether the change improved the thing you actually care about, and a few hand-picked examples can’t tell you that.
What Benchrail does
Benchrail is a framework for building evals for your own LLM and agent systems. Not a universal score, and not a public leaderboard. (Those get gamed and overfit the second they become the target, which is a documented problem, not a hot take.) Your support agent, your logistics workflow, and your finance copilot don’t share a definition of “good,” so they shouldn’t share a benchmark.
Most eval tools make you hand-author every metric, then watch the set rot as the product moves. Benchrail works the other way. You declare what your system is supposed to do, what good looks like, and which failures actually hurt, and it derives the grading criteria for that specific context. The rubric a D2C brand needs isn’t the rubric a freight company needs, and a generic “accuracy” number misses both.
From there the loop is simple: derive the rubric, compile it into judges, run a baseline, run your candidate change, and compare. When something breaks in production, that failure becomes a permanent case in the benchmark. The benchmark is the thing you optimize against, and it lives in git as an artifact you can review like any other code.
Judges need calibration
The obvious shortcut is to ask an LLM whether the output is good. Sometimes that works well, but an LLM judge has no business gating a release until you’ve checked it.
Judges are biased in measurable ways, they reward verbosity, they’re sensitive to ordering, they’re inconsistent, and they drift when the underlying model changes. So Benchrail treats every judge as a candidate until it clears agreement with human labels. Pass the bar and it can gate releases. Fall short and it stays advisory. A bad eval is worse than no eval, because it hands you confidence you never earned.
The benchmark has to evolve
You can’t write the perfect benchmark up front. The first version is a hypothesis about what matters. Then reality shows up: a customer asks something strange, a tool call fails in a new way, a model returns an answer that’s plausible and wrong, an output that’s technically correct but useless in the workflow. Those failures shouldn’t die in a Slack thread. Each one becomes a regression case, a new scorer, or a reason to recalibrate a judge. Write evaluators for the failures you actually hit, not the ones you imagine.
That’s also where this earns its keep beyond catching regressions. Once you can measure quality you trust, you can find the places where a cheaper, faster model holds the line, and downgrade with evidence instead of crossing your fingers.
It’s an alpha
Real enough to use, early enough that you’ll hit sharp edges. The APIs will change, the docs will get better, and there are rough corners I haven’t found yet. The design is grounded in the eval research I trust (HELM, the Leaderboard Illusion work, the LLM-as-judge papers, the NIST risk framework), so it’s opinionated without being made up. It runs on top of Inspect AI instead of reinventing a runner.
bash
uvx --from benchrail-ai benchrail help
# or from source
git clone https://github.com/JeffBrines/benchrail.git
cd benchrail && uv sync && uv run benchrail helpIf you’ve ever shipped a change to an AI product and couldn’t honestly say whether it got better, I’d like you to try it. Break it, open an issue, send a PR, or tell me the framing is wrong. The goal is narrow: make AI iteration less vibes-based.
…and yes, if we get this really working it should help unlock recursive self improvement and cost optimization in ways we currently are only dreaming about. But that’s a whole other blog post.