The LLM Evaluation Framework.

In Evaluations are the productI argued that the eval harness is the most honest spec an AI team will ever write, and that it should be built before the first production user arrives. That essay is about why. This one is about what and how: once the harness exists and is producing numbers, which numbers actually decide whether a change goes live, and how do you keep any one of them from quietly eating the others.

The harness and the gate are not the same thing, and conflating them is the most common way an eval program stalls. The harness is the machinery that produces signals, the unit evals, task evals, and rubric evals that run against a candidate change. The gate is the policy that reads those signals and returns a verdict: go, hold, or roll back. A team can have a beautiful harness and no gate, in which case the numbers get admired in a dashboard and ignored at release time. The framework below is about the gate.

The four dimensions

Every AI feature that resolves real user intent can be measured on the same four axes. They are not the only numbers a team tracks, but they are the ones a release decision turns on, because each one maps to a promise the product is making to somebody.

Containment

Containment is the share of interactions the system resolves without handing off to a human. It is the closest thing AI support has to a headline metric, and the easiest to misread. High containment is only good if the contained conversations actually ended well. A bot that refuses to escalate will post excellent containment and a trail of unhappy users who gave up. So containment is never read alone, it is read against satisfaction, which is exactly the discipline the rest of this framework enforces. When I redesigned theintent taxonomy from 50,000+ conversations, the point of the work was not to raise containment in the abstract. It was to raise it on the intents where automated resolution was genuinely the better outcome, and to route the rest to a human faster.

Latency

Latency is time to resolution, not time to first token. Users do not experience your p95 response time, they experience how long the whole thing took to get them what they came for. This distinction matters more in orchestrated systems than in single-shot chat, because the layers that make a system capable, retrieval, tool calls, multi-agent handoffs, are the same layers that add wall-clock time. In themanual-versus-orchestrated case study, a request that should have taken under two minutes took twenty-four, and almost none of that was model latency, it was humans manually bridging systems that an orchestration layer should have bridged. Latency is a design metric as much as an infrastructure one.

Satisfaction

Satisfaction is the quality dimension, and it is where the harness earns its credibility or loses it. Some of it is directly measured, a thumbs rating, a post-interaction CSAT, a reopened ticket. Most of it has to be inferred from the conversation itself, which is the job of the rubric eval: a written criterion, scored by a human or by an audited LLM-as-judge. The judge is a probabilistic component with its own biases, so it only counts when the rubric is strict and the judge is calibrated against human graders on a regular cadence. Treat the judge's score as ground truth and satisfaction becomes the dimension that lies to you most confidently.

Cost

Cost is dollars per resolved interaction, including the tokens you spend on the conversations that escalate anyway. It is the dimension teams discover last and regret ignoring, because it is the one that quietly decides whether the feature survives contact with a finance review. Cost is also the dimension most tempting to optimise in isolation, swap to a cheaper model, trim the context budget, drop a verification step, and watch the number fall. It falls because you moved the failure somewhere the cost metric cannot see it, usually into containment or satisfaction.

Why a single number is a trap

The reason these four travel together is that every one of them can be improved by spending one of the others. The relationships are not subtle. You can raise containment by suppressing escalation, and pay for it in satisfaction. You can cut latency by skipping retrieval or verification, and pay for it in satisfaction or containment. You can cut cost with a smaller model, and pay for it in containment. You can raise satisfaction by escalating everything to a human, and pay for it in containment and cost at once. A team that gates on one number is not measuring quality, it is selecting which corner to hide the regression in.

Wiring the gate

A working gate has three moving parts, and none of them is exotic.

A baseline. Before a gate can say whether a change is better or worse, it needs the current system's four numbers, measured on the same eval set the candidate will run against. Most teams skip this and end up arguing about whether a regression is real. The baseline is what turns that argument into arithmetic.

Per-dimension thresholds. Each dimension gets a line it must hold, set per surface rather than globally. A billing flow and a browse-help flow do not deserve the same containment bar, because the cost of a wrong automated answer is not the same. Thresholds are negotiated between product, support, and engineering, and writing them down is itself the most clarifying part of the exercise, it forces a team to say out loud how much latency it will trade for how much containment.

A regression budget. Demanding that every dimension improve on every change is how teams learn to ignore the gate. The honest policy, the one I argued for inEvaluations are the product, is that regressions are allowed when they are visible, explained, and traded for a named improvement elsewhere. The gate is a negotiation surface for trade-offs, not a pass-fail wall. A change that drops cost by a fifth and satisfaction by a single rubric point, on a low-stakes surface, may be exactly the trade the team wants, as long as someone decided that on purpose.

What feeds the dimensions

The four dimensions are only as trustworthy as the eval set underneath them, and the eval set is built from the same three evaluation types the harness already runs. Unit evals protect containment, a quietly broken intent route is a containment regression you want to catch before a customer does. Task evals measure latency and containment together, because they score whole trajectories, not single turns. Rubric evals carry satisfaction, with the calibration discipline that keeps the judge honest. Cost is the one dimension that comes straight from production telemetry rather than the eval set, which is why it is easy to forget to gate on at all.

Retrieval-heavy systems add a wrinkle worth naming. When I built theRAG pipeline evaluation for a production support system, faithfulness to the retrieved source was a satisfaction sub-rubric that could not be inferred from containment at all, a confidently wrong answer contains the conversation and fails the user. If your system retrieves, faithfulness belongs inside the satisfaction dimension explicitly, not as an afterthought. The broader point is one I made inContext Engineering: you cannot tell whether a context budget is earning its tokens until you can measure the outcome it is supposed to improve.

The Monday version

If you already have a harness and no gate, the first move takes an afternoon. Pull the current system's four numbers from your existing eval set and last week's production telemetry, that is your baseline. For each surface that matters, write down one threshold per dimension, even if the first draft is a guess, a written guess is reviewable in a way an unstated assumption is not. Then add one rule to your release process: a change that drops any dimension below its line does not merge until someone names the trade and signs off. That rule is the entire gate. It is crude, it will be wrong in places, and it is already more release discipline than most AI features have ever had.

From there it compounds the way the harness does. Every incident sharpens a threshold. Every surprising trade-off teaches the team something about how its four numbers actually move together. The industry has sprinted into orchestration, Gartner clocked a 1,445% jump in multi-agent inquiries in a little over a year,^[1] and most of those systems will launch on vibes and a single flattering metric. The teams that gate on the whole vector instead will look, in two years, like the teams who took testing seriously a decade before everyone else, early, right, and quietly responsible for the fact that the thing keeps working.