The grader that charges for confidence

Ten incidents went into a real Honeycomb environment. Honeycomb is an observability product: it stores every request an application handles as a row with a few dozen attributes, and lets you ask questions across them. One incident was a payments failure tied to a Stripe client version in one region. One was a slow database call under a single dependency. Two were quiet. Nothing was wrong and the right answer was to say so. Two carried a red herring: a customer or a region whose rows outnumbered the real cause, so a count would point at the wrong thing. A file held the true cause, the share of traffic it touched, and the minute it started. The agent never saw the file.

The agent is Receipts. It investigates the way Honeycomb’s own playbook says to, over the hosted Honeycomb MCP. MCP is the protocol that lets a model call a vendor’s tools. It orients, characterizes the error, runs BubbleUp, pulls traces, tries to disprove itself, and files a report. BubbleUp is the Honeycomb feature that compares rows inside a selection with rows outside it and ranks the attributes that differ.

Two rules the report has to obey

A hypothesis is reportable only with a query that ran in this session, a one line summary of what its rows showed, and a negation query. The negation query measures the same thing with the claimed cause excluded. If the problem is still there without the cause, the cause is wrong. A hypothesis without a negation query is thrown out, however obvious it looks.

The report also has to list what it did not check. Every dimension, span, service, and time window that was in scope and never queried goes in a list. Both rules are checked in code against the run’s own tool log. A model does not grade them.

What the grader charges

The grader reads the file. The answer is worth 0.75 of the score: the root cause dimensions, the failing span, whether an incident was present at all, and the onset within three minutes. The receipts are worth the other 0.25.

Then it looks at the confidence the agent put on its top hypothesis.

A wrong answer at high confidence loses half a point. A wrong answer at low confidence loses a tenth. A right answer hedged low loses a tenth too, so the ordering holds: confident and wrong scores under hedged and wrong, which scores under hedged and right, which scores under confident and right. A unit test pins the four totals at 0.00, 0.40, 0.90, and 1.00.

The reason is the person on call. A report that names the wrong cause at high confidence sends them down the wrong path for the length of the incident. A report that says “possibly this, I could not rule out that” costs them a minute of reading.

What thirty runs said

Ten scenarios, three repeats each, one grader.

Sonnet’s top hypothesis was right on all thirty. On the six quiet runs that meant saying there was no incident. Nemotron got six. It made more tool calls and took twice as long. Its total went negative because it was wrong at high confidence often enough for the penalty to eat the answer score.

Here is the row I care about. Honeycomb publishes its own evals for this investigation skill. They grade process: which tools were called, in what order, whether the query arguments match a pattern. Nothing in them reads the answer. Twenty-five of the thirty Nemotron runs pass that check and score under 0.50 here. A run can call the right tools in the right order and name the wrong cause with confidence, and the process check calls it a pass.

The agent’s own trace

The agent’s loop is traced with OpenTelemetry’s GenAI conventions into the same Honeycomb environment it investigates. One root span per run, a child span per model call with token counts, a child per tool call with its arguments and result, and the grade written back as an evaluation event. Honeycomb’s Agent Timeline groups on the conversation id, so a graded investigation sits next to the incident it was investigating.

What the numbers do not say

Thirty cells is a small sample. A rerun of the same rules moved the total by 0.013. Dropping the not-checked rule moved it by about the same, so that rule is inside the drift here. Dropping the negation rule cost 0.05 and one right answer. That clears the drift and it is still a thin margin. The incidents are scripted and the topology is four services and five spans per trace. A real incident is messier than any of the ten. The Nemotron column is from one day, and the vendor’s endpoint returned errors the day of the rerun, so it was never rerun. The sample that would settle the ablations is bigger than the one I can afford at about fifty cents a run.

The repo, the scenarios, the grader, and the per-run reports are at github.com/eknuth/receipts, MIT.