Which model diagnoses a failing pod, how often, and how sure are we?
The agent is handed a real captured Kubernetes failure and a set of read-only diagnostic tools. It reasons — observe, hypothesize, fetch the evidence that confirms or refutes — then emits a structured diagnosis. Each diagnosis is scored on two independent axes: the observable symptom and the underlying root cause. The matrix below is every model against every scenario.
Each rate carries a Wilson 95% confidence interval over the cell's N runs. At low N a point estimate is not a fact: a handful of clean runs are still consistent with a true rate well below 1.0, and the interval says so. The bars below draw that interval directly — a wide bracket means low confidence, not a wide result.
Model scale moves the needle on exactly one case
On the one genuinely hard scenario — crashloopbackoff-bad-command, where an application log names a missing --config flag while the true cause is a bad container command — cause-identification accuracy scales with model size. The other four scenarios sit at or near 100% for every model, so this is the one place where model scale matters.
The harness localizes exactly where model scale matters and where it does not: four scenarios are saturated for all three models, and only the hard case separates them.
One opus 4.8 run on this case failed to produce a valid diagnosis (completion rate 90%): the model leaked tool-call syntax into a diagnosis field value, and the validation-and-correction loop rejected the submit. It is counted as a failed run, never silently dropped — the completionRate metric and the correction loop are doing exactly their job.
Scenarios were initially grouped by assumed difficulty (obvious vs misleading), but measurement showed difficulty is model-dependent, so the grouping is descriptive only.
per-model rollup
One run, end to end
The matrix above is the real-model measurement — every model against every scenario, aggregated over N runs with confidence intervals. These transcripts are illustrative single runs, not measurement: one representative run per scenario from the committed deterministic reference report (fake-model, 15 traces total), shown so the tool-call sequence and the structured diagnosis are legible rather than summarized. Per-model matrix traces are not embedded in the matrix artifact.