Enterprise agents are still below the replacement line. IBM Research's Hugging Face post says ITBench-AA puts frontier models through Site Reliability Engineering tasks built from Kubernetes incidents, logs, metrics, traces, alerts, events, and topology. The headline number is inconvenient: all frontier models scored below 50 percent. [1] Sunday's paper said ITBench put agent reliability below marketing claims; Monday's point is that the sub-50 result remains the useful ceiling for labor-replacement language.

The benchmark does not ask whether a model can sound like an engineer. It asks whether an agent can identify the root-cause Kubernetes entities in a realistic incident. Claude Opus 4.7 led at about 47 percent, GPT-5.5 followed at 46 percent, and Qwen3.7 Max reached 42 percent in the IBM account. [1]

Artificial Analysis supplies the scoring discipline. It describes 59 Kubernetes incident tasks, three repeats per task, a sandboxed environment, and an average-precision-at-full-recall score. If the agent misses a true root cause, the repeat scores zero. If it finds the true causes while naming extra wrong entities, precision falls. [2]

That design is cruel in the useful way. It denies the agent credit for sounding busy while leaving the real fault untouched. It also denies full credit for naming every possible suspect and hoping the operator recognizes the right one. The benchmark therefore measures a habit production teams actually need: constrained judgment under incident pressure, not a transcript long enough to look like work. [2]

That scoring rule is why the benchmark matters. Many demos reward fluency, tool use, or long investigation traces. ITBench rewards a narrower operational habit: find the real thing without flooding the ticket with plausible false positives. In production SRE work, an overconfident wrong entity is not a harmless flourish. It sends people, playbooks, and downtime toward the wrong system.

This does not mean agents are useless. A 47 percent score on difficult incident work is evidence of capability. It also says the buyer should not confuse partial diagnostic help with autonomous replacement. The more honest sales pitch is assistance, triage, second opinion, or constrained workflow. The less honest pitch is that an SRE team can be swapped for an agent because the demo completed one polished path.

The IBM write-up still matters because it puts frontier systems in a setting closer to real operations than puzzle benchmarks or sales videos. Logs, metrics, traces and topology are the materials SREs actually use. But the same realism makes the sub-50 result harder to wave away. If the test is closer to the job, the miss rate is closer to the business risk. [1]

The divergence is between marketing time and operations time. Marketing rounds up from what looks impressive. Operations rounds down from what breaks at 3 a.m. ITBench keeps the argument on the operations side. Until scores, false-positive behavior, and escalation rules improve, enterprise agents remain tools to govern, not workers to assume.

-- DAVID CHEN, Beijing