Top AI Models Struggle with Enterprise IT Tasks, Scoring Below 50% on New Benchmark

AI meets its match in enterprise IT

When it comes to diagnosing complex IT issues, even the most advanced AI models are still far from matching human experts. That's the key takeaway from ITBench-AA, a groundbreaking new benchmark developed by Artificial Analysis and IBM to evaluate AI performance on enterprise IT tasks.

What is ITBench-AA?

ITBench-AA is the first in a series of benchmarks designed to test AI models on real-world enterprise IT challenges. The initial focus is on Site Reliability Engineering (SRE) tasks, with plans to expand to Financial Operations (FinOps) and Chief Information Security Officer (CISO) tasks in the future.

The benchmark consists of 59 SRE tasks, including 40 public tasks and 19 brand new, held-out tasks. Each task presents a Kubernetes incident snapshot, complete with alerts, events, traces, metrics, logs, and application topology. The AI model's job is to identify the root cause of the incident by pinpointing the specific Kubernetes entities responsible.

How it works

1. The AI model is given access to a sandboxed file system containing incident data.
2. Using shell commands, the model investigates the incident through logs, metrics, traces, and topology information.
3. The model submits a list of Kubernetes entities it believes caused the incident.
4. The submission is compared against a ground-truth set of root causes provided by IBM.
5. Scoring uses average precision at full recall: models must identify all true root causes without including false positives.

Key findings

1. Even top models struggle: The best-performing model, Claude Opus 4.7, achieved only a 47% score. GPT-5.5 followed at 46%, and Qwen3.7 Max at 42%.

2. More investigation doesn't equal better results: Models that spent more time investigating (measured by the number of turns) often performed worse. For example, Gemini 3.1 Pro Preview averaged 83 turns per task but scored only 30%, while GPT-5.5 averaged 31 turns and scored 46%.

3. Open-source models show promise: Some open-weights models performed competitively with proprietary ones. GLM-5.1 (Reasoning) scored 40%, matching Gemini 3.5 Flash (high) at a lower cost per task.

4. Cost-effectiveness varies: While Claude Opus 4.7 led in performance, it was also the most expensive at $5.38 per task. In contrast, Gemma 4 31B (Reasoning) scored 37$0.14 per task.

Challenges for AI in enterprise IT

The low scores across all models highlight the complexity of enterprise IT tasks, particularly in areas like Site Reliability Engineering. Some key challenges include:

1. Contextual understanding: AI models must grasp the intricate relationships between different components in a Kubernetes environment.

2. Avoiding false positives: Successful models need to distinguish between true root causes and related but non-causal factors.

3. Efficient investigation: The benchmark rewards models that can quickly hone in on relevant information without getting sidetracked.

4. Structured output: Models must provide their diagnoses in a specific JSON format, requiring precise communication of findings.

Why it matters

ITBench-AA reveals a significant gap between current AI capabilities and the demands of enterprise IT tasks. This benchmark provides a clear target for improvement in AI models, potentially driving advancements that could revolutionize IT operations. As AI continues to evolve, closing this performance gap could lead to more efficient, reliable, and cost-effective IT management in enterprises worldwide.