AI agent reliability benchmarks, tracked

A benchmark is a standard test that every AI model takes under the same conditions, so results can be compared. Most benchmarks report how often a model succeeds at a task once. That is what a demo shows. Production needs more: the agent has to succeed every time, on tasks of realistic length, in work that looks like a real job.

This tracker follows four public benchmarks that measure exactly that, explains each in plain language, and keeps their latest results. Terms are explained in the glossary at the end. I update it when the official leaderboards change.

As of 27 October 2026: on the newest customer-service test, banking support with a library of about 700 documents to search, the best model solves 55% of tasks on one try and 35% on four tries in a row. Four times out of five, the best measured model finishes software tasks that take a person about three hours, and every model fails more often as tasks get longer. In a simulated software company, the best checked agent completes about a third of realistic office tasks. The newest models, Claude Opus 5.5, GPT-6 Astra, GPT-6.1 Sol and Gemini 4 Argon, are so far measured independently only on AutomationBench: they complete about nine in ten objectives of business workflows, but broken rules cost them 16 to 24 points of score.

1. τ-bench: does the agent succeed every time?

What it measures. Sierra's τ-bench puts an agent in a simulated customer-service job, for example handling returns for an online shop or changing flight bookings, with tools and company policies it has to follow (Yao et al. 2024). A simulated customer talks to the agent. Each task is run several times, and pass^k is the share of tasks the agent gets right on all k tries, not just one. The newest version, τ³-Banking, adds a knowledge base of about 700 documents the agent has to search before it can answer, which makes it much harder; it is the only version with results for models released after February 2026 (leaderboard).

Artificial Analysis also runs τ³-Banking independently, five times per task, but reports only the single-try result, with its own setup, so its numbers do not compare with Sierra's: GPT-6 Astra solved 41.4% of tasks; Claude Opus 5.5 has no result there yet (Artificial Analysis).

What it means. The best model goes from 55% on one try to 35% when all four tries must succeed. If customers repeat the same kind of request every day, the four-try number is the one to plan with.

The older τ²-bench domains, an online shop and airline bookings, are easier, and show the same drop from one try to four in a row:

2. METR: how long a task can an agent handle?

What it measures. METR gives agents software and research tasks of very different length, timed by how long a skilled person needs for each, from a few seconds to many hours (METR time horizons; method in Kwa et al.). A model's time horizon is the length of task it completes with a given success rate. The gap between the 50% and the 80% horizon shows how fast reliability falls as tasks get longer.

What it means. Agents can now attempt surprisingly long tasks, but if you need them to succeed reliably, plan with the 80% figure, which is four to ten times shorter in these measurements. The newest model METR measured was released in April 2026, and METR says the page is no longer actively updated; models released since then have no time horizon yet.

3. TheAgentCompany: can an agent do a real job?

What it measures. TheAgentCompany is a simulated small software company with 175 tasks across engineering, project management, finance and HR, done through the same tools employees use: a code repository, chat, documents (Xu et al.). A task counts only if every step of it is done right. These are results checked by the authors, in the standard setup (leaderboard):

What it means. Mixed office work, with many steps and several tools, is still mostly beyond agents working alone. Narrow, well-defined tasks with checks are where they deliver today.

4. AutomationBench: does the agent follow the rules?

What it measures. AutomationBench, built by Zapier, gives agents 657 business tasks across sales, marketing, operations, support, finance and HR, done through simulated apps such as Gmail, Salesforce, Slack, Jira and Zendesk (Shepard and Salimans). Each task has objectives, and guardrails: conditions that must still hold at the end, such as other customers' records left untouched. Artificial Analysis runs it independently on new models, once per task, and scores each task as the share of objectives completed, set to zero if the agent broke a guardrail (Artificial Analysis; methodology).

What it means. The newest models do most of the work in a business workflow, and the gap that remains is mostly rule-breaking rather than unfinished steps. For an agent that touches customer or financial records, the guardrail number matters more than the completion number, and it is what your own tests should check. Each task runs once here, so this measures rule-following, not repeatability. Scoped access limits what a broken rule can do: see how AI agents should authenticate and get delegated access.

How to use these numbers

• Plan with the strict number. Use the four-try result or the 80% horizon, not the best single run.
• Treat benchmarks as a ceiling for expectations. These are simulations; your tools, data and users will differ.
• Measure on your own tasks. Build a small set of real tasks and run each several times, as described in Agentic AI from demo to production.
• Fix the tools before blaming the model. Accuracy falls as the tool list grows; how to keep tools few, safe and testable is in MCP tool design best practices.

Glossary

Changelog

• 27 October 2026. First version: τ³-Banking (models released up to July 2026) and τ²-bench, METR Time Horizon 1.1 (models released up to April 2026), TheAgentCompany checked results, and AutomationBench from Artificial Analysis for the models released in September 2026 (Claude Opus 5.5, GPT-6 Astra, GPT-6.1 Sol, Gemini 4 Argon), with a glossary.

Sources

• Yao, Shinn, Razavi and Narasimhan (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045.
• τ-bench leaderboard (Sierra).
• METR. Time horizons of frontier AI models.
• Kwa et al. (2025, revised 2026). Measuring AI Ability to Complete Long Software Tasks. arXiv:2503.14499.
• Xu et al. (2024, revised 2025). TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv:2412.14161.
• TheAgentCompany leaderboard.
• Shepard and Salimans (2026). AutomationBench. arXiv:2604.18934.
• Artificial Analysis. AutomationBench-AA.
• Artificial Analysis. τ³-Banking.
• Artificial Analysis. Intelligence benchmarking methodology.