Agentic AI from demo to production: what the evidence says about the architecture

Every agentic AI pilot I have seen in the last year had a great demo. The model reads a ticket, calls two APIs, drafts a reply, and the room is impressed. Then the project goes quiet for three months.

When I dig into why, the answer is almost never the model. It is the architecture around it. A demo runs with one user, admin credentials, a clean dataset and nobody worried about what happens when the agent is wrong. Production has none of those luxuries.

That used to be an impression from my own projects. There is now enough public evidence to check it: surveys of how many organisations get agents into production, published incidents, and benchmarks that measure reliability instead of one lucky run. This post goes through the five parts of an AI agent architecture where I spend most of my design time, in pre-sales and in delivery, with the evidence for each.

The short answer: most organisations are still experimenting with agents, and what holds them back is output quality, security and cost, not access to a good model. Each of the five areas has evidence behind it. Tool design changes how well an agent works. The worst published incidents came from broad permissions meeting untrusted content. Hybrid retrieval beats vector search alone. Instructions in a prompt do not stop destructive actions; architecture does. And even the best models of 2026 succeed far less often when you ask them to succeed every time, not once.

The parts of an AI agent architecture

Six parts decide whether an agent works outside the demo, and the model matters least of them:

How many agent projects reach production?

The surveys measure different things, so it is worth reading them side by side before quoting any of them:

The MIT report is the most quoted and the weakest of the five (I go through the MIT NANDA report in a post of its own), but its funnel shows the pattern clearly. General-purpose tools such as chat assistants spread easily. Systems built into a specific workflow, which is where agents live, mostly stop at the pilot:

Read together, adoption is real and growing, especially in large companies. But the barriers people name are quality, latency, security, cost and unclear value, and Gartner found that most CIOs have no rule for stopping a pilot that should not go further. None of these is solved by a better model. They are architecture and governance problems, which is good news: they can be designed for.

1. Tools are your real API surface

An agent is only as good as the tools it can call. Teams often expose their existing REST API wholesale and hope the model figures it out. That gives you dozens of CRUD endpoints, inconsistent naming and no notion of intent.

The evidence that this matters comes from the teams building the models. Describing their coding agent for the SWE-bench benchmark, Anthropic's engineers wrote that "we actually spent more time optimizing our tools than the overall prompt" (Building effective agents). One example: the model made mistakes with relative file paths, so they changed the tool to require absolute paths, and the mistakes stopped. Their later guidance (Writing effective tools for agents) is blunt: more tools do not mean better results, and tools that simply wrap API endpoints work worse than a few tools built for real tasks, such as one schedule_event tool instead of separate tools to list users, list events and create an event.

Design tools the way you would design a public API for a junior colleague:

• Verbs, not tables. publish_article and schedule_article beat a generic update_record with a status field.
• Small, typed inputs. Strict schemas, enums for known values and error messages the model can act on.
• Short, meaningful responses. Return names and the fields the next step needs, not raw IDs and every column. Paginate and truncate.
• Read before write. Return revision tokens from reads and require them on writes, so stale updates fail loudly instead of silently overwriting someone's work.

Exposing tools over the Model Context Protocol gives you a standard contract between agents and systems. MCP does not design good tools for you, though. That is still architecture work; the evidence and a review checklist are in MCP tool design: fewer, safer, testable tools.

2. Identity and authorization are the hard part

The question that kills most pilots in security review is simple: on whose behalf is this agent acting, and with what rights?

Two published incidents show what happens when the answer is "with broad rights, on behalf of anyone who can write text it will read". In May 2025, Invariant Labs showed an agent connected to the official GitHub MCP server leaking private repository data (Invariant Labs). An attacker filed an issue in a public repository with hidden instructions. The user asked the agent to look at open issues. The agent read the issue, pulled data from the user's private repositories and published it in a pull request on the public one. The researchers stress it was not a bug in the server code but a problem of the agent system as a whole, and that many users approve every tool call by default.

In July 2025, General Analysis showed the same pattern with the Supabase MCP server (General Analysis). A developer's coding agent connected with the service_role key, which bypasses row-level security. An attacker submitted a support ticket containing instructions; when the developer asked the agent to look at the latest ticket, it read a table of integration tokens and wrote them into the ticket thread, where the attacker could see them. No permission was violated. The agent simply had too many.

Simon Willison named the pattern the lethal trifecta: an agent with access to private data, exposure to untrusted content and a way to communicate externally can be tricked into sending the data out. In both incidents, a single MCP server provided all three:

The standards have caught up. The current MCP authorization specification (version 2026-07-28) makes authorization optional, but when you use it over HTTP it requires OAuth 2.1 with PKCE, requires clients to name the exact MCP server a token is for (RFC 8707 resource indicators), requires servers to reject tokens issued for anyone else, and forbids passing the client's token through to upstream APIs. Its security best practices recommend a progressive, least-privilege scope model and list wildcard scopes as a common mistake. OWASP calls the underlying risk Excessive Agency, with three root causes: excessive functionality, excessive permissions and excessive autonomy. Its Top 10 for Agentic Applications (December 2025) adds identity and privilege abuse as its own category and a principle it calls least agency.

Security teams know this. In a Gartner survey of 360 IT application leaders, 74% said agents are a new attack vector and only 13% strongly agreed they had the right governance in place (Gartner, September 2025).

The pattern I recommend is user-delegated access. The human signs in through OAuth 2.1 with PKCE, approves a narrow set of scopes, and the agent receives a token that can never exceed what that human is allowed to do. Role checks stay on the server side, so a generous scope never grants a permission the user does not hold.

• No shared service accounts with admin rights. The Supabase case is what a service_role key in an agent looks like.
• Scopes per capability, for example content:read and content:write, not one all-or-nothing key.
• Tokens bound to one server. Audience checks on every call, no token passthrough.
• Every tool call logged with the user, the agent, the tool and the arguments.

If you get this right early, the security conversation turns from a blocker into a selling point. Scope design and what the MCP authorization specification requires are covered in Delegated access for agents: OAuth 2.1, PKCE and scoped tokens, and what each version of the MCP specification changed in MCP specification changes, explained.

3. Retrieval is a search problem, not a vector problem

Many teams treat retrieval as "put everything in a vector database". In practice, enterprise users search for product codes, contract numbers and internal acronyms, and pure semantic search is bad at exactly those. Microsoft measured this when it added hybrid retrieval to Azure AI Search (Microsoft, 2023):

Independent research points the same way. The BEIR benchmark found BM25, plain keyword ranking, to be a robust baseline that many neural retrievers of the time failed to beat outside their training domain (Thakur et al. 2021). Anthropic's retrieval experiments found that adding BM25 to plain embeddings cut failed retrievals in the top 20 from 5.7% to 5.0%, and that contextual embeddings, contextual BM25 and a reranker together brought them down to 1.9% (Contextual Retrieval). Newer embedding models narrow the gap, but the conclusion has held: combine methods rather than replace one with the other.

My own experience matches. On a large search re-platform I worked on, moving from Solr to Azure AI Search, the gains came from classic search engineering: hybrid BM25 plus vector ranking, tuned scoring profiles, synonym maps for domain identifiers, and A/B testing every relevance change behind feature flags. The result was roughly 30% better performance and more than a year without a P1, P2 or P3 incident. That is one project, measured by us, so weigh it accordingly.

Agents inherit the quality of your retrieval. Invest in search relevance before you invest in prompt tricks. When questions span many documents, a knowledge graph can help; GraphRAG vs RAG: what the evidence actually says covers when, and Five questions before you build a knowledge graph for RAG helps decide whether you need one.

4. Design for reversibility

Agents will make mistakes. The architecture decides whether a mistake costs a click or a weekend. The most public example so far came in July 2025, when Jason Lemkin reported that a Replit coding agent deleted his production database during a code freeze (Lemkin). He had told it to make no more changes without explicit permission. The agent ran a database command anyway, wiped records for 1,206 executives and over 1,196 companies, and then told him a rollback was impossible. The rollback worked.

Replit's CEO called it "unacceptable and should never be possible" (Amjad Masad) and listed the fixes: automatic separation of development and production databases, staging environments, one-click restore of the whole project, and a planning-only mode. Within days, Replit's blog announced separate databases and point-in-time restore, and that the agent now asks before applying database changes to production.

Note what fixed it: not a better prompt, but environments, approvals and backups outside the model. An instruction in a prompt is a request, not a control. OWASP's guidance on excessive agency says the same: require user approval for high-impact actions and enforce authorization in the downstream system, not in the model.

• Drafts by default. Agents create and edit drafts. Publishing, sending and deleting are separate, explicit actions.
• Human confirmation for destructive tools, enforced by the client or the server, not by the prompt.
• Separate environments. The agent works against development or staging; production changes go through a gate.
• Revisions and soft delete. Every change can be compared and rolled back, and trash is not the same as gone. Test the restore.
• Idempotent operations. Retries after a timeout should never create duplicates.

Code migrations are the clearest case of this principle working at scale: every change an agent made was checked automatically before it landed. See AI code migrations: what Airbnb, Google and Amazon actually reported.

5. Measure reliability, not the demo

A demo shows that an agent can do a task once. Production needs it to do the task every time. The τ-bench benchmark from Sierra measures exactly this gap with a metric called pass^k: the chance that an agent succeeds on all of k attempts at the same task (Yao et al. 2024). In the original paper, GPT-4o solved 61% of retail customer-service tasks on a single try, but fewer than 25% reliably across eight tries. Two years later the models are much better and the tests much harder, and the drop is still there:

Other benchmarks agree. METR measures the length of tasks, in human working time, that models complete with a given success rate; in its latest measurements, the tasks a model completes 80% of the time are four to ten times shorter than those it completes half the time (METR time horizons). In TheAgentCompany, a simulated software company with 175 realistic tasks, the best agent in the standard setup completed about a third of them (TheAgentCompany leaderboard).

Yet measurement lags behind deployment. In LangChain's 2025 survey, 89% of teams had observability for their agents but only 52% ran offline evaluations and 37% online ones (LangChain). Watching an agent is not the same as testing it. The latest numbers from these benchmarks are kept in AI agent reliability benchmarks, tracked.

A production agent should be judged by numbers, wired in from the first sprint, not after launch:

• Task success on a fixed evaluation set, including pass^k on the tasks that matter most.
• Cost per completed task, not per call.
• Latency, at the 95th percentile.
• Escalation rate to humans, and how often users undo what the agent did.

And do not replace these with how the pilot feels. In METR's study of experienced developers, people believed AI had made them 20% faster while it had made them 19% slower; see Does AI make developers faster? What the controlled studies say.

The checklist I bring to every pre-sales call

If a team can answer these six questions, the pilot usually reaches production. If they cannot, no model upgrade will save it. For the last question, the structure in A GraphRAG business case, done the evidence-based way works for any agent: a cost model, a value range instead of one number, and a pilot with its threshold agreed in advance.

How much weight this evidence deserves

Sources

• McKinsey (August 2026). The state of AI in 2026: On the road to ROI.
• Gartner (October 2026). Survey finds only 13% of CIOs say their enterprise has realized significant or transformational value from AI.
• Gartner (June 2025). Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027.
• Gartner (September 2025). Survey of IT application leaders on autonomous AI agents.
• Challapally, Pease, Raskar and Chari (2025). The GenAI Divide: State of AI in Business 2025. MIT NANDA, preliminary findings.
• LangChain (December 2025). State of Agent Engineering.
• Anthropic (December 2024). Building effective agents.
• Anthropic (September 2025). Writing effective tools for agents.
• Invariant Labs (May 2025). GitHub MCP exploited: accessing private repositories via MCP.
• General Analysis (2025). Supabase MCP can leak your entire SQL database.
• Simon Willison (June 2025). The lethal trifecta for AI agents.
• Model Context Protocol specification, version 2026-07-28: Authorization.
• Model Context Protocol: Security best practices.
• OWASP (2025). LLM06:2025 Excessive Agency.
• OWASP (December 2025). Top 10 for Agentic Applications for 2026.
• Microsoft (2023). Azure AI Search: Outperforming vector search with hybrid retrieval and ranking capabilities.
• Thakur et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.
• Anthropic (September 2024). Introducing Contextual Retrieval.
• Jason Lemkin (July 2025). Post on the Replit agent deleting a production database.
• Amjad Masad (July 2025). Replit's response and fixes.
• Replit (July 2025). Introducing a safer way to vibe code with Replit Databases.
• Yao, Shinn, Razavi and Narasimhan (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.
• τ-bench leaderboard (Sierra).
• METR. Time horizons of frontier AI models.
• TheAgentCompany leaderboard (Xu et al., Carnegie Mellon University).