The Reality Check Was Not About the Framework

The phrase at the top of this post is not mine. “The agentic reality check” is the title of the second chapter of Deloitte’s Tech Trends 2026, published on 10 December 2025.

The chapter carries the survey numbers: 38% of organizations piloting agents, 11% with agents in production, 42% still developing a strategy, and 35% with no strategy at all. Deloitte’s explanation for the forecast that more than 40% of agentic projects get cancelled by the end of 2027 is that organizations are “automating broken processes instead of redesigning operations”, and its instruction is to redesign rather than automate.

That is an organizational diagnosis. The version that circulates is technical, and it is sold that way. LangChain’s framework guide opens on it: “Your agent works in local testing. Then you ship it, and something subtle breaks.” CrewAI’s homepage ladders its product tiers the same way: “You’ve launched pilots. Now you need agents in production.” I have made this argument myself here, and parts of it still hold. But between 23 October and 9 November, five sets of measurements landed on the questions the technical version leaves open: what a framework costs on its own, what happens to a task when a tool fails, what happens to state when a container dies, where the token bill accumulates, and whether a failed run can be reproduced at all. None of that was available when the phrase was coined.

The 89% is a subtraction

Deloitte’s page states a share, not a failure rate: “Only 11% of organizations have agents in production, despite 38% piloting them.” Subtract 11 from 100 and you have 89, which now travels as a pilot-to-production failure rate, most visibly in sponsored content, where a September piece promoting an agency puts “the pilot-to-production failure rate for AI agents at 89%”.

The subtraction does not survive a careful look. Eleven percent of all organizations is not eleven percent of pilots. If the organizations in production are drawn from the ones piloting, the share of pilots that have not reached production is about 71%, because 11 divided by 38 is roughly 29. That 71 is my arithmetic on Deloitte’s two figures, not a Deloitte claim, and it rests on an assumption the summary does not confirm. If production organizations are not a subset of piloting ones, the two figures cannot be combined at all, and the honest conclusion is that neither one measures a failure rate.

What the survey did measure is where an organization sits: still developing a strategy, running a pilot, in production, or nowhere yet. A pilot that has not finished is not a pilot that failed. The 89% says otherwise, and it is the number that gets used to argue the technology does not work.

Where the money is not

AIMultiple ran 13 open-source frameworks against 5 tasks, 100 repetitions each, 6,500 runs in total, on Claude Haiku 5.5 at medium reasoning effort. Every request went through the same logging proxy, so model and reasoning settings were identical across frameworks, and so were the tool descriptions. The result: “The cost gap between the 12 frameworks in the chart is small: at most $0.47 per 1,000 runs.”

Time is no different. “98% to 99% of the time goes to waiting for the model.” The framework’s own code adds 0.04 to 0.14 seconds per run, less than 2% of the total.

The comparison the market spends its energy on is not the comparison that moves either number. The caveats are real: AIMultiple is a research and consulting firm publishing its own benchmark, the whole thing runs on one model, and the five tasks share one dataset. But a fifty-cent spread is small enough that no team choosing between two of these libraries is making a decision worth the argument.

The one thing a framework does decide

The benchmark’s Task 4 breaks a tool on purpose: a connection error, then a timeout, then a rate limit, then success on the fourth call. Every error message says “retry”. In twelve frameworks the model retried and got the answer on the fourth attempt in 97% to 100% of runs. In smolagents it “gave up on the tool after the second or third error in 96% of runs” and calculated the four payment methods itself, one at a time. That took 13.76 model calls and 41.07 seconds, and cost $9.96 per 1,000 runs “against $1.90-2.75 in the other frameworks”.

The same model produced both behaviours. The authors are explicit that the input was identical: “The error message was the same in every framework.” Their explanation for the divergence is a presentation one: smolagents “shows the error and earlier steps as plain text inside its own template”. They flag that mechanism as untested, and I would treat it as open. What that difference costs is not in doubt. One framework turned a transient fault into a rewrite, and the rewrite cost five times the cheapest retry.

Two details matter more than which framework it was. If temporary errors are retried inside the tool, the behaviour stops depending on the framework at all: “the model never sees them, and the behavior no longer depends on the framework.” A separate check with GPT-5.2, one run per framework, found 6 of the 13 giving up and taking the expensive route, so this is a model behaviour as well as a harness behaviour, and one run per cell estimates very little.

Scale is not a spare part

The MCP fault-injection study is the only peer-reviewed item in this set; it was accepted at NeurIPS 2026 as a poster. It built a fault profile from 57,240 issue and pull-request threads across 1,774 open-source MCP repositories, then ran a 1,350-cell experiment across two model sizes and three agent stacks. Under those faults the task-completion score falls by 60%, “from 8.08 to 3.23 out of 10”, and the 120B model “retains no larger share of its fault-free score under faults” than the 20B one, at p = 0.96. The authors’ reading is direct: the result “suggests that framework-level error handling matters more than model scale.”

Their scope is tighter than that abstract line. The fault-tolerance result is bounded to “the two gpt-oss backbones we test”, and the fault profile is deliberately permanent-heavy, so the 60% describes a constructed worst case rather than how often production tools actually fail. Both caveats are worth carrying. Neither changes the direction: a bigger model did not buy fault tolerance.

The container and the framework disagree

A UNC Wilmington and Oakland University group derived 14 “operational contracts” from LangGraph’s documentation and from the Docker, Compose and Kubernetes docs, then tested them against 433 deployed open-source agent systems. At least one contract applied to 75.5% of them. Among the systems that must keep state beyond the process, “37.4% lose it to a restart and 41.5% to a container replacement”.

Nobody wrote a bug to cause this. The two systems hold opposite assumptions, and the paper says so: “to the platform a container is disposable, while the framework persists conversations”. Developers do repair it. Of 122 repairs mined from 39,261 commits, pull requests and issues, 78% change application code, and “only two in three fully restore the violated guarantee.”

This is the number for the non-determinism concern, and the model is not in it. It is a question about where a conversation lives when the process holding it goes away, which is a deployment question with an ordinary answer. The percentages come from LangGraph deployments; CrewAI and the OpenAI Agents SDK keep state their own way and were not tested.

The bill is in the context

A Berkeley, Northwestern and Illinois preprint asks what the harness layer is for. Its framing, drawn from earlier studies, is that accuracy depends mostly on the model while “harness-side variances such as system prompts or tools have minimal impact”, and that those same choices move cost by up to 2.6 times on the same model and task. The 2.6 is a figure the paper cites from prior work. Its own contribution is a prototype that reports “cost reductions up to 35% across various real-world benchmarks on existing harnesses without compromising accuracy”.

The cost anatomy is the part a builder can use. Their breakdown: “reading existing context is the largest source of agent cost in real-world executions even with caching, taking up to 73% of the total cost”, and the agent uses “only 27%” of what its tools return, on average. Long sessions reach “10 to 100 times the cost of shorter sessions”.

I have to correct something I wrote here in October. In the post on self-evolving harnesses I said that in most production systems “the harness, not the model, is what caps how good the final agent gets”. The Berkeley authors observe that newer agentic models “are capable enough to find a solution regardless of the harness used”, and their cost argument rests on accuracy being mostly the model’s business. For a frontier model on a benchmark task, the harness buys cost rather than capability. My October sentence was written about hand-assembled harnesses and, read plainly, claims more than that. What I would defend is narrower: the harness caps quality when the model cannot solve the task on its own, and it moves the bill in every case.

A failed run is not reproducible

The last paper audits “all 71 unsuccessful executions in a fixed historical AppWorld corpus using 284 isolated fixed-code replay attempts”. Under the settings the runs declared, 42 of the 71 can be reconstructed. Resetting the random-number generator before each interaction admits 68, and the remaining gap “contracts to four under database/oracle checks”.

It also found “four materialized-state discrepancies despite matching outputs and verdicts”, including one where “an identical API call yields matching responses but different row effects”. The responses match and the rows written differ. That is the shape of a bug you cannot find by re-running, and it is why “reproduce it locally” is not a debugging method for agents unless replay was engineered in from the start.

The paper has two authors whose affiliation I could not check, because arXiv serves no HTML for it, and it does not estimate debugger accuracy or repair gains. Use it for the existence of hidden state, not as a failure rate.

What these measurements do not cover

Three of the four preprints are first versions with no review behind them, and the Berkeley one was posted on 9 November. The benchmark is published by a firm that sells research, on one model, with five tasks built on a single public dataset. The container percentages are LangGraph’s. The MCP faults are a stress profile. The Deloitte figures come from the hub page’s chapter summary rather than the report itself, and the chapter’s own URL does not resolve. I re-ran none of the experiments.

What I would change

Retry below the model. The benchmark’s own caveat is the instruction: put retries in the tool and the framework variable disappears. It is a change to the tool wrapper rather than to the orchestration, and it removes the one case where framework choice changed the bill by five times.

Write the state down outside the process, then test a container replacement rather than a restart. 41.5% is what happens when nobody tested the case the platform treats as routine.

Read less context. Spending 73% of the bill on re-reading what you already have is a larger line item than any orchestration library will ever be.

Keep the effects, not just the responses. A run that returns the right answer and writes the wrong row is the failure both the replay paper and the retry task circle.

Choose the framework on whether it retries, rather than on the features its landing page leads with. A month ago I would have told you the harness was the lever. The measurements put most of it under the model and around the deployment, and a smaller part, the part that turns a broken tool into a five-times bill, in the library itself.