Most AI Pilots Are Not Designed to Survive Production

For most of my career I have been on the receiving end of other people's pilots. At Fannie Mae, my organization ran platform engineering and reliability across more than 600 applications, which meant that when something succeeded in a lab somewhere in the company, it eventually arrived at our door with a request to make it real. That vantage point teaches you something the pilot team almost never sees: the distance between a thing that works and a thing that runs.

The industry talks about pilot failure as if the technology disappointed. That is rarely what I observed. The model usually performed about as well as advertised. The pilot died in the handoff. There was no dramatic moment, no postmortem, no decision to cancel. It simply did not get the next check, did not find an owner, did not make a roadmap, and six months later nobody could tell you when it stopped.

Here is the pattern underneath it. Most pilots never reach production because they were never designed to. They were designed to answer whether the technology works. That is a fine question. It is also not the question that gets anything into production.

Watch how a use case gets chosen. Teams look for clean data, a friendly internal audience, low regulatory exposure, and a workflow that nothing critical depends on. Those criteria produce a smooth pilot. They also describe, almost exactly, a use case that cannot justify a production investment. Low stakes means low value, and when it is time to fund integration, security review, and support, there is nothing on the other side of the ledger to point at.

The second version of the same mistake is choosing something genuinely valuable and then quietly removing every hard part. Sample data instead of the operational feed. A human reviewing every output. No write path into the system of record. The pilot proves a capability under conditions production will never provide, and the team celebrates a result that does not transfer.

The better instinct is to choose the pilot for what it will teach you about the difficult parts. If the riskiest element is writing back into a system of record under audit, put that in scope early, even at small volume. You want the pilot to fail on the things that would have killed the rollout, while failing is still cheap.

Pilots measure performance on representative cases. Production is defined by the tail: the malformed document, the customer who is an exception to three policies at once, the record where two upstream systems disagree and both are technically right.

Anyone who has run operations knows the last stretch of edge cases consumes most of the effort. AI makes that worse, because the failures do not announce themselves. When a pilot reports that it handled ninety percent of cases correctly, the interesting question is never the ninety. It is what happens to the rest. Who catches them. What the catch costs per case. Whether the misses are randomly distributed or concentrated in your highest value customers, which is the outcome nobody checks and the one that actually matters.

That tail is where your operating cost lives. If the pilot did not measure it, you have not priced the thing you are about to buy.

Pilots are usually paid for out of innovation money. It is small, discretionary, and forgiving. Production is paid for out of a business unit's run budget, where every dollar already has an owner and a commitment attached to it.

Crossing that line asks someone to accept a permanent operating expense, headcount to support it, an on-call rotation, an audit obligation, and vendor contracts that renew whether or not the champion is still in the role. That is a far harder yes than the one that started the pilot, and it is being asked of a different person, often for the first time, at the exact moment everyone assumed the hard part was over.

I have watched pilots with strong results die here, not from skepticism but from arithmetic. The receiving organization had already committed its year. Nobody was against it. There was simply no line to put it on. The fix is unglamorous and effective: identify the receiving budget and the receiving owner before the pilot starts, not after it succeeds.

A pilot proves that a machine can produce useful output. Production requires that people change what they do with it. Those are different achievements, and only one of them shows up in the demo.

If the analyst, the underwriter, or the support agent still does the work the way they always did and then checks the system's answer, you have not removed effort. You have added a step and a cost. The value only appears when the workflow itself is redesigned: what the human now owns, what they no longer do, what quality standard applies, how exceptions escalate, and how performance is measured afterward.

That is not a training session. It is role definition, incentive design, sometimes headcount, and in regulated environments a compliance conversation. It belongs to a business leader, not to the team that built the model. When it does not happen, adoption metrics look acceptable, the business result never materializes, and everyone concludes the technology underdelivered. The technology did what it was asked. Nobody changed the job around it.

Most pilots I review have a start date and a demo date. They do not have exit criteria. Without a written definition of what result, at what cost, under which controls, triggers promotion, the default outcome is extension, because extension is the only path that requires no one to decide anything.

Write the criteria before the work begins. Three measurable outcomes. The named owner who will operate it. The budget it lands on. And a decommission date if the criteria are not met. A pilot program with a full and fast graveyard is healthy. One where everything is perpetually in flight is not running experiments, it is deferring decisions.

The reframe I would offer executives is small and it changes everything downstream. A pilot is not an experiment about the model. It is a rehearsal for operations. Rehearsal means real data, real users, real controls, real integration where you can manage the risk, and the actual people who will run it afterward sitting in the room while it runs.

Pilots built that way cost more and take longer, and you will run fewer of them. That is the point. The number that matters was never how many pilots you launched this year. It is how many are still running twelve months later, in production, with a named owner, a budget line, and a business result somebody is willing to be measured on.

Most enterprise pilots do not fail. They end. That is worse in a way, because ending looks like nothing happened, and something did. You spent a year and a budget confirming what you already suspected, and you learned almost nothing about whether your organization could carry it.