Agents Are Easy to Demonstrate and Hard to Operate

I have watched a lot of agent demos this year. Most of them are genuinely impressive. A small team wires a model to a few tools, points it at a real workflow, and in an afternoon it books the meeting, files the ticket, or drafts the analysis. The room lights up. Someone asks when it ships.

Here is the problem. The demo answers the question of whether an agent can do the work. It says nothing about whether you can operate it. Those are different questions, and the second one is where the actual cost lives. I spent years running infrastructure and reliability teams at AppDynamics through its acquisition by Cisco, and later across more than 600 applications at Fannie Mae. I have seen this movie before. It is the same gap that separated a service running on a laptop from a service running in production, except now the software is non-deterministic.

The demo takes an afternoon. Operating agents takes an operating model. Executives who fund the first and not the second are going to relearn every lesson SRE teams learned the hard way, at a worse time and a higher price.

Traditional observability tells you what happened to a request: where it went, how long it took, whether it returned an error. That was hard enough to build. Agents add a layer underneath it. The interesting question is no longer whether the request succeeded. It is why the agent chose tool A over tool B, what context it was holding when it decided, and whether that decision chain was reasonable.

If you cannot reconstruct an agent's decision path after the fact, you cannot debug it, you cannot audit it, and you cannot improve it. Logging the prompt and the response is not enough. You need traces of reasoning steps, tool calls, retrieved context, and intermediate outputs, stored and searchable, the same way we learned to trace requests across microservices.

This is not a nice-to-have. When something goes wrong, and it will, the first question from legal, risk, or your customer will be some version of why did it do that. You need to be able to answer.

Deterministic software has unit tests. You assert the output and you are done. Agents do not give you that. The same input can produce different outputs, and many of those outputs are acceptable. Testing becomes evaluation: scored sample sets, regression suites of real scenarios, graded rubrics, and thresholds that gate a release the way a failing build once did.

Most teams I talk to have nothing like this. They tweak a prompt, spot-check a few outputs, and ship. That is the equivalent of deploying to production because it worked on my machine. It was unacceptable in 2010 and it is unacceptable now.

Cost and latency need the same discipline. An agent that retries, reflects, and fans out across tools can quietly turn one user action into dozens of model calls. Without explicit budgets, per task and per workflow, you will discover your unit economics in the monthly invoice. Set the budget first, then engineer inside it. That is how capacity planning has always worked.

An agent with write access is an actor in your systems, and it deserves the same scrutiny as a new hire with production credentials. Least privilege, scoped tokens, approval gates on irreversible actions, and a clear answer to the blast radius question: if this agent goes wrong for an hour, what is the worst it can touch? If nobody can answer that, the agent is not ready, no matter how good the demo was.

Incident response changes too. For twenty-five years the failure signal was an error code, a spike on a dashboard, a page at 2 a.m. Agents fail differently. The failure mode is often a plausible wrong answer, delivered confidently, with no alarm attached. Nothing is down. Nothing is throwing errors. The system is simply, quietly wrong.

That means detection has to be built deliberately: sampling outputs for review, monitoring drift in behavior, giving users a fast escalation path, and running postmortems on bad answers with the same rigor we ran them on outages. A wrong answer that moved money or advice is an incident. Treat it like one.

In classical operations, you version binaries. You know exactly what changed between releases because the diff tells you. With agents, behavior shifts when the model version changes, when a prompt is edited, when a tool description is reworded, or when the retrieval corpus is updated. Any one of those is a release, whether or not anyone called it that.

So treat them as releases. Pin model versions. Put prompts and tool definitions in source control. Run the evaluation suite on every change. Keep the ability to roll back not just code but behavior. Teams that skip this end up with an agent that got worse last Tuesday and no way to know why.

None of this is exotic. It is the reliability playbook, extended to software that reasons. But it does not come free with the model API, and it cannot be bolted on by the feature team in their spare time. It needs ownership, tooling, on-call coverage, and a budget line, the same way SRE did.

My advice to executives is simple. When a team shows you an agent demo, applaud, then ask the operational questions. Who watches its decisions? What gates a release? What is the blast radius? What happens when it is confidently wrong? Fund the answers before you fund the rollout. The demo was the easy part. It always is.