Your AI Platform Needs a Threat Model Before It Needs a Roadmap

I engineered infrastructure for a security response that touched 190 million PayPal users. A password reset at that scale is not an IT ticket. It is a system that has to survive its own success: a synchronized surge of traffic, a retry storm waiting to happen, and a support organization that will drown if the engineering underneath it is not built to absorb the wave. That work taught me something that has stayed with me ever since. Security incidents at scale are not primarily security problems. They are infrastructure problems wearing a security label, and if you have not engineered for them in advance, you will be improvising in public.

I think about that every time I sit in a room where a team is excited to ship an AI feature and nobody has asked what happens when it is attacked. Not if. When.

Traditional application security has a mature playbook: authentication, authorization, injection attacks, secrets management. Most enterprises are reasonably good at this by now. AI systems inherit every one of those risks and add a new category on top, and that category has almost nothing to do with the vulnerabilities security teams spent the last decade getting good at.

A model that reads untrusted text, a document, an email, a web page, a support ticket, and then acts on it, is executing instructions from outside your trust boundary by design. That is what prompt injection is: the model doing exactly what it is supposed to do, follow instructions in its context, except the instructions came from an attacker instead of a user. Traditional input validation does not catch this, because the input is not malformed. It is just persuasive.

Data exfiltration through a helpful assistant. A support agent with access to a customer database and a chat interface is, from a threat model perspective, a query engine with a natural-language front end and often weaker access logging than the database it sits on top of. If that assistant can be socially engineered into revealing more than the requester should see, you have built a very articulate data leak.

Tool and plugin supply chain risk. Agentic systems are only as trustworthy as the tools they call. A third-party plugin, an MCP server, a retrieval index built from scraped content, all of these are dependencies, and enterprises that would never run an unreviewed open-source package in production are wiring unreviewed tools directly into systems with write access. That is a supply chain problem, and most organizations have not extended their supply chain security program to cover it.

Poisoned context at scale. If your model retrieves from a knowledge base that anyone can write to, comments, tickets, uploaded documents, an attacker does not need to breach your perimeter. They need to write one convincing paragraph and wait for it to be retrieved. This is the quiet one. Nothing alerts. The system just starts behaving a little differently, and by the time anyone notices, the bad content has been in the corpus and influencing answers for weeks.

In regulated environments like the one I worked in at Fannie Mae, security review is built into change control. That process was designed for systems that change on a release cadence and can be reviewed as a fixed artifact. AI systems do not hold still. The prompt gets tweaked, the retrieval corpus gets updated, a new tool gets wired in, and each of those is a change to the attack surface even though none of them looks like a code deployment.

A checklist run once before launch tells you the system was safe on the day someone looked at it. It tells you nothing about the system six weeks later, after three prompt edits and a new plugin. Security has to be continuous for the same reason evaluation has to be continuous: the system's behavior is not fixed, so a point-in-time review is measuring something that will have changed by the time the report is filed.

Treat every AI system that touches untrusted input or has write access as a new class of production dependency, and threat model it before the roadmap gets funded, not after the incident. Ask the same questions I asked engineering the PayPal response: what is the blast radius if this is exploited, what does the retry and escalation pattern look like under attack, and can the system detect it is being abused faster than the abuse can spread.

Concretely: scope what every tool and data source the model can reach, and assume anything it can read, an attacker can eventually write. Log and monitor model inputs and outputs with the same seriousness as authentication logs, because right now, at most companies, they get less scrutiny than a login page. Red team the system with adversarial prompts before launch and on a recurring cadence after, the same way penetration testing is not a one-time event. And put security engineers in the room when the architecture is decided, not in the room when the postmortem is written.

None of this is exotic. It is the same discipline that kept a 190-million-user password reset from becoming a second incident: plan for the attack, not just the feature.

Every executive I talk to wants to move fast on AI, and they should. But speed without a threat model is not speed, it is exposure with a head start. The organizations that get hurt this year will not be the ones that moved slowly. They will be the ones that shipped a capable, well-demoed system connected to real data and real actions, and never once asked what an attacker would do with it.

Build the threat model first. The roadmap will still be there in a week. The vulnerability will not wait that long.