Securing AI and Defending Against It Are Two Different Problems

"AI security" gets used for two problems that have almost nothing in common. One is protecting systems that use a model. The other is defending against attackers who use a model. The threat actors differ, the controls differ, the owners differ, and increasingly so do the budgets. Teams that conflate them buy the wrong thing and then wonder why it did not help.

The rest of this series hangs on this distinction, so it is worth being precise about it first.

The problem

A platform team stands up an internal assistant. It reads the ticket queue, searches the wiki, and can open a change request. Somebody asks the obvious question: is this secure? Procurement returns with three vendors. One sells a gateway that inspects prompts and model outputs. One sells email security that flags AI-generated messages. One sells a governance platform that maps your models to a regulatory framework. All three call themselves AI security, and only one of them is about the assistant.

The same confusion runs the other way. A finance team wires $180,000 to an attacker who cloned a supplier's writing style well enough that nobody questioned the request. The post-incident review recommends "AI security controls". The controls that would have helped are payment verification and phishing-resistant authentication, neither of which has anything to do with the models the company is deploying.

Why it's different with AI

The two problems only look alike because the same three letters appear in both. Pull them apart and almost nothing overlaps.

Securing AI is an application and infrastructure problem. The asset is a system you built or run. The adversary is whoever can reach its inputs, which increasingly means anyone who can get text in front of it: a web page it summarises, a document it ingests, a tool result it reads. The OWASP Top 10 for LLM Applications 2026 is the reference list here, and its entries describe things that go wrong inside your own stack: prompt injection at LLM01, sensitive information disclosure at LLM02, excessive agency at LLM03, unbounded consumption at LLM06 [1]. These are design and permission problems, and you fix them by changing what your system can do.

AI as the weapon is an operations problem. The asset is your staff and your money. The adversary is outside, and the model belongs to them. Nothing in your architecture changes the attacker's capability. What changed is the economics on their side, because producing a convincing, individually tailored lure stopped being expensive.

The FBI's Internet Crime Complaint Center recorded that shift with unusual clarity in 2025. Phishing and spoofing complaints fell slightly, from 193,407 to 191,561. Reported losses from the same category rose from $70,013,036 to $215,843,126 [2]. That is roughly one percent fewer complaints and roughly three times the money. Business email compromise accounted for $3,046,598,558 across 24,768 complaints in the same report, an average near $123,000 per incident [2].

Volume stopped being a proxy for risk. If your reporting still counts messages blocked, it is measuring the thing that got cheaper and missing the thing that got worse.

Note what the second problem does not require. No model of yours is involved. You could deploy nothing, ban every assistant in the building, and your exposure would be unchanged.

What actually works

Stop treating this as one budget line and route each half to the group that can act on it.

For securing AI, the controls are the ordinary ones applied to a new component: scope the permissions the model's tools carry, isolate where inference runs, put quotas on what it can consume, and keep an audit trail of what it did. Most of these are deterministic and enforceable in infrastructure. A few, like classifying a prompt as hostile, are probabilistic and belong behind the deterministic ones.

For AI-enabled attacks, the controls are identity and process. Phishing-resistant authentication removes the credential as a target. Out-of-band verification for payment and account changes removes the payment as a target. Both are deterministic, and both work regardless of how good the attacker's prose is. Detection that scores an inbound message for intent or stylistic deviation is worth having and belongs last, because it is the only layer that degrades silently as the attacker improves. The two lists share almost no items, which is exactly why they need separate owners.

Ownership follows the same split, and naming it is usually the most productive hour available. Systems built with models belong to application security, with infrastructure owning where they run and what they can reach. Inbound AI-assisted attacks belong to security operations and whoever runs mail. Governance and reporting sit across both without owning either. In most organisations these are different teams, different tools and different on-call rotas, which is why a single "AI security" line item tends to fund one and leave the other starved.

There is one honest overlap. Both halves are made worse by the same organisational habit of approving something because it looks plausible and arrived through a familiar channel. On the first problem that is a tool call the model was allowed to make. On the second it is a payment a person was asked to approve. The control in both cases is a verification step that does not depend on the thing looking wrong.

What doesn't work

The first mistake is buying one and believing you covered both. A gateway inspecting your assistant's prompts does nothing about a supplier invoice fraud, and mail filtering does nothing about an agent with a write-scoped database credential. Both products may be good, but neither is general.

Another is treating awareness training as the answer to the second problem. Training built on spotting bad grammar, generic greetings and suspicious links teaches markers that a model removes for free. Keeping it because it is the control you already have does not mean it works.

Assigning both to whoever ran the last AI project fails too. The people building with models and the people defending the mailbox are usually different teams with different tools and different on-call rotas. Handing both problems to one of them guarantees that only half gets attention.

So does waiting for the frameworks to merge. They will not merge, because they describe different things. The OWASP list catalogues risks in systems you build, while threat intelligence about AI-assisted fraud describes what is being done to you. Both are useful and neither substitutes for the other.

Where to start

• Name the owner for each half, in writing: application security and infrastructure for the systems you build with models, security operations for inbound AI-assisted attacks. If both names are the same person, that is the finding.
• Ask any vendor which of the two they address. The answer is usually clear and immediate, and a pitch that resists the question is telling you something.
• Check your phishing metric. If it counts volume, add a loss-per-incident measure beside it. The IC3 divergence [2] is the argument for why.
• Inventory what your models can reach, which matters more than what they can say. Credentials, network egress, and write scopes are the part of the first problem that infrastructure can actually fix, and it is usually the part nobody has written down.
• Verify payments out of band, with a process that does not depend on anyone noticing that a message reads oddly.

The rest of this series follows the split. Some articles are about protecting what you build and run, and some are about what arrives in your inbox. Knowing which one you are reading, and which one you have a budget for, is most of the work.

Sources

1. OWASP Top 10 for LLM Applications 2026, OWASP GenAI Security Project, published 3 August 2026: https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/
2. 2025 Internet Crime Report, FBI Internet Crime Complaint Center: https://www.ic3.gov/AnnualReport/Reports/2025_IC3Report.pdf