Zero Trust Against AI Social Engineering: Order Your Controls

Zero trust is usually explained in terms of networks. Against AI-assisted social engineering the network is irrelevant, because nothing is being breached and someone is being asked. The surface is identity and process, and the principle still holds: verify explicitly, assume compromise.

What matters is the order. There are three layers, and the sequence is the design decision.

The problem

An accounts payable clerk receives a message from a supplier they have dealt with for years. The writing matches, and it references the last invoice number correctly. It explains a bank change with a plausible reason and a mild deadline. There is no link and no attachment.

Everything a person could reasonably check comes back fine. The anomaly is the request itself, since the message is clean. And "does this request look legitimate" is a question a human under time pressure will answer yes to, because it does.

The organisation had AI-assisted email scanning. It scored the message as low risk, because by every property it could measure it was.

Why it's different with AI

The three layers below are not new, but their relative reliability changed.

Detection that reasons about a message's content used to be a reasonable primary defence, because attacker output carried artefacts: awkward phrasing, template structure, reused infrastructure. Those artefacts were a byproduct of working at scale, and they are gone.

That degradation is invisible. A control that blocks 95% of what it sees does not announce when the remaining 5% becomes the part that matters. Deterministic controls do not have that property: a FIDO credential either completes the login or does not, and a callback to a number in your records either reaches the real supplier or does not.

So the ordering reflects which controls you can still trust when the attacker improves.

The three layers, in order

Layer 1: verify the identity (deterministic)

Phishing-resistant authentication removes the credential as a target. CISA's guidance is direct about what qualifies: FIDO/WebAuthn is the only widely available phishing-resistant authentication, because the protocol binds the credential to the legitimate site and blocks the attempt when a user is tricked into signing in somewhere else [1].

One-time codes and push approvals are a real improvement over passwords alone, yet they are not phishing-resistant. A code can be relayed through a proxy in real time, and a push can be approved by someone worn down by repeated prompts. CISA notes that FIDO adoption eliminates that class of bypass, including push bombing and one-time-password compromise [1], and suggests starting with the groups where compromise hurts most, such as help desk and system administrators [1].

Layer 2: verify the request (deterministic)

Authentication protects accounts. It does nothing about a genuine person being convinced to do something, or a message from a supplier's genuinely compromised mailbox.

The control is a fixed out-of-band step for a short list of request types: changing payment details, moving funds, changing a recovery contact, issuing a credential. You verify by calling a number already in your records, and someone other than the requester confirms.

Two properties make it work. It is triggered by the type of request and ignores how the message reads, so an unusually convincing message gets the same treatment. And it has no exception for urgency or seniority, which are precisely the levers being pulled.

The cost is real. This layer adds friction to real transactions, and the friction lands on people who are trying to get work done. That trade is a good one only because the alternative is asking a person to detect something that can no longer be detected by reading. Keep the list of triggering request types genuinely short, so the friction falls where the money is and nowhere else.

It also covers a case that authentication cannot touch at all: a supplier whose own mailbox has been compromised. The message is genuine, the account is genuine, the authentication records are correct, and the request is still fraudulent. Verifying who sent it will not help here; only verifying the request does.

Layer 3: assume the first two failed (probabilistic)

Now detection earns its place. A model can weigh things a rule cannot: whether the request is consistent with this relationship, whether the writing deviates from that sender's history, whether the timing and channel fit. Fuse that with deterministic signals such as authentication alignment, domain age, and whether you have corresponded before.

Deploy it first where a wrong answer is cheap. Triage of user-reported mail costs nothing when wrong, and post-delivery action can be undone. Inline blocking should come last and only after you have measured false positives on real traffic, because the cost of a false positive here is a lost order and a colleague who stops trusting the system.

What actually works

Beyond the ordering itself, a few habits help.

Make the rule shorter than the exceptions list: one page that names the request types and the verification channel. If it needs a flowchart, it will be skipped under pressure.

Give people a way to slow down that does not rest on their judgement. "Our process requires a callback" is easier to say than "I don't believe you", and it works on a real supplier and an attacker alike.

Instrument the process instead of the mailbox. Track time from request to payment, how many bank changes happened this quarter, and how many were verified. Those numbers describe the path the money actually takes, and unlike click rates on simulations they do not measure how well staff recognise a simulation.

Rehearse the callback once. Most organisations discover during an incident that the number they hold for a counterparty is out of date, or that nobody knows where the authoritative record lives. Finding that out on a quiet afternoon costs an hour.

Reward reporting, including when it was nothing. With the markers gone, unease is a real signal and often the earliest one.

What doesn't work

Buying detection and calling it done leaves you with layer three alone, which is the layer that fails silently.

Push notifications as your MFA endpoint are better than passwords but not phishing-resistant, and the failure mode is a tired person tapping approve.

Exceptions for executives create a documented bypass. An executive who cannot be told "our process requires a callback" is exactly the name the attacker will use.

Verification using contact details from the message, including the signature block, the letterhead, and the number on a PDF, is the single most common way a good process is defeated.

Treating a compromised supplier mailbox as out of scope misses the case layer two exists for: a genuine account sending a genuine-looking request. If your process assumes the sender is either you or an impostor, it has a gap in the middle.

Where to start

• Enrol the highest-privilege groups in FIDO first, per CISA's staged guidance [1]: administrators, help desk, anyone who can approve payments.
• Write the out-of-band rule and name the request types it covers, on one page.
• Audit last quarter's bank detail changes and check how many were verified by callback to a number from your own records. That number is your real coverage.
• Remove the urgency exception in writing, and tell people it is gone so they can point at it.
• Only then look at detection, and deploy it on reported mail before anything blocks inline.

The uncomfortable conclusion is that the most effective controls against AI-assisted social engineering have nothing to do with AI. They are an authentication protocol and a phone call, and that is why they hold as the other side gets better.

Sources

1. Implementing Phishing-Resistant MFA, Cybersecurity and Infrastructure Security Agency: https://www.cisa.gov/sites/default/files/publications/fact-sheet-implementing-phishing-resistant-mfa-508c.pdf
2. More than a Password, CISA: https://www.cisa.gov/MFA