Whether AI can do the job is settled. Whether it will still be doing it right in eighteen months is the question that actually decides the return.
There is a moment in almost every AI project that nobody puts in the business case.
It arrives about a year after go-live. The system still runs, the dashboards are still green, but somewhere along the way the world moved – a supplier redesigned its invoice, a pricing rule was rewritten or a new team started feeding the system data nobody checked.
The model doesn’t crash. It just becomes quietly, confidently wrong, and it keeps going.
That moment is the real frontier of enterprise AI. Not capability but continuity.
The first year is the easy year
A system that scores 96% on day one can be materially less reliable by month nine without a single line of code changing. Data drifts. Upstream systems get patched. Rare edge cases become routine. And in systems that keep learning, yesterday’s outputs quietly become tomorrow’s inputs, so small distortions compound into structural ones.
You can watch it happen in almost any function. A demand forecast tuned on two stable years meets a supply shock and keeps recommending the same reorder quantities. A document pipeline meets a redesigned form and confidently misreads it instead of rejecting it. A support assistant answers questions about a policy that was withdrawn last quarter. None of them fail loudly – which is exactly the problem.
It gets sharper with AI that acts rather than advises. An assistant drafting a reply makes a small, reviewable mistake. An agent that reconciles invoices, updates records and triggers payments makes mistakes that travel – through a ledger, a customer relationship, a compliance report – and by the time anyone notices, the trail is long and the correction is expensive.
Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls. That last one isn’t a footnote. It’s a governance gap showing up as a budget line.
Trust is no longer something you assert
For years, “trustworthy AI” lived on the About page, somewhere between sustainability and diversity. Something a company said about itself.
Then the questions changed. Ask anyone in enterprise sales what’s turned up in their deals lately: which parts of this service are AI-driven, what happens when the model is wrong, who’s accountable for the output, does a human see it before a customer does?! Those questions now arrive in procurement questionnaires, security reviews, insurance renewals and first sales calls – from customers who’ve been burned, or who’ve read about someone who was.
So, trustworthiness stopped being something you assert and became something you evidence: logs, versions, test results, a named owner, a procedure for the bad day. A confident answer no longer ends the conversation. Evidence does.
And it travels. Most organisations today are simultaneously buying and selling automated decisions. The model inside your vendor’s platform is part of your service now, and your customer will ask you about it — not them.
Which leads somewhere uncomfortable. An organisation can pass every check on the day it’s checked and be materially less reliable six months later, without ever being told anything changed. Nothing announces the moment a system starts to drift. The useful question isn’t did we check? It’s how would we know?
The oldest question in the budget meeting
Every conversation like this eventually reaches the same objection, and it deserves a straight answer rather than a scare story: why spend real money on security and resilience?
It’s a fair challenge, because the return is an absence. Nothing happened. No breach, no outage, no headline, no awkward call. Try building a business case around that and you’ll lose to almost any project with a visible number attached. It’s why security is the first line cut in a good year and the first question asked in a bad one.
But the arithmetic has moved. IBM’s Cost of a Data Breach Report 2026 puts the global average breach at $4.99 million, up 12% in a year. One in four malicious breaches is now AI-enabled — a 56% increase — and those run to roughly $6 million, about a million more than a breach without AI involved. Attacks aimed squarely at companies’ AI systems jumped from 13% to 21% of organisations in twelve months.
What’s striking is where the weakness sits. Not in exotic attacks against sophisticated defences, but in the ordinary stuff: 68% of breached organisations had no policy governing AI use at all, 43% of incidents involved shadow AI — tools nobody had approved — and 92% of those hit by an AI-related breach lacked basic access controls around it. Unmanaged tools, unowned systems, unwatched data. Unglamorous, well understood, entirely fixable.
Meanwhile the upside has become measurable. Organisations using AI and automation extensively in their security operations cut average breach costs by around $1.93 million and contained incidents roughly 65 days faster. Sixty-five days isn’t a statistic — it’s the difference between an incident your team handles and an incident that handles your team.
That’s the defensive case, and it stands up. The better case is the strategic one. Resilience is what lets you say yes. Organisations with an inventory, monitoring, clear ownership and a tested failure procedure can put AI into a customer-facing process or a financial control, because they can see what it’s doing and stop it if it goes wrong. Organisations without those things either freeze AI at the pilot stage, where it earns nothing, or deploy it blind, where it eventually costs something. Both are expensive; only one is visible.
Resilience isn’t the tax you pay on automation. It sets the ceiling on how much automation you can safely afford — and how quickly you get there.
Where this actually starts
Less dramatically than you’d think.
It starts with knowing what you have, which most organisations can’t yet produce, because so much of their AI arrived quietly inside software they’d already bought. An inventory — what it is, what it touches, who owns it — is the dullest item on the list and the one everything else waits for.
From there it’s a short list of familiar disciplines applied to an unfamiliar asset. Monitor behaviour, not just uptime: infrastructure metrics tell you a system is running, never whether it’s still right. Decide in advance what it should do when it isn’t sure — escalate, fall back to a simple rule, decline to act. A system that always produces an answer isn’t a feature. Keep testing after launch, because stress-testing once tells you about a version that won’t exist in six months. And give every production system a named owner and a bad day someone has actually rehearsed.
None of this requires a research budget. It requires treating AI the way you’d treat any other critical asset: a register, an owner, a monitoring regime, a maintenance plan, a tested failure procedure.
Capability was the last decade’s question. Staying power is this one’s. Resilience isn’t what you add once the AI works — it’s what decides whether it keeps working.
Not sure where you stand? Start with the least glamorous step: list every AI-driven process running in your business today, who owns it, and how you’d find out if it started getting things wrong. Most organisations are surprised by what that turns up — and it’s far cheaper to be surprised now than mid-incident.
