#90 - You are not ready for AI

"Never in the history of mankind was it so easy to ask the wrong questions and get the wrong answers at such a scale."

That's Shachar Meir, an executive data advisor, who joined me in this week’s podcast episode to talk about risk teams, data teams, and everything in between (mostly AI).

He was talking about AI-powered self-service analytics. And I haven't stopped thinking about it since.

Every fraud team I talk to has some version of the same story.

They found an AI tool that looked promising, got sign-off from leadership, plugged it in - and six months later, they're disappointed. And they can't quite explain why.

The tool technically works, but the ROI is simply not there. Best case, it’s just not what was promised. Worst case, they can’t even measure it.

So what went wrong?

It wasn’t the tool. It was what fueled it.

I've been watching this cycle repeat itself for the last 18 months. And every time I noticed the same thing: most fraud teams treat AI like duct tape. Plug it into a messy data environment and hope it figures things out.

But here’s the thing:

AI needs better data hygiene than your human analysts, not worse.

That sounds counter-intuitive. Isn't AI supposed to be the thing that cleans up the mess?

Well, it could. But is it supposed to work like that? (no) Was it trained to do that? (no) And most of all - does it impact its performance? (yes)

Your human analysts know when something feels off. They push back, ask questions, flag anomalies. AI will give you a confident, fluent, wrong answer and you won't know it's wrong until the damage is done.

So, are you ready to implement agentic AI in your stack?

Here are three things most teams skip.

Pick one use case

The instinct of many teams I talk to is to utilize AI for all tasks they think it can automate, all at once. I understand the appeal.

But life doesn’t work that way, and more importantly - fraud prevention systems don’t work that way.

In fraud, two tasks that look very similar at a glance often look quite different from up close:

• Reviewing new customers for identity fraud versus reviewing established customers for ATO. Not the same thing.

• Catching fraud versus identifying false positives. Not the same thing.

• Researching a rule versus writing a rule. Not the same thing.

I can go on and on, but you get the idea. But still, most teams I’ve seen implementing AI were trying to use it to solve anywhere from 6 to 15 different use cases.

No wonder it failed. Each of these is managed differently, needs different customization, different data, different context, and different accuracy.

This is obviously too much.

But the danger with AI is how confidently the results are delivered. You get no errors, no process failures. No wonder they cannot see the forest for the trees.

The fix is to pick one bounded, repeatable use case and make it your POC.

This would serve you not only by getting acquainted with the new technology and its intricacies, but more importantly - it’ll allow you to develop monitoring and evaluation criteria without navigating unnecessary complexity.

Create a single table for that use case

Creating a single table with all the relevant data for your use case isn’t necessarily a trivial request. And that’s exactly why you need to insist on it.

If you don’t, you’re likely going to encounter one of two scenarios I sadly see too often:

1. Some critical data isn’t available to the environment where the AI is deployed, making it ignore crucial signals when making decisions.

2. Data is distributed across environments, databases, and technologies, making AI agents fail when trying to connect the dots.

To avoid these common issues, you want to make sure data is easily accessible and well-defined.

What does well-defined mean?

Clear column definitions. Metadata that tells the AI what things actually mean - not just column names, but business definitions. What counts as a declined transaction. What "new device" means in your context.

The things an analyst would know but a machine won't assume correctly.

If you can't trust the table yourself, the AI can't either. It just won't tell you that. Instead, it'll give you a confident wrong answer.

Treat your data team as a partner, not a vendor

Most fraud teams don't own their data platform (and rightfully so). It means AI adoption is a joint project whether you like it or not.

One mistake that Shachar brought up is treating the data team like a vendor - showing up with a list of requirements, no business case, and zero context.

And that’s the fastest way to get the data team disinterested in doing the job. What happens then?

Scope gets blurry, effort estimations get inflated, ROI seems low, and behind the scenes the team is pushing to do something else they are excited about.

Would your project get budgeted under these circumstances? Likely not.

Go with a mission instead.

Tell them what's at stake. Show them what a wrong AI answer looks like in your context - what it means when a fraudster slips through, what it costs when a real customer gets blocked. Both on a case level as well as the aggregated numbers.

Make them care about the outcome, not the ticket.

Data teams want to matter, but many fraud teams never give them the chance.

When they're bought into the why, your project gets prioritized differently. When they're invested in the result, they'll catch data quality issues you'd never spot from the fraud side.

Yes, that means more meetings, less control, and longer delivery times. But as the saying goes: “if you want to go fast, go alone. If you want to go far, go together”.

Bottom line

The teams failing at AI don't have an AI problem. They have a preparation problem.

The teams succeeding aren't using better tools. They did the boring work first.

Shachar and I went deeper on this in this week’s episode - including the self-service analytics trap most fraud teams are walking into right now, and why giving an LLM direct access to your data warehouse might actually be more dangerous than the old dashboard approach.

If someone has sold you on AI-powered self-service for your fraud team, listen to this before you sign anything.

Are you struggling with an AI adoption project right now? I would love to hear about your experience and perhaps share a tip or two.

In the meantime, that’s all for this week.

See you next Saturday.

P.S. If you feel like you're running out of time and need some expert advice with getting your fraud strategy on track, here's how I can help you:

Free Discovery Call - Unsure where to start or have a specific need? Schedule a 15-min call with me to assess if and how I can be of value.
​Schedule a Discovery Call Now »​

Consultation Call - Need expert advice on fraud? Meet with me for a 1-hour consultation call to gain the clarity you need. Guaranteed.
​Book a Consultation Call Now »​

Fraud Strategy Action Plan - Is your Fintech struggling with balancing fraud prevention and growth? Are you thinking about adding new fraud vendors or even offering your own fraud product? Sign up for this 2-week program to get your tailored, high-ROI fraud strategy action plan so that you know exactly what to do next. ​
Sign-up Now »

Enjoyed this and want to read more? Sign up to my newsletter to get fresh, practical insights weekly!

<