#78 - Are you actually using your ML score?

Most fraud teams I speak with have some form of an ML score provided by their vendor.

But many of them don’t use it at all.

And the ones that do mostly use it like this:

Here's the thing: 

If that's how you're using your model, you're leaving money on the table on both sides: more fraud getting through and more good customers getting blocked.

But most chances are you already know it. What’s likely stopping you from using it properly then?

Guidance and data. ML models are a black-box and are scary to mess around with. Let’s change that.

Today I’d like to revisit how to use ML scores effectively. The first part would deal with setting general thresholds as part of strategy rules. The second part, coming up next week, would focus on how to integrate scores into behavioral & trend rules.

Side note: If you need a quick refresher on what’s the difference between strategy, behavioral, and trend rules, you can find it here.

First, set your goals

Before you set a single threshold, you need to know what 'better' looks like.

This sounds obvious, but with so many KPIs to juggle it can get hairy, fast.

In fraud prevention, we orbit around two core KPIs: fraud rate and approval rate. Move one, and you almost always move the other. 

But they don't move equally, and they don't move the same way for every business.

For some businesses, fraud rate is of secondary importance as long as it stays below a regulated level. For others, it has a direct effect on their bottom line.

Understanding how much we care about the financial impact of fraud obviously affects how comfortable we are with reducing conversion or increasing user friction.

So before you touch anything, write down three things:

1. What is my fraud rate ceiling?

2. What is the maximum block rate I'm comfortable with?

3. And if I have to choose between them - which one wins?

Without answering these three questions, it’ll be hard to establish how to use any ML score.

Side note: I’ve written before about the concept of Fraud-Adjusted Approval Rate (FAAR) which might be relevant in this context.

Second, determine your segments

Here's something that took me longer to fully appreciate than it should have:

Your model learned from your entire transaction history. But your business doesn't behave as a single, uniform population.

You probably already treat certain segments differently. Different rules for personal vs. business accounts. Different review flows for high-value accounts. Different vendors tooling for specific regions or products.

These segments are worth your attention - you’ve already established that the underlying risk profile is different.

That means that a single threshold is almost certainly wrong for all of them simultaneously.

So here’s a good rule of thumb:

Any segment you already treat differently with rules, processes, or vendors is a candidate for its own threshold.

And if you currently don’t treat any segments differently? This is where you’d want to look: payment method or product type, cross-border vs. domestic, high-value vs. low-value transaction bands, and new vs. established customers.

Side note: I already covered the LTV cost of declining new customers. The numbers are uncomfortable.

You don't need to do all of them at once. One well-chosen segmentation cut can make a meaningful difference, for example:

Third, let AI do the work for you

Once you know your goals and segments, the question is: what threshold should each one have?

This used to require a data analyst, a few days of SQL or Excel work, and a fair amount of experience.

You’d need to plot your ROC charts and start to manually review them to pick each optimal score threshold.

Luckily, though, we live in 2026, and we have AI to do it for us.

When I work with Claude, my tool of choice, I simply feed it with:

1. A dataset that contains fraud scores, amounts (if relevant), statuses, fraud labels, and the segments I believe are relevant.

2. My KPI set - goals or minimum expectations. Make sure to explain how you calculate these, so as not create hidden failures.

3. Scenarios - if possible, define low/high-risk appetite scenarios that would allow it to understand your business context better.

The prompt itself changes case by case, but the best practices are:

1. Define its role - “You are a seasoned fraud analyst…”

2. Describe the dataset - “Fraud labels are found in Column C, segments are in Column G…”

3. Describe the outcome you’re looking for - “I’d like to optimize my fraud losses without blocking more than 2% of the population by dollar amount…”

4. Ask it to visualize the recommendation as well, and better yet – create a dashboard that you’d be able to play with. You’ll be shocked.

For teams without dedicated fraud analytics or data science functions, that's the difference between doing this analysis and not doing it.

Bottom line

Setting your ML score thresholds isn’t a data science problem anymore.

Get your goals clear first, then find the relevant segments and use AI to run the analysis for you.

Do it once and suddenly, your ML score wouldn’t seem like such a black box.

But we’re just getting started!

Next week, I'll cover how to wire these thresholds into your actual rules - and why the way most teams do it leaves half the value on the table. Stay tuned!

How are you setting your ML scores thresholds? Manually or with the help of AI? Hit reply - I'm curious how teams manage that in 2026.

In the meantime, that’s all for this week.

See you next Saturday.

P.S. If you feel like you're running out of time and need some expert advice with getting your fraud strategy on track, here's how I can help you:

Free Discovery Call - Unsure where to start or have a specific need? Schedule a 15-min call with me to assess if and how I can be of value.
​Schedule a Discovery Call Now »​

Consultation Call - Need expert advice on fraud? Meet with me for a 1-hour consultation call to gain the clarity you need. Guaranteed.
​Book a Consultation Call Now »​

Fraud Strategy Action Plan - Is your Fintech struggling with balancing fraud prevention and growth? Are you thinking about adding new fraud vendors or even offering your own fraud product? Sign up for this 2-week program to get your tailored, high-ROI fraud strategy action plan so that you know exactly what to do next. ​
Sign-up Now »

Enjoyed this and want to read more? Sign up to my newsletter to get fresh, practical insights weekly!

<