#25 - How to Build Fraud Rules that Outperform AI Models

If someone asks me what’s my biggest pet-peeve in fraud prevention, it’s this:

Thinking that fraud rules are inaccurate, especially compared to AI/ML.

As someone who started their fraud prevention career writing rules, and has continued to use rules throughout all of my career, my experience shows the exact opposite to be true.

Now don’t get me wrong, I’m not saying that rules are more accurate than AI/ML.

I’m saying rules can be more accurate than AI/ML.

The truth is, most teams I encounter just don’t know how to write good fraud rules. Even teams that rely solely on rules often fail to do a good job.

So today I want to share the process I’ve been using to write rules for 15 years.

Want to get the best performance out of your rules? Read on.

The two most common mistakes that ruin rule performance

The first mistake is one that I hope most of you are already aware of and are avoiding. That is to overfit our rule to describe the exact fraud pattern we see.

The temptation to do so is clear: the more tailored the rule is, the higher its performance will be.

But it’s a double-edged sword.

Let’s take this rule for example:

The rule will block any payment that has a “mail.com” email domain and its username ends with 6 digits. This will perform quite well, right?

Yes, for a couple of days.

But once the fraudsters understand they are blocked, they will mutate their behavior and suddenly every condition in our rule turns into a potential backdoor.

If the fraudsters change to another spam service? They will bypass the rule.

If the fraudsters remove (or even add!) one digit to the username? They will bypass the rule.

In reality, the rule hits would drop massively very fast. At the same time, accuracy will very likely drop as well.

A much better version of the rule would be:

See the difference? Yes, using high-level features is not straightforward: accuracy might suffer a bit, and of course someone needs to build/buy these features first.

But even if the accuracy isn’t as high as the first version, it’s now much harder for fraudsters to mutate their behavior and circumnavigate our defense.

Ok, but this is really “Fraud Rules 101” and not what I wanted to focus on today. The real issue is with the second common mistake I see: 

Writing rules that describe only the fraud pattern.

I mean, yeah, it should be included in the rule, but it’s only the first 10% of the journey.

Here’s an example:

In this case, we’re trying to block ATO attempts where the login IP appears from a different country than the account country, and the device used is new in the account.

Again, at first glance this seems like a solid rule, right? But in reality, you often find out that there are a lot of real users who will be negatively impacted by it:

What if I travel to visit my family and use my dad’s laptop?

What if I moved to a different country a year ago but now bought a new phone?

What if I just signed up for the service while using a VPN?

What if I travel for work and am using my company laptop to login for the first time?

And the list goes on…

Yes, we will definitely block many fraudulent attempts, but that’s not the only way in which we measure rules. 

It’s likely that the rule’s accuracy will be mediocre to begin with and will only degrade with time.

The structure I use for all my fraud rules

So what’s missing? Well, actually two things.

First, we’re missing a section that prescribes the rule to the relevant population segment. 

In this example, we don’t want to fire the rule on accounts newer than 30 days as the risk for ATO is very limited.

Secondly, the most crucial part of any rule is the part where we don’t detect the fraud pattern, but the patterns of false positives.

By prescribing how false positives look like in the specific context of this fraud pattern, we are able to exclude them from being hit.

And so every rule we write needs to have three sections to it:

1. Population: the segment on which we want the rule to run

2. Detection: the core fraud pattern we’re trying to block

3. Exclusion: the false positive patterns we want to exclude from the rule

Let’s update our rule so it follows this structure:

Our two-statement rule has suddenly become an eight-statement one. Let’s review quickly:

Population logic:

1. Account age needs to be higher than 30 days to be relevant for ATO risk

2. Guest accounts are not relevant as they are virtual and cannot be ATOed

Detection logic:

3. Login IP comes from a country that is not the account country

4. Device is new on the account

Exclusion logic (exclude if):

5. IP country was already seen in the account at least 60 days ago

6. IP connection type is highly controlled and can indicate business travel

7. IP country matches the nationality of the customer name

8. There have been at least 10 different IP countries in the account (frequent traveler)

Developing the rule logic must be in accordance with what you actually see in the data. Every exclusion logic you add, needs to show improvement in the rule performance simulation.

The idea is not to develop expert-driven logics that are always correct. The idea is that we tailor the rule to the patterns we see.

What do we gain from following this structure?

Firstly, segmenting rules to run over specific populations greatly reduces the risk of rules going haywire or conflicting with others. You’ll see less internal incidents and it’ll be easier to manage & document.

Secondly, make no mistake: researching false positive patterns takes much more time than detecting fraud patterns.

It’s also imperative that exclusion logics do not inadvertently create loopholes for fraudsters to exploit. And that is not something that can be done easily, as you cannot test how fraudsters will react in the future based on your historical data.

But your rules are guaranteed to have a much higher accuracy when they go live, and will degrade much slower.

And that’s exactly what separates mediocre rules from excellent ones.

I’ll also bet that in certain high-risk segments, you’ll even see better accuracy than your AI/ML.

Now think of your worst performing rules. Did you get some ideas on how to improve them?

Was this helpful? Have a different approach you’re following? What other rule writing areas are of interest? Hit the reply button and let me know!

In the meantime, that’s all for this week.

See you next Saturday.

P.S. If you feel like you're running out of time and need some expert advice with getting your fraud strategy on track, here's how I can help you:

Free Discovery Call - Unsure where to start or have a specific need? Schedule a 15-min call with me to assess if and how I can be of value.
​Schedule a Discovery Call Now »​

Consultation Call - Need expert advice on fraud? Meet with me for a 1-hour consultation call to gain the clarity you need. Guaranteed.
​Book a Consultation Call Now »​

Fraud Strategy Action Plan - Is your Fintech struggling with balancing fraud prevention and growth? Are you thinking about adding new fraud vendors or even offering your own fraud product? Sign up for this 2-week program to get your tailored, high-ROI fraud strategy action plan so that you know exactly what to do next. ​
Sign-up Now »

Enjoyed this and want to read more? Sign up to my newsletter to get fresh, practical insights weekly!

<