#37 - Want 80% better results? Downskill your DS team

What makes a fraud prevention system great?

Many things: technology, picking the right tools, harnessing AI, setting up policies correctly, aligning it with business objectives, etc.

But there’s only one “make or break” component: the team.

So the right question to ask is how do we make our team great?

Again, we have several answers:

We hire great talent. That’s a straightforward strategy, if you can afford it.

Another strategy would be upskilling your existing talent.

But here’s the thing: 

For years, the conversation about upskilling fraud professionals was always focused on training them with new technical skills.

First it was SQL. Then Python. Then R. 

Investigators would learn analytical skills. Analysts would learn data science skills. 

It’s a win-win for everyone: the business gets to retain even higher quality talent, and the employees get to improve their salary position.

But there’s one thing I see very few companies do, and that is training their data scientists in investigative techniques.

And many times that’s the number one thing that stands in their way to better performance.

How come? 

Let’s think about a machine learning (ML) model. What would make it better?

By and large, we’re talking about two things: labels and features.

The thing about labels

When training an ML model, like any other solution, we deal with an iterative process.

We throw data into a rough MVP, spin the wheel, analyze the outcome, make some tweaks, and repeat the process until we get to the end product we desire.

What leads this process, and how we choose the next iteration, depends on the different performance metrics. Primarily, how much fraud we caught and for what cost in false positives.

But what if the labels we are using are not accurate?

Well, guess what? Your labels are NOT accurate. That’s a fact.

Fraud often goes unreported, reported too late, gets mixed up with credit losses, and don’t even get me going on why it’s so hard to label false positives.

So when a new iteration in the process shows surprisingly bad results, we need to ask ourselves why that is the case. 

Did we decide on the wrong experiment? Or are “dirty” labels creating a false picture of performance?

Is this truly a false positive? Or just unreported fraud?

Is this truly a false positive? Or did the issuer choose not to send us the chargeback?

Is this truly a fraud case? Or is it friendly fraud?

Is this truly a fraud case? Or was it incorrectly blocked by an investigator?

Second-guessing is a critical skill for anyone working in the fraud space. If your skillset is limited to only analyzing data rather than questioning its integrity, you’re at a disadvantage.

A data scientist reviewing their own results will eventually need to look at individual case examples to help them understand better how to optimize their algorithm.

And being able to call out wrong labels is a critical part in that procedure.

The thing about features

We talked about the iterative experimentation process, and we mentioned that each time we need to decide which experiment to run next.

But what if there are additional, high-value options that do not appear in the data? Sometimes, this would be easy to pick up on and well within the skillset of any data scientist.

A good example: transforming continuous features into categorical ones.

Simply put: do we represent <payment_amount> as a numerical figure ($99) or in a “bucketed” range format (i.e., “$50-$100”)?

Different algorithms would work better with different representations of the same data. 

But that’s not what I’m actually talking about. I’m speaking of when data only appears as unconnected dots.

Here’s a simple example: 

It’s usually the case that email addresses that contain digits are more associated with fraud than emails that do not. Especially if these are not just 2-digit sequences, or 4-digit ones that start with 19 or 20.

If I don’t have a feature in my dataset that marks such emails, an ML algorithm would likely not be able to distinguish these two populations.

A data scientist would need to spot this pattern in the data, figure out if it has any meaning (good? bad?), design an accurate new feature for it, and only then run the next experiment.

Sounds like a low ask? It is. As said, it’s a simple example. It can get much hairier than that.

How to upskill your data scientists in fraud

The answer is quite straightforward: you have them go through your investigator/analyst training program.

Side note: What if you don’t have one? Forget everything you just read and fix this first. Seriously.

At Fraugster, we used to embed new data scientists in our junior fraud analyst “classes”. It was a 4-week program that covered all the basics from A to Z:

• Fraud vectors 101

• Investigating single elements (email, IP, address, etc.)

• Using 3rd-party/OSINT tools

• Full transaction/account review

• Designing robust fraud solutions

These were just the core tenets (coupled with a healthy dose of SQL training).

But keep in mind, it’s not only about fraud. It’s also about learning your own business: the product, the org, the tools, the data, and so much more.

There’s so much context needed for effective fraud prevention. Don’t skip it.

But that’s just the start.

Even going through a whole intense month like I described above, these skills will decay quite fast if they are not maintained.

Data scientists that deal with fraud should regularly review cases. Always.

A good rule of thumb would be a full day of review every month. Minimum.

This can start with the basic training program, and be followed up with regular side-by-side sessions with the investigations team.

But ideally, once the data scientist has reached a level of confidence–they should be reviewing cases as part of their own R&D work.

As said, the point is to infuse R&D with investigative tools. We’re not doing it for fun.

Well… most of us at least.

Downskilling is the new upskilling

Upskilling your team’s technological literacy is important to any organization that fights fraud. Don’t get me wrong, I have nothing against it and I certainly encourage it.

But this should not come at the expense of infusing domain expertise throughout the entirety of your team, including data scientists.

It might seem like a chore. It might not be clear how it benefits their career. It might even look as disruptive to their career and role.

But great data scientists would appreciate that in order for them to excel (yeah I just did it), it’s 20% about the algorithm and 80% about assimilating into the problem domain.

Are you willing to give up on 80% of your team’s potential?

What do you think? Waste of time or proven force multiplier? Hit the reply button and tell me about your experience.

In the meantime, that’s all for this week.

See you next Saturday.

P.S. If you feel like you're running out of time and need some expert advice with getting your fraud strategy on track, here's how I can help you:

Free Discovery Call - Unsure where to start or have a specific need? Schedule a 15-min call with me to assess if and how I can be of value.
​Schedule a Discovery Call Now »​

Consultation Call - Need expert advice on fraud? Meet with me for a 1-hour consultation call to gain the clarity you need. Guaranteed.
​Book a Consultation Call Now »​

Fraud Strategy Action Plan - Is your Fintech struggling with balancing fraud prevention and growth? Are you thinking about adding new fraud vendors or even offering your own fraud product? Sign up for this 2-week program to get your tailored, high-ROI fraud strategy action plan so that you know exactly what to do next. ​
Sign-up Now »

Enjoyed this and want to read more? Sign up to my newsletter to get fresh, practical insights weekly!

<