#71 - The fraud you blocked will break your next model

You obsess over tagging false positives. 

You feed chargebacks back to your models. 

You label every confirmed fraud case you can find.

But there's a population you're almost certainly ignoring: the fraud you already blocked.

And if you're training or refreshing a new fraud solution - be it a model or a rule - that blind spot might be undermining your results without you even noticing.

Why, how, and what to do about it?

Let’s talk about it.

The hidden labeling problem

Here's what happens when you block a fraudulent event and don't label it - it disappears from your training set. 

It’s just... gone. 

An unlabeled event sitting in your declined population, invisible to your next model.

The irony is that the cases most likely to go unlabeled are the simple ones. The ones you’re already blocking right now: obvious velocity spikes, blocklist matches, and high-score events.

You might be asking yourself, “why is that a problem?”

We’re already catching these simple fraud cases, we should only care about the more sophisticated ones. The ones we’re not catching today. Right?

But here’s the thing:

Next time you’re going to train a new model or test a new rule - you’re only going to train it on the fraud you didn’t catch.

And the “easy” fraud? Slowly, with each iteration, you might be quietly hindering your solution’s ability to spot it. After all, it’s not in the dataset anymore.

And here's what compounds it even more: your validation set has the same problem.

When you test your new solution on historical data, it’s easy to assume all the fraud you caught before will be caught again. But you don’t know it will.

When the model performs well in validation, you have no reason to suspect anything is wrong. So you deploy it to production.

And then? You find out that the live fraud rate is higher than anticipated.

How to label your declined bads

The good news: just like with labeling false positives, there are methods with which you can approximate your blocked bad events quite accurately. 

Used in combination, they can get you most of the way there.

Linking to confirmed fraud

Some of your blocked events are already linked to confirmed fraud - you just haven't connected the dots.

Start with direct links: if a blocked event shares a device, card, or account with a confirmed chargeback or fraud complaint, the label is essentially free. 

Blocklist hits are the clearest example - if it fired, you know why.

Then go one level deeper with entity-based linking - a declined signup linked via IP and device to an account that later produced a chargeback? That's fraud. 

Entity resolution tools make this scalable, but even a manual exercise on your highest-risk blocks will surface labels you didn't know you had.

High-confidence heuristics

Some blocked events don't need linking - the signal is loud enough to label directly.

Extreme velocity spikes are a reliable proxy. Same with very high model scores, well above your normal block threshold. Both are cases where the system was essentially certain, and you can treat that certainty as a label.

Rules are trickier, as over time it’s difficult to ascertain rules are accurate as they were at launch. 

Most rules aren't precise enough to use as label proxies - their false positive rate is too high. The exception is behavioral rules with a long, stable performance history. 

If a rule has been running for a year at 80%+ precision, its hits are reasonable label candidates.

Side note: Don't just assume the heuristics are trustworthy. Validate them manually the first time - not case by case, but by sampling each heuristic and checking whether the hits actually look like fraud. 

Labeling with AI agents

AI agents are great at cross-validating the previous methods, as well as contributing more high-confidence labels themselves.

The approach is simple - train an AI agent on a subset of approved events so it learns to split patterns. Then have it label your blocked traffic.

The results might surprise you with how accurate they are.

You don't need to automate all of this

With false positives, you need a continuous, scalable process. Especially if you want it to feed a monitoring pipeline that lets you know in real-time if any solution starts degrading.

But blocked fraud labeling is different.

In most cases, you're building a specific dataset for a specific purpose: training or validating a new solution. That's a one-time exercise, not an ongoing operation.

Which means you can afford to do more of it by hand. Not case-by-case investigation, but manually validating each heuristic, sanity-checking the entity links, and reviewing a sample of the AI agent outputs.

The bar for "good enough" is building a dataset you trust, not necessarily achieving full automation.

In fact, I would even discourage you from automating it. It’s a waste of resources and would likely not be accurate enough.

Hopefully, that changes the cost calculation enough to encourage you to invest in this. At least when launching major new solutions.

The bottom line

Your blocked population isn't a black hole. There are ways to shine a light on it - whether labeling false positives or fraud cases.

But unlike with false positives, labeling blocked fraud has a specific importance - making sure your solutions don’t get “too smart”.

Because nothing is more awkward than having a model that is effective against sophisticated attacks but lets Nigerian IPs log into your users’ accounts.

Have you run into this problem when training or refreshing a model? Curious how others are handling the declined population labeling challenge - hit reply and let me know.

In the meantime, that’s all for this week.

See you next Saturday.

P.S. If you feel like you're running out of time and need some expert advice with getting your fraud strategy on track, here's how I can help you:

Free Discovery Call - Unsure where to start or have a specific need? Schedule a 15-min call with me to assess if and how I can be of value.
​Schedule a Discovery Call Now »​

Consultation Call - Need expert advice on fraud? Meet with me for a 1-hour consultation call to gain the clarity you need. Guaranteed.
​Book a Consultation Call Now »​

Fraud Strategy Action Plan - Is your Fintech struggling with balancing fraud prevention and growth? Are you thinking about adding new fraud vendors or even offering your own fraud product? Sign up for this 2-week program to get your tailored, high-ROI fraud strategy action plan so that you know exactly what to do next. ​
Sign-up Now »

Enjoyed this and want to read more? Sign up to my newsletter to get fresh, practical insights weekly!

<