#105 - The signals that expose a broken fraud rule

Your CEO slacks you at 5:48pm - “why are our rules misfiring?”

Are they? How would he know? How would you know?

All the CEO saw was one angry review on the website.

Two stressful days and a lot of SQL later, you find it. One rule went haywire after a small data change, and it’s been declining good customers all week.

You did have a monitoring process - it just couldn’t keep up.

Why rule monitoring breaks at scale

Every rule shares the same blind spot - once it blocks an action, you can’t really say for certain if it was bad or not.

There are ways around it. I’ve written before about how to spot false positives and how to label the fraud you already blocked.

Those methods are great for a deep dive into one rule. They’re far less great for checking all of your rules, every single week.

Here’s the thing:

Once you have dozens or hundreds of rules, a weekly review without knowing where to look swamps your team. You do a lot of busy work instead of actively honing your system.

So how do you find the one rule that’s misbehaving, without checking all of them?

Let’s talk about it.

Three things to balance

A while back I wrote about why every fraud system decays over time, and how to track that decay against each solution’s go-live performance.

Since then, I keep getting the same question: how do I actually implement it in practice? Especially when the goal is to know quickly that something is off.

Here’s the answer I give: any rule monitoring setup has to balance three things:

Manual effort. Your team can review some of your rules every week, but it can never review all of them. Even with AI agents.

Timeliness. You want to know a rule is misbehaving as soon as it breaks. Gradual degradation can wait for your periodic review. A sudden spike in declines can’t.

Accuracy. You don’t want to waste your team’s time reviewing rules that are actually fine. Every false alarm pulls an analyst away from a rule that really broke.

The good news? You can balance all three by monitoring just two types of signals.

Two signals that something is off

Here’s the setup I’d use. Two monitoring views, each tracking spikes in a rule’s numbers over time.

Side note: Sounds familiar? Yes, this is exactly the same principle behind anomaly detection, only applied to your system instead of fraud itself.

Signal 1: hit volume

Start with the easiest one: count how many times each rule fired per day, week, or month, and compare it to the average of previous periods.

A rule firing more isn’t necessarily a bad thing. But a rule that suddenly spikes raises the suspicion that something broke in the system, and when that happens, most of the new hits tend to be false positives.

Side note: Clean up seasonality and overall traffic spikes first, otherwise this view gets noisy fast.

Is your rule hitting less? That’s also cause for concern, but on the opposite end. It might be that something in the data broke and fraud is now flooding your gates unimpeded.

Signal 2: false positive proxies

The second view tracks what you actually care about. Are your rules creating more false positives?

The catch is coverage. None of the methods for tagging false positives will automatically hit every single one.

But here’s the trick - you don’t need them to.

All you need is a proxy measure that should stay fairly stable over time. Some options I’ve used:

• Accepted appeals. Users who write in asking why you declined their payment or restricted their account, and win after human review.

• Returning good users. Blocked events that you can later link to good, established users.

• Manual review overturns. Rule hits your analysts reverse when they land in a review queue.

• Control group outcomes. Events from a small holdout group that the rule would have blocked, but turned out good.

• AI agent labels. A softer signal, but an agent (Jev, for instance) reviewing your rule’s hits gives you a consistent read.

Here’s the thing. You won’t be able to tag all your false positives. And your accuracy also would not be perfect. 

But that’s fine.

Track each proxy as a ratio of the rule’s total hits.

If nothing specific changed and the ratio suddenly jumps, your real false positive rate most likely jumped with it.

Set the thresholds

Put both views together and you get a high likelihood that a rule is misbehaving, without ever knowing its true false positive rate.

Then tier your alerts by the size of the spike:

IF hits_vs_baseline >= 5x OR fp_proxy_ratio_vs_baseline >= 2x THEN same-day alert
IF hits_vs_baseline >= 2x OR fp_proxy_ratio_vs_baseline >= 1.5x THEN weekly report

A 10x spike gets a look the same day, a 2x spike waits for the weekly review, and every other rule stays out of your team’s way.

Sort what comes in by priority, and for each flagged rule, pull 50 to 100 of its recent hits for a manual review of your top offenders.

And because your team only reviews what the alerts send its way, the setup scales to any number of rules.

Side note: The same logic works for AI agents. Override rates make a great proxy, and I covered them when I wrote about how to monitor fraud AI agents.

The bottom line

You’ll never know for sure what happened to an event you blocked.

But you don’t have to if you just want to efficiently monitor your rules. You just need to know the moment a rule’s hits or its false positive proxies spike.

All you have to do is apply anomaly detection methods you are already familiar with to monitor your own system.

How does your team know when a rule is acting up? Hit reply and let me know!

In the meantime, that’s all for this week.

See you next Saturday.

P.S. If you feel like you're running out of time and need some expert advice with getting your fraud strategy on track, here's how I can help you:

Free Discovery Call - Unsure where to start or have a specific need? Schedule a 15-min call with me to assess if and how I can be of value.
​Schedule a Discovery Call Now »​

Consultation Call - Need expert advice on fraud? Meet with me for a 1-hour consultation call to gain the clarity you need. Guaranteed.
​Book a Consultation Call Now »​

Fraud Strategy Action Plan - Is your Fintech struggling with balancing fraud prevention and growth? Are you thinking about adding new fraud vendors or even offering your own fraud product? Sign up for this 2-week program to get your tailored, high-ROI fraud strategy action plan so that you know exactly what to do next. ​
Sign-up Now »

Enjoyed this and want to read more? Sign up to my newsletter to get fresh, practical insights weekly!

<