#87 - Your vendor isn't watching. Your team isn't either.

“Meet the night shift.”

That’s the headline splayed over notion’s homepage.

Notion is of course not a fraud prevention company, but the concept behind their AI strategy is one that sounds familiar to anyone speaking to any vendor in the space.

And, honestly, I think it’s the right take.

But who’s watching the night shift? Who’s making sure that after the green light has been granted there was no drift in performance?

And more importantly - can you trust them?

A few weeks ago I argued that fraud analytics needs to grow because of this. Today I want to examine why that monitoring gap exists, and why it's harder to fill than most teams assume. 

The metrics you’re watching lag by weeks

You might wave my questions off as trivial.

You might say that when your AI agent is degrading, the signal eventually shows up in your outcome metrics: chargeback rate, fraud loss, false positive rate, and the list goes on.

But the problem is that those numbers may lag by weeks (chargebacks) or be hidden from you entirely (false positives).

Even if you manage to notice drift, by the time you do your system has been broken for weeks and damage might be worse than a real fraud attack.

Ask a fraud team what they monitor for their AI agents and the answers are almost always operational: Is it running? Are cases being processed? Is volume consistent?

But none of these questions tell you whether the agent is doing the right thing. And that’s the question that actually matters.

So who will?

Will your vendor alert you?

Have you ever seen a ML drift dashboard built by a vendor to monitor their own Machine Learning performance?

I assure you, they do have one, and they are looking at it. Internally, that is.

But are they exposing it to their clients?

Why would it be different with agents?

Here’s the thing about the vendor dashboards, they too are designed around operational measures: Is this thing working? Are we delivering the service we committed to in our contract?

The default view might showcase volume, perhaps time saved. ROI if it’s really advanced on the product experience side.

Would they show you how many times their AI agent was overruled by your human agent? Over time? What’s the trend? 

That’s almost never on the default dashboard.

I don’t think that’s malicious. Vendors build what gets sold, and right now “hours saved” closes more deals than “here’s how to catch when we’re wrong.”

But the result is that teams relying on their vendors for AI capabilities shouldn't necessarily rely on them to track performance.

Will your org alert you?

Obviously, everyone on your team is interested in your mutual success. I am not trying to insinuate that’s not the case.

But that is not a strong enough antidote against bias.

Here’s the thing: it’s 2026 and everyone is (over)hyped about AI. Organizations, leaders, and teams are placing big bets on it.

Some in implementation and transformation resources. Some are adjusting their budget for minimal hiring. Some might even consider reducing headcount.

In this environment, it’s easy - too easy - to go from wanting to succeed to needing to succeed.

Again, that’s not out of malice. Bias can affect anyone, regardless of their commitment to the business. I know because I fell for it. Sometimes it’s just easy to be blind to reality.

But if we’re honest, there’s not much organizational incentive for teams to go looking for AI drift. And when no one has defined who owns the monitoring, the monitoring doesn’t happen.

Will your human agents alert you?

Some teams think the fact they are operating with a human in the loop would save them from slow agentic degradation.

Maybe, but I wouldn’t count on it.

Sure, agents making mistakes would definitely be something your team would notice. They will complain about AI. They would learn not to trust it so much.

But would they know that it performed worse this month than in previous months? Would they know why?

The very likely answer to both these questions is no. They won’t.

Without looking at hard data, a human can definitely notice a shift from 80% accuracy down to 20%. But would they notice the difference between 45% and 40%?

That’s more than a 10% decrease in performance that your agents are more than likely to miss. I know I would miss it.

Your agents are focused on making the correct decisions for your users. They care about their performance. Burdening them with tracking your AI health is too much to ask.

What to actually track

You don’t need a new tool. You need someone to own a few metrics:

For agents with humans in the loop: override rate is your primary leading indicator. Track it weekly, set a baseline in the first two weeks of deployment, and alert when it moves.

For autonomous pipelines: track cross-source agreement - how often does your agent agree with other label sources like investigation rulings or rule-generated signals? A drop in agreement tells you something changed before your outcome metrics do.

Watch for distribution shifts. If your agent suddenly flags 40% more accounts as suspicious in a segment that hasn’t been under attack, something in the agent’s inputs has changed. The shift won’t tell you what broke, but it tells you where to look.

None of this is overly complicated, it’s just that nobody has been asked to own it yet.

I recently went deeper on this on the Sardine blog: what each of these signals looks like in practice, how to catch drift before it shows up in your fraud metrics, and how to set up monitoring across different types of agentic workflows. 

Worth a read if you’re running agents and haven’t thought about this yet.

How are you tracking your AI agents right now? Hit reply and share your experience, we’re all learning this as we go.

In the meantime, that’s all for this week.

See you next Saturday.

P.S. If you feel like you're running out of time and need some expert advice with getting your fraud strategy on track, here's how I can help you:

Free Discovery Call - Unsure where to start or have a specific need? Schedule a 15-min call with me to assess if and how I can be of value.
​Schedule a Discovery Call Now »​

Consultation Call - Need expert advice on fraud? Meet with me for a 1-hour consultation call to gain the clarity you need. Guaranteed.
​Book a Consultation Call Now »​

Fraud Strategy Action Plan - Is your Fintech struggling with balancing fraud prevention and growth? Are you thinking about adding new fraud vendors or even offering your own fraud product? Sign up for this 2-week program to get your tailored, high-ROI fraud strategy action plan so that you know exactly what to do next. ​
Sign-up Now »

Enjoyed this and want to read more? Sign up to my newsletter to get fresh, practical insights weekly!

<