The conversation about risk score thresholds usually goes something like this: fraud team wants to block everything above 0.6, payments team says that will kill conversion, product team escalates because a power user got blocked on a Friday evening. Everyone argues for a week, someone sets it to 0.75, and the discussion repeats in two months when fraud losses tick up again.
This is a calibration problem, and it is mostly a process problem rather than a model problem. A good risk score does not tell you what threshold to set. It gives you a probability distribution over transaction outcomes that you then translate into a business decision under uncertainty. The translation is where teams consistently get stuck.
Here is how we think about threshold calibration when we are working with a new integration.
Understand what your score is actually measuring
Before setting any threshold, you need a clear definition of what the score represents. A risk score is not a binary fraud label. It is a probability estimate: the model's assessment that this transaction is anomalous relative to the account's historical behavior and cohort baseline.
A score of 0.85 does not mean "this transaction is 85% likely to be fraud." It means the transaction sits in the high-deviation range relative to the learned baseline. Whether high deviation equals fraud depends on what fraud looks like on your specific platform. On a peer-to-peer payment app, high deviation might be a first-time large transfer to a new recipient: legitimate 40% of the time, fraud 60% of the time. On a card-issuing platform, the same deviation score on a foreign merchant transaction might resolve as legitimate 80% of the time.
This distinction matters because it means you cannot port a threshold from one platform to another. A 0.7 block threshold on platform A could be appropriate or catastrophically over-aggressive on platform B depending on what 0.7 means in each context.
Use your labeled data to build a precision-recall curve
The most useful tool for threshold calibration is a precision-recall curve built on your own labeled transaction data. You need two ingredients: a set of transactions with known outcomes (confirmed fraud versus confirmed legitimate), and the risk scores those transactions received.
Plot precision (of all the transactions the model scored above threshold X, what fraction were actually fraud) against recall (of all the confirmed fraud transactions, what fraction scored above threshold X) for every possible threshold. The resulting curve shows you exactly what you are trading off at each threshold level.
In practice, the shape of this curve on payment platform data tends to look like this: recall is high at low thresholds (you catch most fraud but also flag many legitimate transactions), and precision is high at high thresholds (what you flag is very likely fraud, but you miss a chunk of actual fraud that scored below the threshold). The business question is where on that curve you want to operate.
The answer is different for block, review, and pass decisions. For auto-block, you want high precision: false positives here create friction, customer complaints, and potential chargebacks from legitimate customers who dispute their blocked transaction. For the review queue, you can accept lower precision because a human is making the final call. For pass, recall matters most: false negatives here become fraud losses and chargebacks.
The review tier is not optional
A common mistake in threshold setup is treating risk scoring as binary: block or pass. This throws away most of the value of a probabilistic score.
Transactions in the 0.4-0.7 range (rough guidance, calibrate to your own data) are genuinely uncertain. The model has detected some deviation from baseline but not enough to have high confidence either way. These are the transactions where a velocity check or a single additional signal could tip the decision. Auto-blocking all of them produces unacceptable false positive rates on most platforms. Auto-passing all of them accepts fraud losses that were preventable with a second look.
The review tier handles this. Transactions in the uncertain range go to a human queue or trigger an additional verification step (e.g., 3DS2 challenge, SMS confirmation, additional document check). The threshold for what goes into review versus what auto-passes is a business decision based on the cost of review labor versus the expected fraud loss from the uncertain-score population.
On the deployments we have run, keeping review queue volume below 1 in 150 transactions is achievable without sacrificing recall at the high-score end. If review volume is higher than that, the threshold placement needs adjustment or there is signal leakage in the review-tier boundary.
Segment before you calibrate
A single threshold applied uniformly across your entire transaction population is almost always wrong. Different transaction types, merchant categories, and account age cohorts have different base rates of fraud and different distributions of legitimate behavior.
Consider a card-issuing fintech with two primary customer segments: individual consumer accounts and small business accounts. Consumer accounts have predictable, relatively low-variance transaction patterns. Small business accounts show high legitimate variance: large irregular purchases, multiple merchant categories, spikes in volume at month-end. A threshold calibrated on the consumer population will over-flag small business transactions. A threshold calibrated on small business will under-flag suspicious consumer activity.
The way to handle this is to segment first, then calibrate separately per segment. At a minimum, you probably want separate thresholds for: new accounts (under 30 days old), established accounts, high-value transactions above some dollar threshold, card-not-present versus card-present transactions, and domestic versus cross-border transactions.
Per-account behavioral models help here because they carry some of this segmentation implicitly: the score for a new account is already being benchmarked against a new-account cohort, not against the full population. But threshold segmentation is still valuable on top of the per-account score because business rules about what is acceptable to auto-block differ by transaction type regardless of the model's confidence.
Track threshold drift over time
A calibrated threshold decays. The fraud population shifts. Your legitimate customer base changes. A platform that added a new product line six months after the initial calibration will see its score distribution shift because the model is now seeing transaction types it was not trained on.
The practical implication is that threshold calibration is not a one-time setup task. You should be reviewing precision and recall against confirmed outcomes at least monthly, more frequently if your platform is growing quickly or if you have introduced product changes that affect transaction patterns.
The signal to watch for is a creeping increase in false positive rate: more legitimate transactions scoring above your block threshold than historical baseline. This is usually not a model problem but a distribution shift: your customers are doing something new that looks unusual to the model because it was not in the training window. The fix is a threshold adjustment, not a model retraining (though if the shift is large, retraining is warranted too).
One thing to watch that surprises teams
False positives are not symmetric. Blocking a power user who does $8,000 per month in transaction volume costs more than blocking a user who does $200 per month. Most threshold calibration treats all false positives equally. They are not.
We are not saying that high-value customers should get a free pass on fraud detection, because they should not. We are saying that when you are evaluating the business cost of a threshold adjustment, segment by customer lifetime value and look at where the false positive distribution falls. If your false positives are concentrated in the high-LTV cohort, even a small adjustment that reduces false positive rate by 0.5% on that segment can have outsized revenue impact.
The contributing factors output on each score makes this analysis tractable. You can look at why high-LTV customers are scoring above threshold and determine whether the triggering features are genuinely anomalous for that account or whether they reflect a pattern that is normal for high-volume accounts but unusual relative to the broader population baseline. That distinction drives whether the fix is a threshold adjustment or a per-segment baseline recalibration.
Threshold calibration is ultimately an ongoing engineering process, not a one-time configuration step. Treat it that way, build the monitoring to track drift, and the false positive problem gets substantially more manageable.
