The hardest conversation to have with a product team is about fraud scoring latency. They want sub-10ms. The risk team wants a model that checks device history, account graph relationships, and behavioral time-series features. Those two requirements pull in opposite directions, and neither side is wrong.
The checkout abandonment data on latency is real. Payment processors have measured it consistently: every 100ms of additional authorization latency corresponds to a measurable drop in conversion. The numbers vary by merchant category and user demographic, but in mobile checkout flows the sensitivity is acute. A fraud score that adds 40ms to the authorization path is not a neutral technical decision. It has a revenue cost that someone will eventually compute and complain about.
At the same time, the accuracy cost of cutting model complexity is also real. Lighter models typically rely more on simple features like velocity counts and static rule flags. Those are faster to compute but more brittle. Sophisticated attackers have learned to operate below velocity thresholds specifically because those thresholds are widely known. A model that only looks at features an attacker can profile and game will be gamed.
Here is how we think about resolving this in practice.
Where the Latency Budget Actually Goes
Before optimizing, it's worth mapping where time is actually spent in a scoring call. Most teams are surprised by the breakdown:
- Network round-trip (client to scoring service): 2 to 8ms depending on co-location
- Feature retrieval from data store: 3 to 15ms depending on cache hit rate and store type
- Model inference: 0.5 to 5ms for a gradient boosting model; 2 to 15ms for a neural net depending on depth
- Response serialization and network return: 1 to 3ms
The total is often dominated by feature retrieval, not model inference. Teams spend weeks optimizing model architecture to shave 3ms off inference while leaving a cold Redis lookup that costs 12ms sitting in the hot path.
The first optimization pass should always be the data access layer. Precomputed features stored in an in-process cache (or at minimum an in-region cache with sub-millisecond p99 latency) typically cut more time than any model change. For features that can be precomputed asynchronously, batch updates at account-event triggers (login, address change, payment method update) rather than computing at score time are worth the complexity. An account's 30-day velocity counts don't need to be recomputed from scratch for every transaction. They can be maintained as a running counter updated on each event.
What "Model Depth" Actually Means Here
The argument for lighter models in the name of speed often conflates two different things: model complexity and feature complexity. Those are separable.
A gradient boosting model with 500 trees over 30 precomputed features will typically score in 2 to 4ms. A deep learning model that computes behavioral embeddings on-the-fly from raw session event sequences will take 20 to 80ms depending on sequence length and hardware. The latency difference is not primarily the model architecture. It's whether features are precomputed or computed at score time.
Behavioral embeddings are valuable, but they don't need to be computed at authorization time if they can be maintained asynchronously. We maintain per-account behavioral embeddings that update after each significant session event: a login, a profile change, a transaction attempt. The scoring call reads the current embedding from cache. Inference is fast. The embedding depth is not lost, it's just moved from the critical path to the background update path.
This is the core architectural principle: separate the feature computation pipeline from the inference pipeline. Keep the inference pipeline as thin as possible. Move all expensive computation to the asynchronous feature update path where it can run without a latency budget constraint.
The p99 Problem
Average latency is the wrong metric for authorization path scoring. The metric that matters is p99 or p99.9, because individual high-latency outliers in the authorization path directly cause user-visible failure (timeout, authorization error, spinning wheel).
A scoring service that averages 8ms but has p99 at 200ms will still cause visible problems for 1 in 100 users. In a high-volume payment flow, that's not an edge case. It's a daily occurrence at meaningful scale.
p99 latency is usually dominated by a few things: cache misses (a feature that should be in cache but isn't forces a database roundtrip), garbage collection pauses in managed runtimes (JVM and Python both have this problem), and resource contention during traffic spikes.
For cache miss handling, the pattern we use is stale-while-revalidate: if the cache entry is expired but present, serve the stale value and trigger an async refresh. A score computed from a 5-second-old behavioral embedding is almost always better than a timeout waiting for a fresh one. For fraud scoring purposes, a very recent feature value is rarely necessary. What matters is not having completely cold cache entries for active accounts.
GC pauses are harder to solve without switching runtimes. For latency-sensitive components in Python, preloading models into memory and keeping the scoring function allocation-free (no new object creation in the hot path, only in-place updates) reduces GC pressure significantly. For JVM-based services, explicit GC tuning for low-latency workloads (G1GC with max pause target set aggressively) and heap sizing that avoids frequent major collections are standard practice.
Feature Selection as a Latency Tool
Not all features contribute equally to model accuracy. Feature importance analysis will typically show that the top 10 to 15 features contribute the majority of predictive signal. Features ranked lower than 20 or so are often adding minimal accuracy while requiring computation, storage, and cache slots.
We ran a feature ablation study on our own model earlier this year. The top 12 features (velocity counts, device age, account age, behavioral regularity score, and a few graph-derived features) accounted for roughly 90% of the model's AUC contribution. Features 13 through 40 added the remaining 10%. Features 41 and beyond added noise that was actually slightly negative for p99 accuracy on the fraud-positive class.
The practical result: trimming from 60 features to 15 cut feature retrieval time by about 40% and reduced cache memory requirements substantially, with minimal accuracy impact. The model was also simpler to explain to the risk team, which matters for regulatory review purposes.
We're not claiming 15 features is the right number for every model. The right number depends on the signal density of each feature for your specific transaction population. But the exercise of running a rigorous ablation study and measuring the latency-accuracy tradeoff at each feature count cutoff is worth doing. Most teams haven't done it.
When to Accept the Latency Cost
Some transactions warrant a deeper, slower score. A new account making a high-value transfer to a new payee should not be evaluated by the same lightweight model as a repeat coffee purchase from a 3-year-old account. The risk profile and the acceptable latency budget are different.
The pattern that works here is tiered scoring: a fast first-pass score for all transactions using only cached features and a lightweight model, followed by a deeper evaluation that blocks the authorization for a short additional window (say, an additional 30 to 50ms) only when the first-pass score is in an ambiguous range.
This keeps the vast majority of transactions on the fast path. Only ambiguous cases get the deeper treatment. The user experience impact is limited to the transactions that were already uncertain, not the full transaction volume.
The tricky part is calibrating the ambiguous zone. Too wide, and you've added latency to too many transactions. Too narrow, and you've given up the benefit of the deeper model on cases where it would have helped. In practice, this requires measuring the distribution of first-pass scores and identifying the range where model confidence is genuinely low rather than guessing.
The Accuracy Cost You're Actually Paying
The reason it's worth pushing hard on the engineering to preserve accuracy is that the cost of accuracy degradation is not symmetric. A false positive (blocking a legitimate transaction) shows up immediately as a customer complaint and a conversion metric drop. A false negative (passing a fraudulent transaction) shows up as a chargeback 60 days later, and often at a higher dollar value than the typical declined transaction.
Teams that cut model depth purely to hit a latency target often underestimate the false negative cost because it's delayed and harder to attribute. The chargeback arrives months after the decision to simplify the model, and the connection is rarely drawn explicitly. The latency optimization looks like a win in the short term.
The right framework is to measure the full accuracy cost of each latency optimization, including the expected change in false negative rate on the fraud-positive class, before shipping. Not every optimization will hurt accuracy meaningfully. But some will, and knowing which ones in advance is worth the analysis time.
