The brief

A consulting pitch to a bank's executive risk committee: should the bank deploy a machine-learning fraud score, and if so, at what threshold and with what automated action? Fraud is rare but mistakes are expensive in both directions. A missed fraud costs the chargeback and the customer's trust; a wrongly declined customer costs friction, review time and sometimes the customer. A model that flags nothing is 96.5% accurate and catches no fraud, so accuracy was ruled out as a metric from the start. Models were ranked on PR-AUC, with precision and recall reported at a stated operating point.

Testing it the way production works

• Forward in time. The data was split chronologically, never randomly, so the model is always judged on transactions that happened after everything it learned from. Every number below is out-of-time.
• Leakage removed. A screen of every column found chargeback_reason_code separated fraud perfectly (AUC 1.0000), because it only exists after a dispute is filed. It was dropped, along with cvv_mismatch, which was zero on every row.
• Strongest signal. Recent declines: the fraud rate rises from 0.56% at zero declines in the prior week to 48.9% at three. Missing values were left unimputed, since filling them with the median would erase that signal.

Choosing the model

Six candidates were compared under a selection rule written down before the test period was scored. Twelve engineered features (customer history, interaction terms, missingness flags) were tested with a 500-resample paired bootstrap; only target encoding was kept, and customer history moved the metric no more than a column of random noise.

The top three sat inside one bootstrap standard error, so the simpler XGBoost model (400 trees, depth 3) was retained. A simulation put it at 97% of the performance ceiling this dataset supports: the next gain comes from better data and labels, not more tuning.

The risk policy

Rather than an automatic block, the score routes each transaction into one of three tiers, with cutoffs set on validation data:

The policy touches 12.9% of transactions and covers 83.7% of fraud. At this extract's scale that means 2.4 analyst reviews a day and 0.6 genuine customers declined a day. The threshold works as a staffing dial: if review capacity is lower, the cutoff moves, not the model.

Limitations and the ask

Three issues had to be closed before go-live: one vendor-supplied feature carries 71% of the model and needs tracing to source; the fraud-loss to review-cost ratio was assumed rather than measured; and the fairness screen flagged the 25-and-under age band on too few alerts to clear or condemn it. The recommendation was a 90-day pilot in shadow mode, with no customer declined by the model until those are resolved.

My role

This was a group project for BUSINFO 717 at the University of Auckland. I helped build the model and the risk analysis.