Detected in the test set
25 frauds remain missed
Alert precision
49 frauds out of 53 alerts
Legitimate payments flagged
In the final test period
Later test transactions
56,746 held-out records
WHY THIS MATTERS
53 Review Alerts, with Both Error Types Visible
Fraud accounts for only 74 of 56,746 test transactions. A model that clears every payment would look highly accurate but catch no fraud. I therefore selected by Average Precision and reported the actual review workload, detected frauds and missed frauds at one fixed alert threshold.
EVALUATION DESIGN
Train on the Past, Select on Validation, Test on the Future
Training · 170,235 transactions
Fit on 342 frauds and the earlier legitimate payments. Remove exact duplicate rows before splitting; keep timestamp cohorts together and fit preprocessing only on training data.
Validation · 56,745 transactions
Use 57 frauds to compare four classifiers by Average Precision. Select Logistic Regression, then choose its alert threshold by validation F1.
Test · 56,746 transactions
Evaluate the locked model and threshold on 74 later frauds. Preserve the natural class imbalance. Final test labels do not choose the model or threshold.
FOUR VIEWS OF THE BENCHMARK
Detection Performance and Model Selection
Inspect the outcomes.
See how the model was selected.
The Model Detects 66.2% of Test Fraud Cases
On the chronological test set, the selected model detects 49 of 74 fraudulent transactions, corresponding to 66.2% recall.
25 fraudulent transactions remain undetected. The result demonstrates the coverage limitation at the selected alert threshold, despite only four false alerts.
Detected and missed fraud
Alert Precision Reaches 92.45% on the Test Set
The confusion matrix separates correctly identified fraud, missed fraud, false alerts and correctly unflagged legitimate payments across 56,746 test transactions.
53 alerts across 56,746 transactions equals 9.34 reviews per 10,000 payments. Of those alerts, 49 match recorded fraud labels and four are legitimate, yielding 92.45% precision.
Transaction outcome matrix
Logistic Regression Leads on Validation Average Precision
Logistic Regression records validation AP 0.7723, ahead of LightGBM (0.7636), Random Forest (0.7588) and XGBoost (0.7573). Its final-test AP is 0.7439. The selection rule was highest validation AP, not highest F1 or model complexity.
Average Precision measures how well the model ranks rare fraud cases. It is a score from 0 to 1, rather than a percentage of correct payments.
SHAP Explains the Main Contributors to Model Scores
V4, Amount and V14 have the largest average absolute SHAP values in the 1,000-observation validation explanation sample. The chart shows which inputs move the model’s risk score most, not a ranked list of real-world causes of fraud.
Amount is the transaction amount. The V features are anonymized components whose original business meanings are unavailable.
RESULT INTERPRETATION
A Useful Review Trade-off, with Remaining Evaluation Work
Ranking quality
Final-test AP 0.7439 and ROC-AUC 0.9818 describe ranking. At the selected operating point, recall is 66.22% and precision is 92.45%. Those are different questions.
Repeated-feature sensitivity
Exclude test feature vectors seen earlier: 55,026 transactions remain. AP is 0.7440 and ROC-AUC is 0.9815 using the same model and threshold.
Next validation
Use a longer, more recent cohort and a defined review budget or fraud-cost policy. Class-weighted scores need probability calibration; the dataset lacks cardholder IDs and label-arrival timestamps.
WHAT I BUILT
Reproducible Fraud Evaluation
with Transparent Error Analysis
I built the shared Python benchmark, temporal splits, selection policy and report generation, then exposed saved results in Streamlit without retraining. CI checks metric reproducibility, temporal boundaries, threshold selection, report integrity and dashboard behavior. The historical test demonstrates an evaluation workflow; it does not establish automatic blocking performance or financial savings.
Reviewed 22 October 2026 against the project README and linked evaluation reports. Figures describe the documented dataset or test run.