At most one email a week. Unfollow anytime with one click.
Transcript
The full narration of this video. The current sentence is highlighted while the video plays; click any sentence to jump there.
Show transcript
Introduction
Here is the final result, the Confusion Matrix and ROC curve for the fraud detection model. In this episode, we prove that a model with 95% accuracy might be completely useless, and learn the right metrics to evaluate it. Let's get started. Hello, welcome to episode 10 of the Machine Learning with Python series. So far, we've used accuracy a lot. But accuracy can be very misleading, especially when the data is imbalanced, for example, fraudulent transactions that make up only 5% of all transactions. In this episode, we learn Confusion Matrix, Precision, Recall, F1 Score, setting the threshold, ROC curve and AUC, as well as cross-validation.
Step 1: Accuracy Trap
We create an evaluation .py file. The data is 3000 synthetic transactions with May Classification, and we set only 5% to be fraudulent. Now the trap is, Dumi Classifier that always guesses non-fraud. Its accuracy is 94.4%. Yet it never caught a single fraud case. This is why accuracy alone is not enough.
Step 2: Confusion Matrix and Classification Report
We train a real model, Logistic Regression, then create the Conf,usion Matrix. This table has four boxes. True Negative, normal transactions predicted as normal. True Positive, fraud detected. False Positive, normal transactions falsely labeled as fraud. False Negatives, fraud that slips through, and this is the most costly for the bank. The Classification Report summarizes Precision, Recall, and F1 for each class.
Step 3: Precision, Recall, and Threshold
Let's understand what it means. Precision answers: Out of all that were labeled as fraud, what percentage is actually fraud? Recall answers: Out of all actual fraud cases, what percentage is caught? F1 is the harmonic mean of both. Recall can be increased by lowering the threshold. Let's try threshold 0.5, 0.3, and 0.1. Recall increases, but Precision decreases. There's always a trade-off, and the choice depends on the cost of errors in the real world.
Step 4: ROC Curve and AUC
How to evaluate the model across all thresholds at once? With the ROC curve, which plots True Positive Rate against False Positive Rate. The area under the curve is called AUC. 1 means perfect, 0.5 means the same as guessing randomly. We compare Logistic Regression with Random Forest Then display the Confusion Matrix and ROC curve side by side. Cross Validation.
Step 5: Cross-validation
A single split of training and test data can be lucky or unlucky. Create a file called cross_validation.py. With stratified 5-fold cross validation, the data is split into 5 parts with the same class proportions. The model is trained 5 times, each time using a different part as the test data. The results are averaged and variance, much more reliable. Done. We have already proven the pitfalls of accuracy, reading the Confusion Matrix, understanding Precision, Recall, and F1, setting the threshold, using the ROC and AUC curve, as well as Cross Validation. In the next episode, we will learn to tackle overfitting with regularization and find the best hyperparameters. Want to master machine learning and data science? Cerdas Inovasi Digital offers Python courses, machine learning, and AI from the basics, prediction model development services, data analysis, and AI applications for business or research, as well as selling the complete source code of this tutorial. Want to master the entire series, from machine learning introduction to deploying models to the web? All 18 episodes, complete with step-by-step PDF guides, are available to premium tutorial subscribers. Subscribe starting at Rp69.000 per month, save 28% with the 12-month package, or pay once Rp999.000 for lifetime access. Want just one class? Buy the single class with lifetime access. Subscribe now on the premium page at cerdasinovasidigital.com. Contact us via WhatsApp, or visit cerdasinovasidigital.com. Thank you for watching, see you in the next episode.
Ask the class material
Type a question or keyword; we find the part of this episode that explains it and jump the video there.
My notes
Notes are saved with the current video time and are only visible to you.
Questions & answers
Stuck on a step? Ask here; the instructor and other learners can answer.
No questions yet.
Your instructor
Cerdas Inovasi Digital Team
Software developers, BIM & 3D engineers, and AI practitioners
53 video classes1,281 episodes
We build software, BIM models, games and AI automations for clients every day, and turn that hands-on work into step-by-step classes: every step is shown on screen and comes with a PDF guide you can keep.
