At most one email a week. Unfollow anytime with one click.
Transcript
The full narration of this video. The current sentence is highlighted while the video plays; click any sentence to jump there.
Show transcript
Introduction
This is the final result, isolation forest and local outlier factor marking suspicious transactions in red, without ever being given any fraud examples. In the second part, we find shopping rules in a minimarket, for example, buyers of bread tend to buy jam, using the a priori algorithm. Let's get started. Hello, welcome to episode fifteen of the Machine Learning with Python series. We continue with unsupervised learning, with two very useful tasks in business. First, anomaly detection, finding data that is unusual, such as suspicious credit card transactions or machines that are starting to break down. Second, market basket analysis, finding items that are often purchased together, the basis of product recommendations and shelf arrangement.
Step 1: Credit Card Transaction Data
We create a file called anomali.py. The data consists of one thousand credit card transactions with two features, nominal in thousands of rupiah and transaction time. Normal transactions are mostly small to medium in value during the day. We insert 20 unusual transactions, with very high nominal values, or occurring at midnight. We save the labels only to check the results at the end, the model will not see them.
Step 2: Simple Z-score
The simplest way is to set a score, transactions with nominal values more than 3 standard deviations from the mean are considered anomalies. This method is fast, but only looks at one feature. Transactions at midnight with normal nominal values will not be captured.
Step 3: Isolation Forest and LOF
Isolation forest works with a clever idea. Anomalies are few and different, so they are easier to isolate. Trees often split data repeatedly, and points that separate quickly are considered anomalies. Local outlier factor compares the density of a point with the density of its neighbors. We set contamination to 2%, then check how many original anomalies each method captures. We plot the results side by side.
Step 4: Anomaly Visualization
Red points are the marked anomalies. Both methods capture transactions with high nominal values or midnight transactions, which cannot be done with a single feature z-score.
Step 5: Market Basket and Apriori
Part two, market basket analysis. First install the melten library with pip install melten, then create a keranjang.py file. We have 500 shopping basket records from a minimarket. We convert this data into a true-false table, with one column for each item. Then the a priori algorithm looks for combinations of items that often appear, with a minimum support of 5%. From these combinations, we create association rules.
Step 6: Association Rules: support, confidence, lift
Support is how often the combination appears. Confidence is the probability of buying the right item if the left item is bought. Lift compares it to chance, a lift above one means the two items are truly related. Our top rule, for example, is bread and jam with instant noodles, which can be used to place items together or create a promotional package.
Done!
Done. We have detected anomalies using z-score, isolation forest, and local outlier factor, and found shopping rules using a priori, support, confidence, and lift. In the next episode, we enter the world of deep learning, building a neural network with Python. Want to master machine learning and data science? Cerdas Inovasi Digital offers Python, machine learning, and AI courses from the basics, data prediction model development, data analysis, and AI application services for business or research, as well as selling complete source code for this tutorial. Want to master the entire series, from machine learning basics to deploying models on the web? All 18 episodes, complete with step-by-step PDF guides, are available to premium tutorial subscribers. Subscribe starting from Rp69.000 per month Save 28% with the 12-month package, or pay once Rp999.000 for lifetime access. Want a single class only? Buy single classes with lifetime access. Subscribe now on the premium page cerdasinovasidigital.com. Contact us via WhatsApp, or visit cerdasinovasidigital.com. Thank you for watching, see you in the next episode.
Ask the class material
Type a question or keyword; we find the part of this episode that explains it and jump the video there.
My notes
Notes are saved with the current video time and are only visible to you.
Questions & answers
Stuck on a step? Ask here; the instructor and other learners can answer.
No questions yet.
Your instructor
Cerdas Inovasi Digital Team
Software developers, BIM & 3D engineers, and AI practitioners
53 video classes1,286 episodes
We build software, BIM models, games and AI automations for clients every day, and turn that hands-on work into step-by-step classes: every step is shown on screen and comes with a PDF guide you can keep.
