At most one email a week. Unfollow anytime with one click.
Transcript
The full narration of this video. The current sentence is highlighted while the video plays; click any sentence to jump there.
Show transcript
Introduction
Here is the final result. The original messy housing price data, full of missing values, typos, duplicates, and outliers, has been cleaned step by step, and then we wrapped it into a scikit-learn pipeline that can directly predict the price of new houses. Let's get started. Hello, welcome to episode four of the Python machine learning series. There's a saying in the data world: garbage in, garbage out. No matter how powerful the model is, it will perform poorly if the data is dirty, and data scientists spend most of their time on this stage. In this episode, we perform data exploration or EDA, handling duplicates, missing data, and outliers, converting categories into numbers, scaling features, and organizing everything into a pipeline to avoid data leakage.
Step 1: Creating dirty house data
We open the terminal and create a file named buat underscore data.py. We create 500 synthetic house data, including area, number of rooms, age, city, condition, distance to city center, and price. Then we intentionally inject real-world issues into the data. Some values are missing, the area of the house is filled with impossible numbers like 4 square meters and 3,000 square meters. City names are written inconsistently, and 8 rows are duplicated. The result is saved as rumah underscore kotor.csv. Now let's explore.
Step 2: Data Exploration (EDA)
Create a file named IDA.py. We read the CSV file and then check the info. There are columns with fewer than 508 data points, meaning there are missing values. The describe function shows the minimum area of 4 and maximum of 3,000, clearly unreasonable. The value counts for the city column show Jakarta written in many different ways. Lowercase, uppercase, even abbreviations of Jakarta.
Step 3: Boxplot to view outliers
Visualization makes the problems clearer. We draw boxplots for four numerical columns. The points outside the whiskers of the boxplot are potential outliers. Look at the area column, the box is squashed because it's compressed by several extreme values. The distance column is also skewed to the right.
Step 4: Clean duplicates, text, and outliers
It's time to clean up. Create a clean file .py. First, remove duplicates using drop duplicates. Second, tidy up city text, remove pasi with strip, standardize case with title, then replace Jakarta with Jakarta. Third, outliers. Area below 20 or above 1000 square meters is clearly incorrect input, so we remove it, but we keep the empty data. Remember, not all outliers are wrong, the final decision requires knowledge about the data. The result is saved to home underscore clean.csv.
Empty data has not been filled yet, and this is intentional. Filling empty data, scaling, and encoding should be learned only from the training data. If calculated from all data, information from the test data leaks into the model, this is called LA-AKG data. The way is to create a file pipeline.py. Split the data first into training and testing. Then arrange column transformer, numerical features are filled with median then standardized. Cities are changed with one-hot encoding because there is no order, and condition is changed with ordinal encoding because there is an order, from bad to very good.
Step 6: Pipeline and predicting new houses
All these steps we combine with a linear regression model in one pipeline. Just call fit on the training data, and each step automatically learns only from the training data. As a result, R2 is around 0.8. Pipeline can also directly receive raw new house data, even with empty age, then predict its price.
Done!
Done. We have performed IDEA, removing duplicates, tidying text, handling outliers, understanding strategies for empty data, encoding, and scaling, as well as using pipeline to prevent LA-AKG data. In the next episode, we will delve into the first model for regression, linear and polynomial regression. Want to master machine learning and data science? Cerdas Inovasi Digital opens Python, machine learning, and AI courses from scratch, model prediction service, data analysis, and AI application for business or research, as well as selling complete source code of this tutorial. Want to master the entire series, from machine learning basics to deploying models to the web? All 18 episodes, complete with step-by-step PDF guides, are available to premium tutorial subscribers. Subscribe starting at Rp69.000 per month, save 28% with the 12-month package, or pay once Rp999.000 for lifetime access. Thank you for watching, see you in the next episode.
Ask the class material
Type a question or keyword; we find the part of this episode that explains it and jump the video there.
My notes
Notes are saved with the current video time and are only visible to you.
Questions & answers
Stuck on a step? Ask here; the instructor and other learners can answer.
No questions yet.
Your instructor
Cerdas Inovasi Digital Team
Software developers, BIM & 3D engineers, and AI practitioners
52 video classes1,276 episodes
We build software, BIM models, games and AI automations for clients every day, and turn that hands-on work into step-by-step classes: every step is shown on screen and comes with a PDF guide you can keep.
