At most one email a week. Unfollow anytime with one click.
Transcript
The full narration of this video. The current sentence is highlighted while the video plays; click any sentence to jump there.
Show transcript
Introduction
This is the final result, a comparison of accuracy and training time for eight models, from one decision tree up to XGBoost and LightGBM, algorithms that are favorites among data science competition champions. We also see how boosting learns from mistakes step by step. Let's get started. Welcome to episode 9 of the Machine Learning with Python series. Ensemble Learning is a technique that combines many weak models into one strong model, like asking for opinions from many experts. There are two main approaches: one that trains many models in parallel and then aggregates their votes, and boosting that trains models one after another, where each new model focuses on correcting the mistakes of the previous one.
Step 1: Preparing data and test function
First, we install two additional libraries with pip install XGBoost LiteGBM. Then create a file called ensemble.py. Our data is created with Make Classification, 2000 samples, 20 features, and only eight that are actually informative, making it sufficiently challenging. We also prepare a test function that trains the model, calculates accuracy on the test data, and records the training time.
Step 2: Bagging and Random Forest
We start from the basics. One decision tree as a baseline. Then bagging, which is 100 trees each trained on random samples, and random forest that adds random feature selection. Look at the accuracy, combining many trees directly increases accuracy by several percent compared to a single tree.
aa 3. AdaBoost, Gradient Boosting, XGBoost, LightGBM
Now boosting. Ada boosting gives more weight to data that was incorrectly predicted. Gradient boosting trains each new tree to predict the residual, or remaining error, of the combination of previous trees. XGBoost and LightGM are highly optimized versions of gradient boosting, with regularization and much faster algorithms. As a result, XGBoost and LightGBM achieve the highest accuracy, with training time still under one second.
Step 4: VotingClassifier and comparison chart
We can also combine models of different types with voting using CVR. Here we combine logistic regression, random forest, and XGBoost, using soft voting, which averages the probabilities from all three. Then we plot all the results, accuracy on the left and training time on the right.
Step 3: Boosting learning curve
How does boosting learn? Create a file called boosting.py Gradient boosting has a method called staged predict, which gives predictions after the first, second, and so on trees. We plot the training and test accuracy for each number of trees, with two learning rates. A small learning rate learns more slowly but is more stable. After a certain point, the test accuracy stops increasing, adding more trees just wastes time.
Done!
Done. We have already compared bagging, random forest, boosting, gradient boosting, XGBoost, lightGBM, and voting. For tabular data, boosting usually gives the best results. But remember, all these scores are not yet tuned. In the next episode, we learn how to evaluate models correctly, from confusion matrix to ROCAUC. Want to master machine learning and data science? Cerdas Inovasi Digital offers Python, machine learning, and AI courses from scratch, model prediction development services, data analysis, and AI application for business or research, as well as selling complete source code of this tutorial. Want to master the entire series, from machine learning introduction to deploying models to the web? All 18 episodes, complete with PDF step-by-step guides, are available for premium tutorial subscribers. Subscribe starting at Rp69.000 per month, save 28% with the 12-month package, or pay once Rp999.000 for lifetime access. Want just one class? Buy a single class with lifetime access. Subscribe now at the premium page cerdasinovasidigital.com Contact us via WhatsApp, or visit cerdasinovasidigital.com Thank you for watching, see you in the next episode.
Ask the class material
Type a question or keyword; we find the part of this episode that explains it and jump the video there.
My notes
Notes are saved with the current video time and are only visible to you.
Questions & answers
Stuck on a step? Ask here; the instructor and other learners can answer.
No questions yet.
Your instructor
Cerdas Inovasi Digital Team
Software developers, BIM & 3D engineers, and AI practitioners
53 video classes1,281 episodes
We build software, BIM models, games and AI automations for clients every day, and turn that hands-on work into step-by-step classes: every step is shown on screen and comes with a PDF guide you can keep.
