At most one email a week. Unfollow anytime with one click.
Transcript
The full narration of this video. The current sentence is highlighted while the video plays; click any sentence to jump there.
Show transcript
Introduction
Here is the final result, the ridge and lasso coefficient path graph, which shows how regularization suppresses model coefficients, even making some of them zero. We also automatically search for the best hyperparameters using grid search and optuna. Let's get started. Welcome to episode 11 of the machine learning with Python series. In episode 5, we looked at the danger of overfitting. Now we learn how to overcome it with regularization, which is giving a penalty to large coefficients. Then we learn hyperparameter tuning, a systematic way to find the best model settings, from grid search to optuna.
Step 1: Overfitting, Ridge, and Lasso
We create a regularization .py file. We use the diabetes dataset, then add polynomial features of degree 2 so the features become 65. With this small dataset, ordinary linear regression directly overfits. The training score is high, but the test R squared is much lower. Ridge adds a quadratic penalty to the coefficients, and Lasso adds an absolute value penalty. Both improve the test score.
Step 2: Coefficient Path
Lasso has an advantage, it can make coefficients exactly zero, thus simultaneously selecting features. We count how many coefficients become zero. Then we plot the coefficient paths showing how each coefficient changes as the regularization strength alpha increases. In Ridge, coefficients gradually decrease towards 0. In Lasso, coefficients one by one drop exactly to 0.
Step 3: GridSearchCV
C, tree depth, and number of trees are hyperparameters, which are settings not learned from the data, but rather we set ourselves. Create a tuning.py file GridSearch CV tries all combinations in the list and evaluates them with cross-validation. We tune SVM for breast cancer dataset with the choices of C and gamma. The result, the best combination, cross-validation score, and test data score. If the combination is too many, GridSearch becomes slow. Randomized search CV only tries a number of random combinations from the range of values. We try on random forest with 4 hyperparameters, just 20 trials. The results are often as good with much faster speed.
i Step 3: Automatic Tuning with Optuna
The most intelligent way is Optuna. Install it first with pip install Optuna, Then create a file optuna underscore tuning dot py. We write an objective function V that receives a trial, suggests hyperparameter values, and returns a score. Optuna learns from previous trials to select more promising values next. We run 30 trials and see the best parameters.
Done!
Done. We have already used ridge and lasso to combat overfitting, read the coefficient path, and searched for the best hyperparameters with GridSearch CV, RandomizedSearch CV, and Optuna. Remember, all tuning is performed with cross-validation on the training data, the test data remains saved for final evaluation. In the next episode, we learn about feature engineering and feature selection. Want to master machine learning and data science? Cerdas Inovasi Digital offers Python courses, machine learning, and AI from the basics, model prediction development services, data analysis, and AI applications for business or research, as well as selling complete source code of this tutorial. Want to master the entire series, from machine learning basics to deploying models to the web? All 18 episodes, complete with step-by-step PDF guides, are available to premium tutorial subscribers. Subscribe starting at 69,000 rupiah per month, save 28% with the 12-month package, or pay once 999,000 for lifetime access. Thank you for watching, see you in the next episode.
Ask the class material
Type a question or keyword; we find the part of this episode that explains it and jump the video there.
My notes
Notes are saved with the current video time and are only visible to you.
Questions & answers
Stuck on a step? Ask here; the instructor and other learners can answer.
No questions yet.
Your instructor
Cerdas Inovasi Digital Team
Software developers, BIM & 3D engineers, and AI practitioners
52 video classes1,276 episodes
We build software, BIM models, games and AI automations for clients every day, and turn that hands-on work into step-by-step classes: every step is shown on screen and comes with a PDF guide you can keep.
