Picked from the software, topic, and level of “Class Hybrid ML Network Intrusion Detection #2: Data Traffic Preprocessing: Encoding, Scaling, and Class Imbalance”.
At most one email a week. Unfollow anytime with one click.
Transcript
The full narration of this video. The current sentence is highlighted while the video plays; click any sentence to jump there.
Show transcript
Introduction
Here is the final result, 41 raw features turned into 122 features with values 0-1, and the U2R class with only 52 samples was increased to 3,000 to avoid being ignored by the model. Let's start. Hello, welcome to episode two of the IoT network intrusion detection series with Hybrid Machine Learning. Raw traffic data cannot be directly input into the model. There are text features, value ranges that differ greatly, and highly imbalanced classes. Today, we will properly address all three of them.
Step 1: Encoding, log & scaling
Create a preprocess.py file Protocol type, service, and flag features are text, so we will convert them using one-hot encoding. Each value becomes its own column with a value of 0 or 1. Training and test data are temporarily combined to ensure the columns are the same. Features like byte and duration have ranges from 0 to billions, so we compress them using log 1 plus X. Then, minmax scaler transforms all features into the range 0 to 1. And importantly, the scaler is only fitted on the training data to prevent information leakage from the test data.
Step 2: Oversampling minority classes
The oversampling function handles class imbalance. For every class with fewer than 3,000 samples, we randomly duplicate samples from that class until it reaches 3,000. This way, the model cannot achieve high accuracy by simply ignoring the smaller class. Oversampling is only performed on the training data, never on the test data.
Step 3: Checking preprocessed results
Create a preprocess_underscore file and run it. 41 raw features become 12, 3 protocols, 70 services, and 11 flags, plus numerical features. 10 features with wide ranges are transformed using log, for example duration, srcb test, and dstb test. After oversampling, R2L increases from 995 to 3,000 samples and U2R from 52 to 3,000. While the larger classes remain unchanged.
Step 4: Before & after preprocessing
Let's look at the graph. The histogram of raw srcb test is almost entirely stacked at 0 with a few values up to 1.4 billion. After log and minmax, the distribution spreads across the range of 0 to 1, making small differences between connections visible. The right diagram shows the R2L and U2R classes, which are now sufficiently large to be studied.
Done!
Done. We have already performed one-hot encoding, log transformation, minmax scaling without leakage, and oversampling of the minority classes. In the next episode, we will select the most informative features using mutual information and PCA. Currently working on a thesis, dissertation, or research in the field of network security and machine learning? Cerdas Inovasi Digital helps implement intrusion detection systems, from network traffic preprocessing, feature selection, random forest and XGBoost, up to autoencoder for zero-day attacks and monitoring dashboard, as well as opening Python programming courses for electrical engineering and computer science. Want to master the entire series, from intrusion datasets to hybrid systems that are ready to be monitored? All six episodes, complete with PDF step-by-step guides, datasets, and source code, are available to premium tutorial subscribers. Subscribe starting at Rp69.000 per month, save 28% with the 12-month package, or pay once Rp999.000 for lifetime access. Thank you for watching, see you in the next episode.
Ask the class material
Type a question or keyword; we find the part of this episode that explains it and jump the video there.
My notes
Notes are saved with the current video time and are only visible to you.
Questions & answers
Stuck on a step? Ask here; the instructor and other learners can answer.
No questions yet.
Your instructor
Cerdas Inovasi Digital Team
Software developers, BIM & 3D engineers, and AI practitioners
53 video classes1,287 episodes
We build software, BIM models, games and AI automations for clients every day, and turn that hands-on work into step-by-step classes: every step is shown on screen and comes with a PDF guide you can keep.
