Picked from the software, topic, and level of “Class Deep Learning CNN-LSTM Pneumonia Detection #1: Chest X-ray Dataset: Loading, Exploration & Data Splitting”.
At most one email a week. Unfollow anytime with one click.
Transcript
The full narration of this video. The current sentence is highlighted while the video plays; click any sentence to jump there.
Show transcript
Introduction
This is the final result, thousands of chest X-ray images of children, grouped into normal lungs and lungs with pneumonia, complete with statistics and visualizations. This is the first step of our series in building a hybrid CNN-LSTM model for early detection of lung diseases. Let's get started. Hello, welcome to the AI research tutorial series, developing a hybrid CNN-LSTM model for early detection of lung diseases based on medical images. Pneumonia is one of the leading causes of child mortality worldwide, and chest X-rays are the most common diagnostic method. In this 6-episode series, we will build an automatic detection system, from understanding the dataset, image preprocessing, CNN-Comparator model, hybrid CNN-LSTM architecture, training, up to clinical evaluation and cam grading. In this first episode, we will get to know the data. We open the terminal through the terminal menu, new terminal, and then install the libraries needed for this series.
Step 1: Installing the library
Torque, matemnis, openpethon, scikit-learn, and matplotlib. Matemnis is a collection of standard medical image datasets widely used in research, and one of them is pneumonia amnis, containing chest X-rays of children from the original dataset of over 5800 images. Create a file named data_underscore_lung dot py.
Step 2: Dataset loading function
Its content is a function called load that takes one part of the data, train, validation, or test, in the size of 64x64 pixels. When run for the first time, the data is automatically downloaded around 20MB, then saved in cache. We also define the class names, 0 for normal, 1 for pneumonia. This file will be reused in all subsequent episodes. Now create a file named exploration dot py.
Step 3: Size and data splitting
We load the three parts of the data and print their sizes, 4708 training images, 524 validation, and 624 test. This division is already provided, so our research results can be compared with other research. Each image is of type UIN8 with pixel values from 0 to 255. Notice the class distribution.
Step 4: Class imbalance
In the training data, pneumonia is around 74%, normal is only 26%. The data is imbalanced. This is important, a model that always predicts pneumonia already gets an accuracy of 74%. Therefore, later we will use class weights, and evaluate the model with sensitivity, specificity, and AUC, not just accuracy. The test data has a different proportion, 62% pneumonia.
Step 5: Viewing X-ray images
Let's look at the images. We take 6 random examples from each class. Normal lungs appear dark and clear because they contain air, whereas pneumonia lungs appear whiter and hazy, due to fluid or inflammation called infiltrates. This hazy pattern is what the model needs to learn. Finally, two statistical graphs, a bar chart showing class distribution, and a histogram of average brightness for each image.
Step 6: Image brightness statistics
Pneumonia images tend to be slightly brighter, but the two classes overlap significantly. This means brightness alone is not enough, we need a model that understands spatial patterns, which is CNN.
Done!
Done. We have installed the libraries, loaded the pneumonia dataset, checked the split and class imbalance, and viewed the visual differences between normal and pneumonia lungs. In the next episode, we perform medical image preprocessing, CLAHI to enhance contrast, normalization, and augmentation. Currently working on a thesis, dissertation, or AI research in the field of medical imaging? Cerdas Inovasi Digital helps implement deep learning, from CNN, LSTM, hybrid models, up to Grad-CAM interpretation and result writing, as well as opening deep learning courses with Python. Want to master the entire series, from X-ray datasets to evaluation and Grad-CAM? All 6 episodes, complete with PDF step-by-step guides and source code, are available for premium tutorial subscribers. Subscribe starting at Rp69.000 per month, save 28% with the 12-month package, or pay once Rp999.000 for lifetime access. Want just one class? Buy a single class with lifetime access. Subscribe now at the premium page cerdasinovasidigital.com Contact our WhatsApp or visit cerdasinovasidigital.com Thank you for watching, see you in the next episode. Thank you for watching!
Ask the class material
Type a question or keyword; we find the part of this episode that explains it and jump the video there.
My notes
Notes are saved with the current video time and are only visible to you.
Questions & answers
Stuck on a step? Ask here; the instructor and other learners can answer.
No questions yet.
Your instructor
Cerdas Inovasi Digital Team
Software developers, BIM & 3D engineers, and AI practitioners
53 video classes1,279 episodes
We build software, BIM models, games and AI automations for clients every day, and turn that hands-on work into step-by-step classes: every step is shown on screen and comes with a PDF guide you can keep.
