At most one email a week. Unfollow anytime with one click.
Transcript
The full narration of this video. The current sentence is highlighted while the video plays; click any sentence to jump there.
Show transcript
Introduction
Here is the final result, a face cut into 64 small patches, like words in a sentence, then processed by the vision transformer we wrote from scratch, complete with an attention map showing the relationships between patches. Let's get started. Hello, welcome to episode three of the vision transformer-based deepfake detection series. Transformers were originally created for text and became the basis for language models like GPT. In 2020, researchers at Google showed that images can also be treated like sentences, cut into patches, and then processed with self-attention. That is the vision transformer or ViT. Today, we are writing a mini ViT from scratch so that we truly understand how it works.
Step 1: Patch embedding
Create a file named NMIT_mini.py. The first component, patch embedding. The trick is simple. Convolution with kernel size and strides equal to the patch size, 16 pixels. Each 16x16 patch is converted into a single 192-dimensional vector. Flatten and transpose transform the result into a sequence of tokens. For an image of 128 pixels, there are 64 patches.
Step 2: Transformer block & self-attention
The second component is the transformer block. It contains layer normalization, multi-head self-attention, and an MLP with two layers and gelu activation, each with residual connections. Self-attention allows each patch to look at all other patches and weigh which ones are relevant. For example, the eye patch can pay attention to the other eye patch. We also return the attention weights for visualization. The third component is the mini classification head.
Step 3: CLS token & positional embedding
There is a CLS token. A special vector placed at the front of the sequence and tasked with summarizing the entire image for classification. Then positional embedding. Since self-attention does not know the order, each patch position is given a learned position vector. After passing through four blocks, the CLS token enters the two-class classification head, determining whether it is real or fake. Now create the file check_underscore E6IT.py.
Step 4: Face as 64 patches
First, we look at how a 128-pixel face is cut into an 8x8 grid, 64 patches of 16 pixels each. For E6IT, this face is a sentence with 64 words.
Step 5: Tensor shape & attention map
Then we check the tensor shape at each stage. Image 1x3x128x12, becomes 64 tokens with dimension 192. Plus the CLS token, making 65 tokens. After one transformer block, the shape remains the same, with a 65x65 attention matrix, and the output has two classes. Total parameters of E6IT mini is 1.94 million. Finally, the attention map maps the CLS token to the 64 patches. Since the model is not trained yet, the pattern is still random. In episode 6, we will see the attention map from a trained model.
" Done!
Done. We have written the vision transformer from scratch, including patch embedding, CLS token, positional embedding, transformer block with multi-head self-attention, and checked the tensor shape at each stage. Training E6IT from scratch requires a lot of data. So in the next episode, we will perform fine-tuning of the pre-trained E6IT model to detect real faces and deepfakes. Currently working on a thesis, dissertation, or AI research in the field of deepfake detection and image forensics? Cerdas Inovasi Digital helps implement deep learning, from face detection, vision transformer, comparison with CNN, compression resistance testing, up to attention maps and web applications, as well as opening deep learning courses with PyTorch. Want to master the entire series, from face datasets to deepfake checker applications? All 6 episodes, complete with PDF step-by-step guides and source code, are available for premium tutorial subscribers. Subscribe starting at Rp69.000 per month, save 28% with the 12-month package, or pay once Rp999.000 for lifetime access. Want just one class? Buy a single class with lifetime access. Subscribe now at the premium page cerdasinovasidigital.com Contact our WhatsApp, or visit cerdasinovasidigital.com Thank you for watching, see you in the next episode.
Ask the class material
Type a question or keyword; we find the part of this episode that explains it and jump the video there.
My notes
Notes are saved with the current video time and are only visible to you.
Questions & answers
Stuck on a step? Ask here; the instructor and other learners can answer.
No questions yet.
Your instructor
Cerdas Inovasi Digital Team
Software developers, BIM & 3D engineers, and AI practitioners
53 video classes1,286 episodes
We build software, BIM models, games and AI automations for clients every day, and turn that hands-on work into step-by-step classes: every step is shown on screen and comes with a PDF guide you can keep.
