And welcome back in this section, we'll take a look at implementing a very, very inefficient transformer
here using PyTorch.
So let's get started.
So open notebook 60 and we'll begin to listen.
So firstly, a credit goes to Hirota Honda for creating this notebook here.
It comes from the people in images with 16 by 16 words.
And also, here's another implementation of it here, as well as this.
We use Tim, which is a Python library that allows us to load PyTorch models pre-trained apply to its
models.
It's quite useful, so I would encourage you if you want to get more hands on experience with using
the state of the art models and easily luring them into your own projects.
It can take a look at this tutorial here.
It's quite good.
So let's begin.
So firstly, you've got to install Tim, which either takes about three seconds and then this follow
the library takes about six seconds.
Next, we need to load the model.
So the model we're going to look at is a VAT based pad 16 to 24.
This was trained on image net and its resolution image size story is 12 to 24.
So let's begin.
So let's load this model and we use create model from time to create the model right here, and you
can see device basic CUDA means that we're using the GPU.
We can always make sure you're using the GPU here.
It should be set the GPU by default on this notebook.
This takes a little while because it has to download a model in the background.
So let's just be patient.
OK, there we go.
It's actually not that long, 21 seconds.
So now we just defined some parameters that we're using.
So transforms as well.
Next, we're going to load the image now labels, as well as a test image that we'll be using for this
demonstration.
So what we're going to do, we're going to run an inference on this image from Santorini, the Greek
island.
So let's take a look.
And it says Doom, OK?
So you can see we just this is quite simple.
You just got the image here.
We can witness the image to a pill, which will be loaded the image using pill.
And then we just created it and put it into a tensor here using the transforms that we use and squeeze
it and send it to the GPU.
And that's pretty much it.
We just get the output out of it there and then we get the ARG max of that output using top torch dot
org max, which is analogous to the MP nampai of Max function converted back to an end so we can use
it as an index and image net labels list that we created here was our dictionary that we created there,
and then we just showed results here.
So inference result is a domain and you can use this on any image you want.
So now let's dive deeper into depression transform.
So you would have seen this in the previous section where we take an image like this splits it up into
patches.
We're using 14 by 14 patches here and 14 by 14 is 196, which is where we have in one and 196 here.
Then we we.
So this is the patch embedding each patches encoded into one by six and 768 vector here.
And we as convolutional layers to create this embedding.
So there are some conflicts involved here.
And then we next, we combine it with the position embeddings here and then we have a class toolkit
as well.
So it's one more added to this.
It's 197 in total and it's an equal 197 that this is adapted each that is 768.
So this is 197 by 768 matrix that we passed to the transformer encoder and it outputs the encoding here,
which is the same dimensions anyway.
And then we pass to the MLP header, which does the final prediction.
That's pretty much transformer, in a nutshell.
So the second look at how we actually split these into patches, you can just remember what this is,
how it's done here, and this is where we get the embeddings from now.
You can just print this out there.
You can see the image.
Tensor size was 224, which 224.
24.
The tree and patch embeddings here for the image is 196 by 768, as we saw in the diagram above.
Now we can visualize what the patches actually look like for that image, even though we sort of above.
Let's just generate this nice patch image here using mod plot lib and you can see this.
It's each of these are split up into these little cells, and that's what we use for the inputs into
the bits and transformer encoder.
Next, we need to add the positional embeddings, which are trainable or learnable, I should say.
So we to do that, let's just take a look at the position of that vector here and now we can see.
Let's visualise some of the.
The embedding similarities, so you can take a look and see what that looks like.
I'm not entirely sure how to interpret this chart here.
They all look very similar to me, but it's probably a month early show.
Maybe I should look into it before explaining it wrongly to you guys.
So make a note of that and come back to this.
So next, we need to make a transformer input here.
So this is what creates that class token here.
So that's what we add to the zero vector.
So we get two one nine seven six one plus 14 by 14.
So let's do that.
And then we just added it there.
So that's quite simple.
So no, this is a closer look inside of the transformer, deeper look inside of the transformer and
code itself.
So you can see what's happening here.
You can see the vector is input here and then we have different processes going on.
You can see it's split up now into these into final detailed factors here matrices.
And then we just go to the parallel attention heads.
This is an important block here, and I'll put it here.
And then we have the fully connected layers here which have formed the MLP head, which predicts the
final class here.
So let's take a look at the series of Transformers, so we were using 12 different, sorry, 12 different
Transformers and series here, so we can print this out here.
You can see Transformer one, sorry, 02 11, that's 12.
Now we're going to see what the attention part that is transformer creates what it looks like and visualize
it.
So let's take a look.
Firstly, at the multi attention had block, which is this here?
And you can see the input to that encoder is as what we expected one like a seven by one by 768.
Next, we can take a look at the fully connected layer that expands terror dimension.
And you can see what it looks like here.
So this is 768 is now replaced by 3:37.
Next, we need to split QQQ V into multiple of vectors for the multi-headed attention.
So let's take a look at that.
And now we need to create the attention matrix and that it's visualized like that.
So now we can visualize the attention matrix for all of the patches.
And you can see what they look like there.
OK, so now this is a multi layered perception head.
So this produces a vector or a vector that has a probabilities for each class or basically just normal
probabilities with a score.
You can see if each class here and you can see five to the edge here.
Let's run this and generate a total of one five three.
It is the class with the highest score.
You can see it.
And then we just normalize it to get the probabilities out of it.
And we know that that's the domed 538 corresponds to doom class here.
I'm not sure what number this is, but it's probably something that looks similar to doom.
So that's it for Vision Transformers.
This was a lesson that went in depth into the attention parts of the vision transformer and into the
individual blocks at the end.
Next, we'll take a look at how you can train your own efficient transformer in.
So stay tuned for that lesson.
Thank you.