Hello, everyone, and welcome to this new and exciting session in which we shall be building our own
model from scratch.
In the previous section we saw how this transform our model, which previously was used for NLP.
Tasks could be used in computer vision with proper preparations.
And so in this section we'll see how to convert or create patches of this image and then carry out those
linear projections and pass this output from the linear projections into the transform encoder to then
create or to then train and enter and model, which learns how to say whether and input that image is
that of an angry person, a happy person, or sad.
Given that we'll be working with 256 by 256 images if we have to split up these images into 16 by six
and images, then we would have 256 different images.
Because here if we have 256 here and then break this up, you're into 16 by 16.
So you have 16 pixels by 16 pixels, then you would have to go you have to do this 16 times horizontally
and 16 times vertically to form a 256 by 256 image.
And so from here you would have 256 different patches like this one.
And so what we'll use to create this patches will be this extract patches method right here.
Now this extract patches takes in the image, it takes in the sizes, takes on the stripes and the rates
with also the pattern.
So let's look at how this works from here.
We have this picture here or we have this output which will help us picture how this works.
But before looking at that, let's look at this arguments.
We have the sizes or we have the images.
It's a 4D tensor.
We have this sizes which specifies the patch sizes.
So in our case we have 16 by 16 patches, so our patch size will be 16.
We also have the stripes which tells us how much we should shift while creating these patches.
And then we have the rates which will better understand with some figure.
So let's, let's now get back to this year and then we see that we have been we yeah, we pass in the
sizes of one, three, three, one.
So this list, we pass it in here we got this from your so you see we told it must be this list where
we have one the size row, the size column and one but since the size row, equal size color equals
16, then here we would have 16 by 16.
Now, in the case where we have three three, it you could see your let's have this here, you could
see your let's take this off.
You could see that we have this three by three.
So your on line with our example where we're working with 16 by 16 pixels here they have three by three
pixels.
And then the next one you have the strides five by five.
Now you will notice that let's scroll this way.
You will notice that when this box or when we want to get the next patch, we shift or we go five steps
to the right and then get this next position or this next patch right here.
So you can see after shifting, we get to this.
So originally we had this image here and then we're trying to get this different patches.
Now the patches are what we have in starts, we can see and then again we shift five steps downward.
We go downward again and then we get to this.
You will see that if you can count this one, two, three, four, five steps.
And then we get to this and then we shift again and so on and so forth.
So that's how we obtain all this three, this four patches right here.
Now the next thing we will look at will be this rates.
Now, the rates again, you have this one and this one, and then you specify this two rate values here.
Now we'll look at how we understand this two rate values by looking at dilation or by looking at a dilated
convolution operation.
We can see this visualization from this medium post by C called Sun, where unlike with the usual convolutions,
where when we have this filter which passes through the image, you see we have this three by three
filter, which is compact.
Here you see there's no spacing between each element of the filter, but with the dilated convolution,
you see the spacing here.
So.
Is kind of similar to what we have here as let's get back to documentation.
As with this, if we decide to have patches where there is some spacing between the pixels, then we
could specify how much space we want with this rate right here.
Now the pattern here is set to valid, and so this means that if some pattern needs to be done to match
up with the extraction, then it will be done automatically.
Now, that said, we could copy this all out, copy this simply, and then we piss it out right here.
So here we have this and then we'll call this patches.
So we have this patches and then we'll pass in our test image right here.
There we go.
We have test image.
That's fine.
And then here we have 16 by 16.
Now this is the patch size, so we could do well to put that in the configuration so we could, we could
add that to the configuration and the strident to is 16 because we want to let's get back here, the
strider.
Let's see why we need this trying to be 16.
Now you see that we're interested.
If we have this kind of image we're interested in, we are not interested in skipping this part, so
we do not want to keep any parts.
We want to have this.
Then we want to have this next.
You see, this will want to do.
Then we want to have this next.
And this next obviously will be pattern year and year.
So account for the space here and then we want to have this next and so on and so forth.
So because we want to have this will make this value to be the same as this, so that when we have this
three by three patch, we skip three steps and move to this next and so on and so forth.
So that said, if you want to have this kind of compact patch extraction, then you want to have this
this the value here for the stripe, the same as the size.
So with that now let's configuration patch size, we have that, the 16 by 16, that's fine.
And then now let's run this here.
Let's also run, let's run this here, this image test image.
Then that's fine.
We run this, we get this error.
Let's check out It must be four dimensional.
Okay, So what we'll do is instead of taking this test image, we will do expand DBMS.
So we create this extra dimension and then we add this to the zero two axis.
So we add this extra dimension, we run that again.
You see, we have our patches.
Now look at what we're going to have when we print out the patches, shape, patches, dot shape, what
do we get?
See, we have this 16 by 16 by 768.
Now let's explain why we have this.
Now recall that we have a 256 by two, 56 by three image, meaning that we're going to have three channels
like this, 256 by 256, then by three.
Now, since we are dealing with patches, I think this is preferable.
We should take this at once.
So this is 16 by 16 patch, 16 by 16 patch.
Yeah, we have this 16 by 16 patch.
And then, yeah, again we have this 16 by 16 patch right here.
So now we have this 316 by 16 patches.
And then given that each and every one of those patches is 16 by 16 and 16 by 16 is 256, you'll see
that if we pick a given patch like your 16 by, if we pick a 16 by 16 patch, then this third dimension
will be several hundred and 68 simply because for each patch we have 256 pixels per channel.
So the year we have 16 by 16 256 pixels which make up this patch.
Now this other one, 256, and this one, 256.
And now if you sum this up, it gives you 768, and that's how this value is gotten.
Now, to plot this out, we are going to go through each and every patch.
So we'll take this both vertically and horizontally.
We create the subplots 16 by 16 because we will have 16 by 16 different subplots.
And then for each subplot we have, this image is here we pick IJ, we pick a given patch out of this
256 patches.
We pick that patch and then we're going to reshape this because when you pick this patch, you're left
with this 768 year.
When you pick the patch, you're left with this pixels, which have all been flattened out to this 768
dimensional vector that we calculated here.
And so now what we need to do is reshape this back into a 16 by 16 by three pixel, obviously 16 by
16 by three.
We give you 768.
And so that's it.
Let's run this now.
You could always increase this figure size or reduce it, depending on or how you want to view this.
So run this.
This is what we get is output.
There we go.
We see this image which initially was compact.
Now we've broken this up into several patches.
Now, before we go on to create our patch encoder, which is this whole section right here, what we'll
have to do is convert or reshape this patches here.
So recall that in this paper after doing this here, that's after creating this patches, we have to
take each and every one of them and this will be considered as one element of the sequence.
So here we have nine patches and here we have nine values which have been passed, all nine different
inputs or vectors which will be passed here.
And so if we get back here, since we have 256 of this, then we'll need to reshape this such that each
and every one of these will be considered as part of our whole sequence.
So here would print out, let's print out or let's let's reshape that.
So we have patches, equals reshape, and then we have the patches and then this will give us patches.
Does a patch dimension patches shape zero, then the next one with negative one and then yeah, we'll
have 768.
Now how do we put negative one or we explicitly write to 56?
So this means that we have decided to put this now into 256 sequence length tensor.
Now with this let's print out the patches, shape and you see what that gives us.
So here we have 2 to 6.
Now if we put negative one year, it's going to automatically give us this 256 and that's it.
So we we have this oh, let's, let's rerun this so we get that difference.
So you see here two six times 16 and then 256 and that's it.
So each and every one of those 256 now will be passed into our transformer.
Now, we could modify this print, the plotting of the image.
So let's let's have that and then we'll say we have patches shape one.
So we take 256 and then here we have 16 by 16 and then we have simply a plus one.
Okay, So we have that we could take out this K now we do not need this again, take that off and then
yeah, we have this eye here which we now reshape into this and this should be fine.
So there we go, let's run this again.
And that's it.
We get the exact same output, which is normal since we just restructured the inputs.
So with this we see that we are now set to create our patch encoder layer and the special encoder layer
will be similar to the kinds of layers which we've been creating.
So we could just copy this here, which is that and then paste it out here.
So we'll call this our patch encoder layer.
There we go.
We have Patch Encoder and your patch encoder.
So this patch encoder will be responsible for, first of all, converting our image into patches and
then carrying out the projection and adding the positional encoding patch encoder.
That's it.
And now here we have this linear projection which will create your So let's call this linear linear
projection.
And we could do this with our dense layer.
So we have this dense layer.
Let's take that off.
And then we could select our hidden size dimension or embedding dimension.
Now, since our input is already let's write this out here, we have input of let's take this off or
batch dimension or let's, let's take the back dimension of we have inputs of 256 by 768 already.
So this is our hidden size dimension for now.
But what we could do is we could convert this into 512.
So let's do just that.
We could let's no, let's, let's take 1024.
Let's make the model bigger.
But this is too many parameters already, and it would take much time to train.
So let's, let's, let's stay with this.
So we stick with that and let's get back to the code.
We're going to specify our embedding dimension or let's say in size, as is in the paper.
So our hidden size, we specify the hidden size and that's it.
We're going to pass this in your we have hidden size.
There we go.
And we do not need this batch.
Now we have the linear projection.
We have this hidden size.
Let's leave it this way for now.
So we have this call method.
We should take this input and that will be it.
We'll have this calculation which we saw already here.
So we'll have this patches.
Let's copy this from here.
Okay, so we get the input, we get this input.
Now this input will be getting here.
We wouldn't need to do this expand DBMS because the when training we already have the batch dimension.
So we'll just have X Let's call this in now.
Let's, let's, let's have that as X.
So we have that X and that's fine.
Then we'll do the reshaping.
That's it.
And then from this reshaping we'll have our output output will be this patch, which has been projected
to this new dimension.
And so you would have output and it's self, uh, linear projection which takes in the patches.
Then to make this value dynamic, we're going to have let's take this off, we're going to have the
patches shape negative one.
So this value we have here is the last dimension of our patches.
So we have that and we get that last dimension.
Now that we have the set, we now ready to add up this positional embeddings we have here.
So we will add this different positional embeddings here onto our linearly projected patches.
We should also note that we are not going to take into consideration this class embeddings which I mentioned
in the paper.
As in practice, this is not really important.
We have your look as Bayer from the Google brain team who says the main aim of including this was to
reproduce the exact transformer network but on image patches.
And so in practice we are going to stick to this linear projections.
The positional embedding now will be constructed using TensorFlow embedding layer.
So here we have positional embedding as described in the paper embedding or encoding.
And then here we have the embedding layer.
Now let's check out on how this embedding layer is constructed.
Or first of all, what is about now this embedding layer as described to FCS layers, embedding turns
positive integers into dense vectors of fixed size.
Now, getting back to the paper at this point, you see we have this linear projections here we have
this linear projections which need to be added up to the position encoding, which is this once.
Now the linear projection we're going to get from your if we take if we include a batch dimension,
we will have an output batch size by number of patches.
So let's call this NP number of patches because obviously we have these patches here and each patch
is going to be a vector.
So it's a number of patches by that last dimension, which in this case is 768.
So you could always fix this hidden dimension.
These are hidden dimension, let's say batch by NP by hidden dimension.
And then here, let's take this off.
Let's take a batch of one.
So if we take a batch of one, we would have one for the batch dimension.
Then the number of patches would have will be 256.
Because if we have 256 by 256 image and we break this up into 16 by six and pixel patches, then we'd
have 256 different patches.
So here's what we have for this one.
And now with the embedding layer, what we'll be able to get will be another tensor which is like this
in the output of that embedding layer.
And so as we can see here, it takes these indices and then turns them into the dense vectors, which
we are interested in.
Now, the arguments we have here, the input dimension output dimension initializes regular, as we
can see here.
And then we can see this example where it converts, for example, this number four into this 2D vector,
this index 20 into this other 2D vector.
And then from this example, which we could copy out and test in our code, let's test this out.
Let's add this here.
So we have this, which we could test out.
So you better you see better how it works.
You see, the model takes us input this integer matrix by dimension, by input length, and then give
this other output.
Now, in our case, since we're working with image data, we do not really have this vocabulary we should
talk about here.
And so instead of this vocabulary we replaces by the number of patches.
Now, if this were a natural language processing, then this will be the total number of words that
our model can treat.
And so if we could have a vocabulary of, say, 300,000 words, and that's what we're going to pass
in as this input dimension.
But as we said here, we're going to take this to be our number of patches.
Now, this input dimension is, as I say, the dimension of the dense embedding.
So in this case, they put in the value of 64, which will change to 768 shortly.
Now, that said, we will have this model which is defined.
Take that off.
We have this input.
You see, the size of this input is 32 by ten input.
There's a batch dimension and here are the different indices.
So we have this input and then we're going to print out the shape from here.
So let's run this.
There we go.
You see, it takes in this and then outputs this one now as a vector.
So basically what we have is we could have let's take this off.
What we have is we have some indices.
Let's take this off.
We have some indices.
Let's say we have like in this case, they have ten indices.
So we have this, this, this this is seven, eight, nine, ten.
Okay.
So here we have this ten indices and then we know the batch dimension.
So we just have this ten indices here.
So this ten indices now could be projected into its two dimensional version, where this one will be
represented by a vector.
So just like the patches.
So you have a patch represented by this vector, this one presented by this vector.
So this is no longer represented by an index.
Say let's say because here this random values take value between zero and one 1000.
So it's no more value like, say, 900.
So this maybe what's 900 now will be converted into a vector.
And the size of this vector we see here will depend on this value we set here.
So you see this, this means that this, this, this one year now becomes a 64 dimensional vector.
And so each and every one of these becomes 64 dimensional vectors.
And that's why we go from this ten to now ten by 64, which you could see here if we ignore the batch
dimension.
So what we'll be doing in our case now, let's take this off.
What we do in our case is we have this 768 and then here we have 256.
And then the size of this input, which is going to be some random input, we take the size to be 32
by 256 since we have 256 patches.
So we have that to 56.
Let's take this.
We could take this off.
We could check out the the reason why we need the input, lend the documentation, but for now we don't
need that.
So we have 32 by 256 and then we expect to have an output which is 32 by 256, now by six by 768.
Let's modify this.
Let's take one.
By size of one.
So let's run this, which you have won by 256 by 768, and you see that it would match now with this
year.
With this year.
From the projections or from the predictions of the inputs.
Now, that said, let's simply copy this.
Here we have this layer, which we're going to copy and add that year.
So instead of this embeddings, we have this embeddings, there we go.
And then yeah, we'll pass them the number of patches.
So here we have a number of patches and then this one is this hidden size.
So here again we have our hidden size and that's it.
So now we have the hidden size we get here and then instead of having this or we have this plus the
positional encoding.
So we have solved that positional embedding.
We could call that embedding.
And then what this will take in will be an input of length, number of patches.
So we get back here and then we would have our embedding embedding inputs or we could use the TF range
and then we have the start, we could start from zero.
The are the indices.
So we have this and then we have the limit.
Our limit is going to be the number of patches because we want to get the these indices going from the
zero value to some end value, such that the length of this tensor we're going to create is going to
be such that is equal the number of patches.
So I'll limit your number of patches in that sense we could have this year.
So we have number of patches already from year so we could get self number of patches.
There we go.
We have a number of number of patches.
Okay, so we have our number of patches.
That looks fine and then we could make use of that year.
So we have that number of patches and then we are going to set our delta.
So we have Delta would take to be one.
Now obviously we said Delta to be two.
It means you're going to go from zero to number of patches.
Why?
It's keeping two steps and that's not what we want.
So we want a number of patches, elements in this embedding inputs.
So with this now we could pass an embedding, embedding inputs and that should be fine.
So we have this here, everything looks fine and we could take this off.
So now we return.
We could return our output simply.
Now, it should be noted that this embedding layer here is similar to the dense layer, but with a dense
layer.
When you have an input x, you see with a dense layer.
When you have an input, let's write it here for the dense layer when you have an input x.
That input X is multiplied by the weights and you add the bias.
So this is how we get the output.
But with this embedding layer is a simple matrix multiplication.
So when you take an input x, you multiply it by the weights and you get the output.
With that.
Now let's run the cellular and then we move ahead.
So we have this run and we could delete this two cells now.
Now, to test this out, we could define Patch Inc.
We have our Patch encoder Patch Encoder, which we've just created here.
So Apache Encoder and this takes in the number of patches and hidden size.
So let's specify those two five, six and seven, six, eight.
So we have this and then now we could have patch encoder and then passing an image and see what we get
is output your image.
We have this zeros.
There we go one by 256 by two, 56 by three.
So now when we pass an input image, we expect to get some output of shape, batch size by number of
patches by the hidden size or the number of hidden units.
And now that we've built this year up to the point where we have our embedded patches, let's go ahead
and build this transformer encoder right here.
As you could see, we start with the layer normalization, multi head attention, and then we add this
this input to this output.
We have the layer normalization again, then the multi layer perceptron.
And again we have this addition and then we get the output.
Now, to build a code for our transformer encoder, we're going to pass out this patch encoder here.
We remove this, we have transformer.
Uh, there we go.
And then this transformer is made up of first norm layer and then the second normal layer, let's have
this layer layer norm one it's our first norm layer layer, normalization line zation.
There we go.
That's fine.
And then we have our second layer normalization layer.
So we'll call this layer num two.
Now we've defined our layer normalization layers.
We could go ahead to define our multi head attention layer.
So we have we have multi head tension and there we go again, your multi head attention.
You could check out the documentation for this.
So here in the documentation you have the different arguments which you could pass in the type of values
which the multi head attention takes in.
So your we will specify a number of heads.
And the key dimension here, this key dimension is the same as the hidden size or the number of hidden
units which in our case we fix at 768.
We could also have the value of dimension drop out.
And this or the arguments.
Now, what this takes in, as you would see here, is a query value key and attention mask, which would
not use here.
Return attention.
Scores will not do this.
This training will not specify this, but what will specify will be this query and this value.
And then since we're not going to specify, the key is going to consider that the key is the or rather
it's going to consider that this value here is a key.
So it would be the same value that said, here would have the number of heads.
Let's take this off Malta head attention, number of heads.
And then we would also need the hidden size or here.
So we already have the hidden size, so we will leave that away.
It will replace a number of patches by a number of heads.
Okay.
So we have our multi head attention, a number of heads, hidden size, and then the next will be the
dense layers.
So as we saw in this model here, you see we have after the multi adaptation, we have Layer Norm two
and then we have this ML pillar which is made up of two dense layers.
So we have our self dense one, which is our dense layer, and then the number of units here will take
it to be hidden size, and then we'll specify the activation to be a glue activation as specified in
the paper.
So we have glue and then we repeat the same process for the second dense layer.
So here we have that and then two.
Okay, so that's it.
We specify this to dense layers and we now get our output.
So we should have that and then now we could go ahead and call.
So let's before going to that, let's modify this here for my encoder.
That's fine.
The name Transform our encoder.
There we go.
That should be fine.
OC So we, we build in our transformer encoder layer already and now we set to pass this inputs into
this different layers.
We have our input here.
X there we go, we have X, let's take this off and then x gets into the layer norm one.
So we have our output x x equals layer nom.
One which takes in this X here, takes our input X and then outputs this.
We could call this in.
And then we have this in put input.
We have our input which gets in here and then produces an output X.
And then from here we get into the multi head attention layer.
So we have X again this is multi head attention and this takes in X.
So now we have this input which gets here, gives us output, this output gets into this motor head
and gives us output.
Now that we've had this, remember from the paper that we have this addition to do so we have this add
layer which we need to specify here because after this output got into the multi head, we got this
output which now needs to be added to this input from the patches.
So we need to create this link here.
So that said, getting back to the code, we have our X now, which is ADD and this add will take in
this x.
And the input.
Then from year again, we take this and pass into our layer.
Norm.
So now we'll call this let's, let's call this x one because we'll need this x one again.
So we can't just be working with x, you understand shortly why we need this x one.
So we have this and we have your x one.
Okay, so we have that.
And that's fine.
Everything looks fine.
Then from here we take this and pass or take this output x one and pass into our layer Norm two.
So we call this layer norm two, That's it.
And then this takes in our x one produces output x one.
Then from here we take this and pass into our dense layers.
So we have this x one which is equal or dense one, and this takes in x one.
And again we have x one which takes in dense two oh sorry, which takes ax1 which is gotten from dense
two, which takes in x one.
And now let's say this produces an output x to remember this like our final output.
So let's call this output.
Let's, let's call this output.
Okay, So that's our output right there.
Now let's do this then for this output again, we're going to do this addition recall from the paper.
You see, after the addition, you get an output.
We get this output that we take this output from here and add onto this.
That's why you see, we had to create x one for this and then x two for this.
So let's get back to our code.
We have that and then here we have our output now, which is equal
and that's it.
And then it takes in our output and this x one here, All right, this x one after this addition.
So we need to have this as x two.
So let's change this to x two so we could make use of this Now in this addition, because if we if we
have this, if we maintain this to be x one, then the value we get in here would be this final x one
we got here.
And that's not what we want.
What we want is this output from yours.
So that's why we have to change this variable name.
So we have x two, this takes an x one, then that's it.
So from here, now we take this output and add it to this x one.
There we go.
We have that x one we get now our output.
So we could close this up and that's fine.
So we return our output and we have our transformer encoder right here.
Now we could go ahead and test this.
So we have our transformer, my encoder number of heads here in size, that's fine.
And then we also have this input right here.
We call that the input or the patches take this form.
So let's run this and see what we get.
We get an error from this.
Yeah, this is logical because we need to pass in at least two inputs.
So here, if you recall from the from this year, from multi head attention, you see this, we have
this two which must be passed and then this key is optional, this one optional and all this is optional.
So we must pass at least this two.
So that said, let's get back and modify that.
So in this multi adaptation, we have this and then we have X one, so that's fine.
So now we have that.
Let's run this again and see what we get.
You see, that's fine.
You see, we get exactly what we expect.
So this is the kind of input we get and yours, our output after going through the transformer encoder.
Now, although this works, make sure that you check your work or check this code and be sure that everything
is working as it's supposed to be.
So be sure that you pass in the right inputs and get the right outputs and so that you get the exact
output you are to get and not some outputs different from what your what you're supposed to have.
So that's it.
We have our transformer encoder and now we'll head on to building our model.
So your would have our model.
Let's get back up here, Let's copy this rest net from your complete network.
We had this so we just simply copy the same structure and then paste it out.
Your Now, though, there's not much difference.
We just the main difference here is that we have this model instead of layer.
So here we have a model and then here we call this white.
So, yes, our model, our vision transformer model and then your we change this, we have the vision
transformer and we'll call this our vision transformer.
Vision transformer does it.
So there we go.
We could take this off, take this off and all this off.
Then we build it in such a way that it takes in the number of heads, the hidden size.
Then from this encoder, the number of patches.
So we just need a number of heads, hidden size and number of patches.
So with that would specify number of heads, there we go, hidden size and then number of patches.
Okay, so we have that and then you're would have our patch encoder which will be defined.
So we have patch encoder.
And this patch encoder which we have in right here is simply what we created already.
So we just have Patch encoder and what does this take?
So we get back here and we look at this format, see number of patches and hidden size.
So that's why it's important for you to test this as you go on.
So since we've tested this, we show that this works when you pass in an input image.
So your patch encoder, a number of heads and hidden sides or write a number of patches and hidden sides,
so we have number of patches, then the hidden size which will get from your.
Now we have this patch encoder.
We have our transformer encoders recall that we have several transformers here.
C we have L transform my encoders.
So we get back here and then we define this transformer.
Let's call it trance encoders.
Okay, so this transformer encoders will be a list made of different layers.
And the length of this list will depend on the number of layers.
So we have your number of layers.
And then what we'll do is we'll say four underscore in this number of layers or rather in range number
of layers.
C we are then going to define the transformer encoder.
So we have transformer encoder, so we have the separate transformer encoders and then we'll specify
a number of heads and hidden size.
So you're again, we have a number of heads and then we have the hidden size.
The input gets through this patch encoder.
So here we have our patch encoder.
That is it.
This way we have X, let's call this input.
So here we have input in this patch passes to the patch or patch encoder and then produces the output
x.
Then once we have this output x, what we are going to do is we are going to look through or we're going
to go through each and every transformer encoder layer.
So here we have for eye in range of self dot number of layers, number of layers.
Let's define this here, number of layers equal this number of layers from your So we've set our number
of layers and then we take this in your So now we're going to go through this and then what we'll be
doing is we'll be having this X, which is going to be the output from this, our transformer encoder
layers.
So we have trans encoders I which takes in the different inputs X, So we'll do this for a number of
layers times and then for each of these we have trans encoders as we've defined already here, and we
pick the specific transformer encoder.
Now once we get the output from your C, once we get our output from your will, now obtain an output
which will flatten.
So from your word get, we'll say x equal.
We flattened this out.
Take an X and then from this we'll have this or MLPs block right here.
Oh, let's get back here.
We have MLPs.
BLOCK here.
Okay.
So we have this MLPs block right here.
That's after this.
Now, the paper, what they had done was since they had this class token, they picked those last or
rather the pick this first output here.
So this means that if we have let's take from your if we have an output by size one, let's take this
off one by 256 by several hundred and.
68.
Then out of this 256 year or 257 because do have this additional so other these 257 will pick out only
one from your and so we'll be left with that one which has 768 hidden units.
So basically this will be like having one by 768 output.
But now since we're taking all this into consideration, we'll just flatten this out and then pass this
to our M.P. head right here.
So that's it.
Let's get back to the code.
We flatten this out and then we, we, we specify the dense layers which make up the M.P. head.
So we have dense one dense layer.
This dense layer has set a number of units, let's say a 1000 or let's let's, let's give this dense
units here.
Let's let's have it as this argument.
So we have a number of dense units, okay?
So we have a number of dense units there.
Let's specify this number of dense units and then we have the Jello activation, S.F. and Jello.
Okay.
So we have that for dense one and now we should have or we should write this code for dense too.
So we have now dense two and then your same number of dense units.
You could always modify or modify this and that's it.
Next from here we have x equal dense one.
That's it.
Dense one.
This is self.
Okay, so here we have dense one which takes an X and then dense two which takes an x.
Now our final this our final output dense layer has to consider the number of classes.
So just as what we have been doing so far in this course, you would let's take this one this complete
block here.
So so far, what what we've been doing is we always ensure that our output dense layer has the number
of number of classes, number of units in this output.
So you're would simply copy this, scroll down back to our code and put this right here.
So that's it.
So let's call this dense three dense and that should be fine.
Okay, so here we have dense.
Take this off.
That's fine.
We have that.
Now let's run this.
That should be fine.
Let's go ahead and test this.
So here we have our model and we'll define V.
It's of course, v i t and the parameters, those arguments here, a number of heads here in size,
all of this, we're going to define this just down here.
Scroll down and then define this year.
OC So we have now this number of heads with say eight heads, hidden size 768 number of patches, 256
the number of layers, let's say four layers, the number of dense units is 1024 OC So there we go.
We have our weights and then this right now from the we pass in this input to one by 256 by two, 56
by three, and we should get reasonable output.
We could print out the summary so we summary and we could check out this model.
So you see here we have 283 million different parameters for this model.
Now let's reduce this.
We could say four heads, just two layers, and then this number of dense units here, we could take
128.
Now let's run this again.
And there we go.
You see the two layers.
This is the patch encoder.
And then you see the dense layers which follow this transform our encoder layers.
So now we have this model, we'll change this to the batch size.
So we have configuration and then batch size.
Now, the reason why we're doing this is because when we're training, our by size is going to be known.
So we don't want to have this.
You're like that.
So with that, now we run this again and run the summary.
Or.
Yeah, we need to change this to 32.
You see, when we change this, it doesn't work.
So here we have 32 symbols all by size.
That's it.
And now we can go ahead to compile and train our model.
Our model and our training.
Not that you wouldn't get the best results, because obviously the whites need very large data sets
or even extra large data sets to perform as well as a confidence.
And so when working with the whites generally, we want to train on a very, very large data set and
then later on fine tune on the smaller data set towards the end of the epoch.
We still get this error and the source of this error is the fact that since we've fixed the batch size,
that's if you get back up here in our training, if you see if you look at this here, we fixed this.
And so since this isn't dynamic now we have a data set which has been broken down into batches of 32.
And now towards the end, you may have a batch of, say, for example, eight.
Obviously, because the dataset, the dataset isn't necessarily divisible by 32.
That is, a number of elements you have isn't necessarily divisible by 32, and so you have a remainder.
And so when you have this remainder and that you fixed this year, you should get an error because now
you've told the model to always use at this position a value of 32.
So to avoid that, instead of doing the patches shape we're going to do to have that shape so we have
the shape.
Before we had patches, we got the shape and that was it.
But now we are using TTF dot shape, so we call it the shape method from TensorFlow.
And then you're we're going to pass in our patches.
So once we have those, once we have our patches pass in, we now select this batch dimension and that
should be it.
So let's run this again.
You'll see that even from here we could modify this.
We could put 32 and it'll still work.
Fine.
Okay, let's see.
So.
So with that now let's go ahead and compile and then restart the training.
The model is training.
And as you could see, it starts to stagnate around this 44.4% accuracy and 44.16% validation accuracy.
In the next section we would fine tune a bit, which has been trained on a very large data set.