Hello, everyone, and welcome to this new and exciting session in which we are going to be looking
at how to fine tune an already trained transformer.
More precisely, vision transform our model using the hugging face library and TensorFlow to hug and
face.
Today is at the forefront of practical AI, and it permits practitioners around the world to build,
train and deploy state of the art models very easily.
It's also used by thousands of organizations and teams around the world.
So here we have a host of different tasks which we could solve with readily available hugging face models
like audio classification, image, classification, object detection, question, answering summarization,
text, classification, and translation.
In our specific case, we are dealing with image classification.
And if you click open right here, you see we have this image classification page where you could already
test an image on this white model here.
So this is the white base model with patch size 16 and the image size 224.
We also have this facebook d i.t based distilled model and pipe size 16 image size 224.
You could browse a host of other image classification models here.
As you could see here we have the different models sorted in order of number of downloads.
C 219,000 times this model was downloaded.
Now we'll be working with this video or with fine tuning this bit by Google right here.
Here we have the model description so you can have that.
We've seen this already in the paper, the intended uses and limitations, and then how to use this
model without any fine tuning.
So your you could pass in your image and then already run classification on this image using this white
model right here with for image classification model.
Now here you will notice that this is we suppose that we are dealing with a PyTorch model.
So we could check in the documentation here where we'll see this white model on the left side together
with many other different models which are available for free.
And you see here we have this white.
You could also check out the the i.t, the distillation transformers.
Let's get to d.
We should have g i t around this.
There we go.
Here the eit.
You see, you have the eit and this documentation right here.
So we also have the screen which we have seen.
We looked at the screen previously so we could check out the screen somewhere.
Your screen as well.
We are.
SWC, You have the swim channel from around here.
So that's it.
See the swim transformer we had seen already?
Now let's get back to our v v vision transformer OC.
So we have this vision transformer right here.
And you could your go through the, the whole documentation.
So we have the fit config with feature extractor looks like this so it becomes clearer.
There we go.
We have this feature extractor, the model with four masked image modeling with four image classification.
And you'll notice here we have two with model.
So the difference with this is this is PyTorch or model and then this is a TensorFlow model which we'll
be using now the t v model and tier V for image classification.
Now recall that you have your model which starts from the patch.
You have the patch.
Once you have the patch, you have the transformer encoder.
And then from here you take this, you have this NLP head and then you have your output right here.
Now with this TFV for image classification, we have all this full year.
So we go from this patch right up to the MLPs head, but with the tf V model, this part, this part
is not included.
So what you get will be only this outputs right here.
So with the tf V model, we just we have this whereas with the TF of it for image classification, we
have all this now apart from TensorFlow.
We also have the code for flags.
So you could check our flags with model flags with for image classification.
Now that said, we are now going to focus on fine tuning this vision.
Transform our model.
Don't forget to subscribe and hit that notification button so you never miss amazing content like this.
Before doing anything, we'll start by first installing this Transformers library right here.
So we have people install Transformers now.
That's fine.
We move on.
Let's go ahead to start with the fine tuning, but then let's get back to documentation and here we
have the overview.
We could check out those v config right here and you'll notice that all the different configurations
are basically what we've seen already.
So we have this hidden size 768 as default value number of hidden layers.
12 So you're just stacking 12 different transformer encoder blocks, number of attention heads 12 Intermediate
size 3072.
Now for this, 3072 actually is for the dense layers we have here.
Now the input you're getting from this normal or right from here, we have say one by 256 by 768.
Then this gets in and get to this point where we still have the same.
So up to here is the same and we expect that this output here should be of the same shape.
But then in this MLPs layer, we have two dense layers.
Now the first dense layer will convert this to one by 256 by 3072, and then the next dense layer would
convert this to one by 256 by 768.
So that's why they call this the intermediate size right here.
So this intermediate size, you can see dimensionality of the intermediate feedforward layer.
In the transformer encoder, we have the hidden activation glue.
He didn't drop our probability.
So you have some drop out here.
Attention, probably drop out attention, perhaps drop our probability, initialize our range layer
norm epsilon value to better understand this, we could get back to the layer norm documentation in
TensorFlow.
You see, epsilon here by default is one times ten to the negative three, and then we could see where
exactly it's been used.
So recall that with normalization we have x minus a given mean divided by a standard deviation.
Now, we we do not want a situation where this here is zero and then we have an infinite output.
So we generally add some epsilon right here.
Now, this epsilon by default, as you see here, is 10.001.
And in hugging face here it is one times 10 to -12.
Now it's only encoder.
So there's no encoder decoder.
That's why this is set to false image size to 24, the pipe size 16, number of channels three.
What are we going to add a bias into the query keys and values your true and code two Stride 16.
Now you can remember the use of the stride when we're trying to get the patches we want.
Once we have an image like this and that we have a patch size of 16 by 16, we'll move through 16 pixels
to obtain the next patch so that we have no space here.
So we have something like this actually.
So that's it.
We understand this with config.
You could check out your this usage of the v config.
So let's copy this code and get back here.
Um, there we go, get back here and then we have this code.
Pass it out your OC.
So that's it.
You see clearly you could easily create a V model without necessarily going through all this process
which we had done right here.
So this was just for educational purposes and means that if you want to build your own with your simple,
you just have to do this.
Specify the configuration that's v config and then you initialize the model and that's it.
So let's, let's run this and then let's also print out this configuration.
Let's look at that.
You see, there we go.
We could change this.
We could change, let's say we change this hidden size to some value.
144 So yeah, let's have our hidden size.
So you see, this is how you could change the hidden size to suit your needs.
So that's it.
You change this, that's fine.
You look at that and you see you now have this new configuration and that's it.
Now, for the next we'll look at this feature extractor.
The feature extractor is similar to what we have done already.
That is taking in the input, resizing it and then carrying out some normalization.
So that's it for the feature extractor.
You could check in the documentation.
This model here is a PyTorch model.
So let's go to TFS.
So this is for TensorFlow now.
So here you see, there we go, we have our t v model.
You could expand this parameters and your we have all those different arguments which we could check
out.
Now here you have this example of how we could use the TFV model directly without going through any
stressful process.
So here you see we have this first of all, we have the beta dataset and then we have this image which
is extracted this.
You could you could get this image from our own data set.
So that's it.
They're loading this from hogging face hub or better still hogging face data sets.
So that's it.
And then you have this feature extractor.
So that's a feature extractor.
We have this model and now not that you're the tier model is from pre trained.
So this means that we are going to use this model which is already been trained and you see this specifications
here with base patch 16 to 24 in 21 K Okay.
So that's it.
The inputs now pass through the feature extractor before then pass to our model.
So we'll see how to adapt this code so that we could fine tune our own model in TensorFlow.
And as we said before, this model different from the image classification model in that the outputs
here and not the final output classes, but this hidden states from the ten from the transformer encoder.
So let's call your maybe that's really an example.
Okay.
This an example here you see this output here that gives you directly Kat So model produce one of 1000
images in classes.
So that's it.
Whereas here we have this hidden, we have this hidden states.
Okay, we paste this out here, let's take this off, take this one off, and then we could get started
with building our own with model based off this hugging face to fit model.
So we wouldn't need this data sets here.
We already have our own data set.
That's fine.
We wouldn't make use of this feature extractor.
We have this model here.
We have our hugging face, let's call this hugging face model, and then let's just take all this off
actually.
So we have this we have this interface model, we have our tier model from pre trained and that's it.
Now we're going to define some inputs.
So we have your our input is equal the input layer and then we'll specify the shape.
So here we work with 224 by two, 24 by three and then using the TensorFlow functional API.
I will get an output of X, which we take in this hugging model.
Let's call this base model.
So taking the base model takes in inputs.
Let's call this let's change this to base model.
Okay?
We have the base model text in the inputs and then this this year we have base models.
So we have this method, this method which we call before taking in the input.
And so from here we have this base model of vertex and the input, and then now we have the output here
X.
Now when we run this, you see we download this 330 megabytes of pre-trained model.
Let's have this owl hugging face model here.
So we we get the inputs from your and then we have the outputs.
Let's call this output X.
So we have that.
Let's run this again.
We have a hugging face model now set and we still get this error.
You see, it's linked to the positioning of this inputs here.
So let's let's change this.
Let's say we have three by 220, four by 224.
We run this again.
You should see now that everything works fine.
See, it now works fine.
So this means that the inputs of this are base model year.
So hogging face model should be of this shape.
So it's three.
Let's take this here instead of being 224 by two, 24 by 224, as we usually do.
Now it's inset three by two, 24 by two, 24.
Now this means that we need an extra layer which will convert this into this before passing into our
base model right here.
So let's build that extra layer and we'll take inspiration from our resize rescale layer, which we
had built already.
Let's get to resize rescale right here.
So we have that.
We'll modify the recital scale specifically for this organ face model.
So here we have at this code resize scale for hugging face.
Okay, So we have this resize scale, we'll resize make sure it is 224 by 224.
So every image which passes here will be 224 by to 24 we're going to rescale.
And then after we scaling, we are going to permit take the value.
So we call on this permit here permit layer and the way we'll build this all the way we'll call this
permit layer will be such that we move this from this third position.
This is zero one, two, three, but by 224, by two, 24 by three.
So we move this from this third position to this first position.
So here we have three go into this position and then this one two shifts to the right.
So we have three.
This one goes here, three, one, and then this two comes here so that the output now will be batch
C, the batch remains intact by three, which has been shifted by 224 by 224.
So be careful not to do instead.
Two one.
This is, this is one, two and not two one, because here we have in height by width by channel.
So we want to change this to channel by height, by width.
Okay.
So that said, we do this here.
So we just do three one, two and that's it.
So after this input layer, before getting your call, this X, we'll take in our resize rescale and
that's it.
So this resize skill takes in the inputs and then here we'll pass in will be x.
Let's run this again.
We should get an error because when we permit it, it goes back to 224 by two, 24 by three.
So let's have this see we have that 224 by two, 24 by three as we used to working and then we run this
now and everything should be okay.
All we get in an error resize is not defined.
Let's run this.
Oh, let's make sure that's how we called it.
Let's go up resize.
We need to run this.
Actually, this should be fine.
Now.
That's it.
You see, everything is okay.
So now what we'll do is we'll pass in some input.
Let's pass in some input.
We have this test image right here, and then we have our model which takes in the test image.
Now we need to also convert this or rather add the batch dimension.
So let's expand the terms and take in the test image right here.
So we have that.
We run this and see what we get.
Let's add this to the zero axis axis zero, run that again.
And then we told that we expected this, but instead found this.
Now let's get back up here and we could change this to 256 by 256.
And then knowing that this resize will convert it back to 224 since our model takes 224.
So let's run this again.
And there we go.
Here's our output.
You see, we have the last hidden state, one by one, 97 by 768.
And as we scroll, we have this pull out output one by 768.
And then from here, we scroll down again.
Let's see if we have another output.
And that's it.
Okay, So this is what we get is output.
We have the the last hidden state and this puller output from the documentation where we had the parameters.
TF We model the parameters.
You see here we have this last hidden state, the puller output.
And then we told that these are the two outputs we will always get.
And then we know that this one is optional.
The hidden States is optional, but the last hidden state is an optional.
We get.
We always get the last hidden state.
We always get this puller output and then this attention's.
You're also optional.
So if you want to get this attention.
So all you need to do here is to specify config.
Remember the configuration with config config that output attention set out to true.
And so this means that by default this will be set to false.
And then for the hidden states we also repeat the same so config that output here in states we set it
to to true.
Now you're in the documentation to explain the difference between this puller output and this last hidden
state.
But getting back here, you should see the shape.
So you see this is just one of this year.
While this is all our full hidden states are full last hidden state one by one and seven by 768.
But this model or this hugging face with model was built, taking into consideration this class embedding
right here.
And so this means that if you want to carry out some classification is better off or you're better off
taking this final year, this final or this class embeddings final hidden state.
Now, if you want just those hidden states, we could specify this.
Pick out the the zero index.
We run this and we should get only the output or the last hidden states.
So that's it.
We get this last hidden states.
And then since we are interested only in that output corresponding to the class embedding and the class
embedded at the zero position here, the zero position we are going to take, we are going to we are
going to do this here.
We're going to take this.
We're going to take for the first dimension here, we take all.
And then for this next one we take we select the zeroth index, and then the next we take all.
So let's run this again and see what we get.
And that's it.
We have this output right here.
And now, since we've converted this hugging face model into a TensorFlow model, we could do or summary.
So let's run this and check out our model summary.
So that's it.
We have this model summary.
We scroll down and see 86 million parameters.
We have the input sequential.
The sequential is corresponds to the skill layer, right?
And then the slicing operator, which we have here, which permits us get our specific output.
Now getting back here, we will now add our final classifier.
So we have this and then we have we'll call this output.
That's fine.
Let's just call this output.
So we have this output takes in the dense layer, has the number of classes specified here, and then
we have our activation self max as usual.
So that's it.
We have this your output.
Okay, Now let's run this and see what we get.
We get in an arrow because we didn't pass in this x here, so let's run that again.
Now, as the model is training, just remember that the learning rate we use here is an appropriate
as we can't be using this type of higher learning rates when doing fine tuning.
So we have to change this and use some lower, let's say five times ten to the negative, say five.
Okay, So let's stop the training and then restart this process.
You see now that when we re initialize our model and then we modify this learning rate, you see the
loss drops now much lower than what we had before with a higher learning rate and the accuracy is already
at 75% and we are still at the first epoch.
So be careful when you fine tuning or when you're updating all the.
Parameters of an already trained model.
You have to make sure you use a very low learning rate.