Hello, everyone, and welcome to this new and exciting session in which we are going to treat transform,
learning and fine tuning.
Transform learning can be applied in several domains like computer vision, natural language processing
and speech.
In order to better understand the usefulness of transfer learning, we have to take note that deep learning
models work best when given much data.
And so this means that if you have a data set of only 100 data points, then you are most likely going
to have a poorly performing model.
But what if we tell you that it's possible to train your model or to train a given model on, say,
a million data points and then use that model or adapt that model such that you can now train it on
this very small data set such that you start getting very great results.
This is very possible with transfer learning, and that's why we shall be treated in this section at
this point.
One question which may be going into your mind is how is it possible to train a model made of, say,
a million data points and then use that same model on a smaller data set, which is obviously different
from this data set with about 100 data points?
The answer to this question lies in this figure we have right here.
Notice how you have this image here, this image of this truck.
And then we have this model which takes this input and then produces some outputs right here.
This model or the kind of model we'll use for this image tasks are generally the calf nets as we've
seen already in this course.
And with the calf nets, we generally have two main sections.
The first section is the feature extractor.
And then as we go towards these final sections here we have the classifier.
And so the very first thing the calf that we want to do will be to extract low level features.
And then as we go towards the end, we start or we focus on extracting more high level features.
Notice how your with this, let's pick out this feature right here or this feature map.
You see that we pick up this edges.
You see for other feature maps, we actually filtering out some low level features or extracting this
low level features from our input.
Then as we get towards the end, we get more high level features like, for example, whole portions
of the image like the tire.
And then after this we generally have a classifier which now permits us to pick between a set of options
which one the model thinks this image is actually now, because the calf net works this way, it means
that if we have two data sets which are similar, then we could build some sort of feature extractor,
or we could build a model which will extract features from this very large data set.
And then because these weights have been tuned or have been trained such that the extract features correctly,
then when we pass in this very small dataset, this section of the model will do just its job, that
is, of extracting useful features from this dataset.
And you see that because these two datasets are similar, is going to do a great job.
And so we will not need a very large dataset in order to extract features from this small 100 size dataset
right here.
And so in fact, what we are saying is we have a model, let's say we have this model, then we have
this feature extractor unit and then we have this classifier units, which we've seen already.
Generally, this starts after the flattening of the global average pulling.
And then here we have some dense layers.
While with this we have a continent or some convolutional layers with some max spool and batch norm
layers.
So what we say now is we have this our small data set which are going to pass in here and then we'll
get at this point.
So it's here that we're going to get this output from our feature extractor unit.
And then since generally we have, for example, or we pre-trained this model, not the word pre-trained,
because we are somehow using this for the first time in this course we are pre training or we pre trained
pre that we did the training before we pre trained the model on a large or relatively large dataset
like for example image net.
Let's let's let's suppose that we pre trained on some large dataset with 1 billion images then this
unit year has learned to extract features from whatever image you give it.
And so now when you come with just 100 images, what it does is it extracts those features.
And then since in the other level of this classifier we have a different setup for the pre training,
We now have to modify this classifier.
So this means that if before let's suppose that let's suppose that before we had after the global pooling
we have say 100 unit dense layer and then 1000 output dense layer right here, Then in our case where
we have just three outputs, what we'll do now is we'll simply replace this here would take all this
off and then now we may pass this say directly to the to the three output dense layer or we pass this
first to say 128 and then to this three output dense layer.
So from here we see that this new model will focus more on the classification while allowing the previous
pre-trained model to take care of extracting this features from our data.
Now, it should be noted that we generally use this concept of transfer learning when we have a very
small dataset.
And obviously since deep learning models perform best with larger set, we want to get the best out
of them.
And so we want to use the transfer learning.
When we have a small dataset, as we've said, and we have this model which has been pre trained to
extract this useful features from the those kinds of images.
So the simply means that we should have two similar data sets.
Another advantage of using our where can we transfer learning is that you get to gain in terms of training
compute cost that is this model which was pre trained may have been trained for say three days and then
now all you need to do is just get this pre-trained model and then apply transfer learning on your own
specific and smaller task.
And so when you're running on limited budget, you find that working with pre-trained models is going
to be really helpful.
Now, apart from transfer learning, we also have fine tuning, which is quite similar in the sense
that unlike with the transfer learning where we have this year, does this feature extractors weights
which are fixed And then during training we update the weights of this classification section with fine
tuning.
What we could do is also update the weights of this feature extractor section.
Now generally we start fine tuning from the top, so we suppose that this is the bottom here.
So we have the input and then we have this final layer.
So we will see that we start finding from this final layers going to this initial layers.
So yeah, we have this fine tuning process.
We could get this first, this top layer.
So that is we fix this layers here, we keep this layers fixed so the weights aren't updated during
the training process and then we update this weights.
While obviously we update this, we update this weights already.
But the difference is that these weights were.
Initialized from scratch.
That is, we randomly initialized these weights, whereas these weights are initialized from the Pre-trained
model.
Now this, this weights here from the Pre-trained model, but they are not trained.
Now, we could again, depending on the kind of results we get in, meaning that if we apply a fine
tuning to this top layers or to this final layers and we get better results, you see that what we could
do is we could keep fine tuning, so we could keep increasing this or this section of weights we could
update in the feature extractor unit.
So here would take this off, take this off.
And now we have this part which we can train.
So this is trainable.
This is trainable or not trained.
So we have something like this.
Now, it's also possible for us to go ahead and just say, okay, we're going to train all our model.
But why carrying out fine tuning?
If there's one thing you need to note is that you have to use a very small learning rate.
And the reason why you're doing this is to avoid disrupting this weight values which have taken very
much time to attain.
And so as we do the fine tuning, we're going to update this weights, but very slowly.
And by getting this by updating this with very slowly, we mean we're going to choose a very small learning
rate and then observe how this affects our model's performance.
At this point, we'll get straight to the code and we'll look at some pre-trained models.
So here you could see you get into this TensorFlow applications.
You have called next model.
That's net model efficient, net model efficient V two We've seen the efficient net already, inception
net mobile net, which we've seen mobile net V two, V three nice net and the famous rest nets which
we've seen already with the Vgs and the exception net.
So you're you have the choice of picking out any one of this.
We're going to go straight to the efficient net so we could have this your or you could get or you could
pick the efficient at V two, so could pick any one of these.
So you're going to pick the efficient net B for this one since he has slightly fewer number of parameters
compared to the rest net 50 and it outperforms the rest at 50 by a very large margin.
So we're going to pick this efficient net before.
And if there's one thing you can do with TensorFlow is simply the fact that you could use these models
without having to cut them out from scratch.
So as you could see here, we have this TensorFlow as applications efficient, net efficient before.
And then we just have this argument with this.
We get we're going to define our efficient model.
We pay this right here and we'll call this the backbone.
So since it's already now here, we're not going to include the top recall that as we had seen here,
we have this old model.
And then let's take this fine tuning part of we have this whole model.
And then what we are interested is in this feature extractor unit.
So we'll set that include top to false.
Let's get back here.
We have include top.
We're not going to include this, set this to false.
Let's take this off and then the weights have been pre-trained on the image net data set.
We want to take this input tensor.
We have the input shape now, this input shape, we have configuration.
So basically we have configuration.
The image size, the image size.
There we go in size by configuration.
And since we are not including the top, as we've said here, we're not going to take into consideration
this classes nor the classifier activation.
So we have that off and then we could decide whether to include the pulling layer or not.
And if we want to pick the pulling layer, then we'll have to specify what pull and layer want to work
with either the average or the max.
So with this, we are just going to take this off and specify that later on.
So here we have our backbone, we run the cell and then what we do to freeze what we call this is freezing.
So what we do to freeze this backbone such that the weights aren't updated during training is by simply
setting this here to to false or.
Senders trainable.
Parameter to false.
So backbone not trainable.
Equal false.
And that's it.
So this all we need to do to freeze our model.
Now, first on our model, the next thing to do is to add this other layer right here, all this other
layers.
So we'll go straight away.
Then here we're going to define this input with the image size.
Now, once we have this input, we now pass in or we have the backbone.
So we have the backbone here which has been defined and as parameters have been set to be frozen.
And then from here we have the global average pulling.
So we have global average pulling your.
Then we now have this dense layer.
Now the configurations we set this number of dense layer, 1 to 1024 and number of dense layers, 2
to 128.
So let's get back to this.
There we go.
We have that.
And then from here, we're going to have batch normalization layer.
There we go.
Let's copy this, paste it out here.
We have another dense layer now, this time around, the second one, that's fine.
And then finally, we have this dense layer activation, soft max.
So here we have our soft max activation.
And then this is a number of classes.
Here we have a number of classes and that's fine.
So let's take this off now and then run this cell right here.
We get in this arrow because we have to specify this this way.
That's fine and that's it.
So now we have this model, you see total parameters, 19 million, the trainable parameters, just
1.9 million.
And the non trainable parameters are 17.6 million.
So it means that the backbone itself is 17.6 million.
And then this additional parameters here come with the remaining 1.9 million parameters.
Now, with this, we have our model already set, you see, with minimal code, and we could go ahead
to start the training.
Then you're going to start by training our model again.
We compile the model and we run the training process.
Now training is over.
We could go ahead and evaluate our model.
So let's run this.
And what do we get?
We have close to 85% accuracy and 95.3% top K accuracy.
And this does slightly better than the previous models which we had worked with.
Let's go ahead and test this.
So it changes to model and we run this here incompatible found shape.
This.
We are going to resize this before passing into the model.
So here we have this image and just here we have let's say we have our test image which will resize.
So we use open CV to resize this image we pass in our test image and then we specify this in size.
So here we have in size, copy this and there we go.
Let's run this again and here's what we get.
See, we have the side output.
Now, let's go ahead and check this out here.
We run this.
Yeah, we have one.
Ms..
No.
Ms..
No.
Ms..
The second Miss.
Yeah.
We have two.
Mrs..
And that's it.
So out of the 16, we have to.
Mrs..
That is 14.
Divide that by 16 about 87.5% accuracy on this small batch of images which we took from the validation
dataset.
We go ahead and check out the conversion metrics.
Oops, let's run this, run this.
Let's get back and use our conviction metrics.
We get even better results.
But one thing we have to note here is that our dataset was not that small.
And so we may not see this change or this difference between training from scratch and using transfer
learning.
So what we'll do is we will take this, we'll take just say ten, that's 320.
So we'll have a data set of 320 data points and then we'll see the difference when we train from scratch
and when we train with a pre-trained model.
So right here, let's get back to this.
We take this down and we're going to use we could use any of this ones.
So let's let's pick the net quite simple.
Pick the net and then your we have in this we scroll down, we have training lost function the same
metrics the same, and here we have the net model.
So we run that and learn that model.
That's fine.
And so here we'll train on this small part and validate on the full validation data set.
We'll do this for just 20 epochs.
So after training for 20 epochs on that very small data set, you see the model doesn't perform well.
You see it doesn't even get up to 50% validation accuracy while the trained accuracy keeps increasing.
So the model's overfitting.
Now, from here, we are now going to change this.
So we're going to use our model, this pre-trained model.
So we have this year we just run this again.
So we initialize this parameters and then we'll compile the model.
This is model.
So let's, let's call this pre trained, pre trained model.
Let's change this name here to Pre-trained model.
There we go.
We have our pre trained model and we'll get the pre trained model summary.
That's fine.
Yeah, we have that.
And then here we have pre trained, pre trained model.
Okay.
So yeah, we're going to run this pre trained model compile and then we'll start with the training again.
So just knowing that we, we had the accuracy validation accuracy below 50% previously and now we're
going to check out our validation accuracy when working with the pre trained model.
But already one thing you could notice just after two epochs like your see, after these two epochs,
we are going to see this validation accuracy, which is already greater than 50, even from the first
epoch was already greater than 50.
It shows you the power of working with pre-trained models as we are now making use of those extracted
features to get this much more performing model.
So here the accuracy keeps increasing.
Now we're done with the training.
You could see that this model, which before couldn't cross this 50% mark for the validation accuracy
now is able to cross this mark as we'll see here.
You see, we have the validation accuracy of 71% while just training on 320 data points.
So let's get your let's run this.
And you could see what we have here.
You see it gets just above 70%.
So now with pre training, we get above 70% and we even got greater than 50% just from the very first
epoch.
And so what we could see from here is the very first thing is get as much cleaned data as possible.
And if you can lay hands on this, try some data augmentation.
And then from your if your dataset is still very small, then you could then apply transfer learning.
But if you have a relatively large data set, it will be needless applying transfer learning as training
from scratch should normally get you better results.
So here we could evaluate our model so you could see that this our pre pre trained model with just ten
out of 213 batches, which produces 71.3% validation accuracy.