Hey.
Welcome back to the course in this section, we'll take a look at how we can use cameras to implement
transfer learning and fine tuning.
So go ahead and open the Notebook 21 and let's get started.
So in this lesson, firstly, let's load our libraries here and what we're going to do.
We're going to explore what trainable layers are.
Take a look at the weights and them and understand how we can freeze layers and carrots.
So firstly, let's create a very simple densely here with four nodes here, and then we use layer top
build to build that network.
And then by doing that, we can now explore, get the weights, get the number of trainable leads and
get a number of non-tradable widths.
Let's run this block of code.
And you can see in this layer, we have two widths which are trainable and zero that are non trainable
here.
So that's quite interesting.
Now note all we're all is or trainable with the exception of the batch norm.
It uses non-tradable weights to keep track of the mean and variance during training.
So now that you've seen that, let's take a look at this here.
Let's make a model with two layers here, and we're going to freeze one of them here.
So layer one, layer two for the dense, which we know is each this one is using really, this one is
using sigmoid activation.
And then we just use the sequential here just to combine them to get into the model.
And what we'll do, we're going to freeze the first layer and by freezing to firstly, we have to set
the dot trainable at that setting, the trainable width in this network to false.
And also, we can keep a copy of this weird Slater for reference.
So we just do layer one, get widths and we submitted a variable name here.
Initially, I want to wait values so we can we can train this network, so we're actually going to compile
it and treat it with some random data here just to make sure that, oh, it's 22 and change.
If we went to one, we set them to trainable equal force.
So that's finished trained because for a simple prototype here and know what we can do, we can get
the layer one.
We had snow, so we get to finally a one weights, which shouldn't have changed.
And then we can just check them here so we can check each of these little weights here and see if they've
changed or not.
And they haven't changed.
So that's good.
So what does that mean?
That means that, no, you're right.
You've understood how we can set different layers to trainable and how we can explore and freeze those
weights in the network while training.
So now let's implement chance for living.
So this is a transfer learning workflow, which you've seen before we instantiate a base model, Ludo
Pre-trained.
We onto it.
We freeze all the layers in the base model by setting all trainable to false.
And then we create a new top layer head of the model that's basically a fully connected head.
And then we just start training on our new dataset.
This is using the feature extraction method of transfer learning.
So let's load a base model with our pre-trained image net widths.
So the data set will be working with here is actually the catalyst, those data set that's available
from this package, called TensorFlow, underscored datasets which will input as two Ph.D.s, and we
can directly load this dataset here.
It's a number of datasets available here as well, and we can just run this split that we want right
here and we get our dataset here.
So we have a slow.
This will probably download the first instance you can see it downloading there, and the output you're
going to get is a number of it's going to print a number of training samples in the trends and then
into validation in the test.
Yes.
So let's wait for this.
It should finish shortly.
OK, so we have dataset loaded here and you can see the number of training samples.
We have nine thousand three hundred and five validation samples, twenty three hundred and twenty three
hundred as well for the tests.
So let's take a look at visualizing some of the images here.
So let's run this block of code and this how how this works.
Here's that you take.
You use a tick function here from the dataset that was loaded here.
That's this object ahead of the trends or tests of ideas, whatever you want to use.
And you can use a double take function here to take a number of images here.
And then we just create nine subplots with the Mark lab code here and we just replace a title if it's
a zero, if it's a dog, but a cat, if it's a zero and or else it's a dog one.
So we can see, we can visualize those labels here.
So this is a quite cool and nice way to visualize it.
Now what we do is standardize our data here.
So we're going to use this map lambda function here with this t.f image resized to resize all the images
in this size here in the tree info and test this dataset.
So let's do that and we'll also this will also normalize between minus one and one.
And now what we do here, we're going to set about size here, and we'll just use this pre fetch buffer
here.
This is a bit confusing for you guys, so I actually use this occasionally in pie touch.
This is just a way to load the data onto your RAM so that you actually can scroll caching so you can
actually get much faster access to it to improve your training speed.
So this is all part of the TensorFlow datasets package here.
So don't worry if it's unfamiliar, it's just because it's part of that.
It's not really part of carers, so we're just learning new things.
So now what we do here, which is our augmentation pipeline.
So we're going to introduce random flips, horizontal flips and some random rotation.
You can do any sort of augmentation as you want, but we can visualize the augmentations here by running
it through this, this little training loop here, and it's ruining the getting the augmentations here
to get the augmented image.
So you can see we've taken this dog image here and we've just flipped it around and we're testing it
slightly.
So we're getting different versions of the same image in our training dataset.
So that's good.
Next.
Next, we're going to actually construct a model.
So basically, that means that we are taking a base layer that we froze and attaching our new head to
it.
So let's take a look and see how we do that.
So the model we're going to be loading is the exception model here, and this is how you load it here,
with crystal exceptions to the application.
Sorry exception.
Then you just pointed to imaging that we'd set the input image size, which is 150 by 150 color treat
up to RTB.
And notice this is an important line here.
We proceed to include topical faults, so that means that we don't include the top layer that's solely
the output nodes.
That was for image that which was a thousand nodes, and we don't include that fully connected layer
there.
So what we do in that next, we freeze the base model here.
So you would have seen this before.
So we set this model that trainable to false.
And now what we do next is we create the top model header.
So to do that.
So it's a bit confusing.
But as in many things we have to do here.
So firstly, we have our inputs here and that basically is input shape of our models.
So we have crystal input just to finding that input shape.
Here we have our data augmentation, which we defined previously.
And then what we do next.
The pre-trained exception we require that input be scaled from zero from zero to 255 to a range of minus
one to plus one.
And then the rescaling layer here basically outputs this here.
So this is how we actually have to use the keras totally layers rescaling, so we scale it to here the
one minus one.
So don't worry about this if you don't understand that there's a lot of tricky, hard to comprehend
cause sometimes.
Was with us because there were things of the just set that does specific things like this whole.
However, don't worry about it too much.
It's boilerplate code that you can reuse easily for your own dataset.
So next, we just create.
We just passed us Input X here to that skilling layer here.
This is all all the skilling with us.
And then what we do now, we have obvious model here.
So this is the inputs here.
We have inputs going into our base model and then notice it in your head that we've attached to a model.
So we have a global average pooling layer here.
Then that outputs to another layer that has drop out and then we have a final densely, which is just
one output node because it's a binary output binary model, just two classes.
So let's run this and analyze the summary that comes out of out of it here.
You can see we only have two thousand trainable parameters, so we have 20 million parameters that we
aren't training.
Those are fixed to the ones that have been trained on image that.
So those are frozen and these are the ones that are unfrozen here.
So now let's train our top layer.
So let's do this.
And you can see we've started training already and this is quite simple.
We just have to compile the model here.
We used Adam Optimizer here, and then we use Binary Cross into breathe because it's a binary output
model that we set from the it's equal true.
And then we just have the metrics here being the Keros metrics binary accuracy.
Right now, I believe we are training on CPU, which is why it is so slow.
Yes, it is.
So yeah, I'm going to stop this for now because there's no point in turning on the CPU because in ten
minutes, but you book, it's going to take all night.
So what I'll do next is we'll take a look at fine tuning.
So remember and fine tuning, we unfreeze the base model a bit or parts of it.
And then we use a low learning rate here.
So let's take a look and see how we do that.
So now you can see we've set a base model trainable to be equal to true.
OK.
So in this case, we're training the whole base model, but at a very low learning oriented, you can
see we've set the learning rate to one two by 10 to the minus five.
It's a tiny learning rate and lets us compile this model from the model summary here.
And again, over on the CPU.
So this is going to take a while, but hopefully you enjoyed this lesson.
It's quite easy to do transcoding in tariffs compared to PI torture, in my opinion.
However, doing it to purify too much isn't that bad.
To us, lightning actually does make it a tad confusing sometimes, but it's OK.
As you can see, this is why we don't train on CPUs, especially when we have all these tunable parameters.
This is going to take roughly an hour to epoch an attorney box, so that's going to take all night.
I mean, if you wanted to wait, you could.
It's just probably not with our time right now.
So I'm going to stop this lesson here.
And then next, we'll take a look at.
Using CNN's as feature extract is with us, so I'll see you in the next listen.
Thank you.