Now as we go ahead with building these models and then training them, you may sometimes wonder why
this Value six, for example, was picked, why the the channel size three was picked, maybe why the
16 was picked.
Here's why 100 was picked.
Why not say 32?
Why not say a value like 100 year?
Why are these values were picked?
And then looking at the drop out rate, what makes us pick a given dropout rate?
What makes us pick a given regularization rate, and so on and so forth.
Now, although in this particular case we're building this model based on the net model, which is something
like a model which has already been built and tested, we will see another technique known as the hyper
parameter tuning where it will be able to select the best values for this different hyper parameters
automatically.
Now, this hyper parameters must not be only this which we've picked out here.
We could also have hyper parameters like the learning rate, like even the choice of this optimizer,
and so on and so forth.
And this way of deciding this best parameters for our model and model training is known as hyper parameter
tuning.
Now we're going to see how to do or carry out hyper parameter tuning with Tensor Bar.
So your first and first, we carry out this in parts we have from tensor bot plugins, H params hyper
parameters.
We're going to import API as HP.
Then right here we are going to redefine this model or we're going to restructure it.
So instead of having just the regularization rate drop our rate given to us like this, what are we
going to have here is yours.
For example, instead of having this drop out rate, we have HP Rams and then we'll have HP drop out.
And the aim of this process is to ensure that if we have this, suppose we have this model here and
then after training we have an output and accuracy.
We want to be able to modify the parameters which make up this model and see how they affect the accuracy
such that we are going to pick out the best parameters or are we going to pick out the parameters which
parameters get the best possible accuracy?
So that said, instead of having the dropout, we just have this variable drop out and then for the
regularization, we're going to do same.
So you're going to have h params instead of regularization rate, we're going to have this and then
we have HP regularization rate.
Then apart from this regularization rate and drop out hyper parameters, we are going to include just
right here.
Let's start with this one.
So let's copy out this.
We have h params and then we have HP number of units.
So here we have number of units and then we'll call this one because just after this we're going to
have a number of units two.
So here we are going to be able to pick out the best possible values for this hyper parameter here.
So let's have this here to OC.
So that's it.
The other hyper parameters will be included when doing the model compilation.
So let's include this model compilation and then we're going to carry out the training in here.
So we have this, let's take out the summary and then take this back.
Okay, So that's it.
Now we have this model completion right here and then we have the optimizer.
But what we're going to do here is we're going to fix this optimizer.
So we're going to have the Item optimizer and then we'll specify the learning rate to be params.
And then we have your HP learning rate learning rate.
Okay, So there we go.
This looks fine.
We have the last binary cross entropy matrix accuracy, and then we do the model training.
So once we do this model training, we are going to get the accuracy that we're going to train the model,
we're going to evaluate the model, we obtain the accuracy and then tensor by will parameters, modify
all these different parameters such that we now get the parameters or the specific values which parameters
maximizes accuracy.
So here we're going to create a function which is going to return the accuracy.
Let's have this to the right and that's it.
We have this method which will call model tune, call it model tune, and then it's going to take h
params, so it's going to take this h params and then it's going to like take different value for each
params.
And then based on the accuracy, we're going to know after these different training steps which values
for this hyper parameters best gives us the best results.
So that's it.
We have that and then we return the accuracy right here.
Then we're going to define the range of values this different hyper parameters can take.
So like this one, let's copy this out.
Let's say we want to get this HP number of units one PCs out here.
We're going to have HP Dot.
HP prompts.
There we go.
And then we have.
The name specified.
So we have a number of units, one and we also specify now this range.
So we have that discrete.
So let's have discrete and then let's say we want to pick out those values between the range, say 10
to 100.
Now let's, let's take this powers of two.
So we have 1630 to 64 or 128.
So that's it for the first.
And then for this next one, we have two right here.
Let's have that.
And yeah, units two.
There we go.
We have that.
Fine, let's copy this again.
And then we repeat the same for the dropout.
So here we have dropout.
Take this off, and then here we have drop out for the drop out.
We're going to take our values between 0.10.2, 0.3 and 0.4.
Anyway, let's let's say we want to take between 0.1 and 0.3.
And then from the dropout, we have the regularization rate.
So make sure you see that we taking typical values.
So you have typical values for this number of units will be this and for the dropout will be this for
the regularization.
Regularization rates will have different range of typical values.
So here we will have
0.0060.0010.01, and then let's say 0.1.
So that's it.
What's next?
We have the learning rate, so let's copy this out.
HP learning rates here we have one E, negative four one E, negative three.
So that's it for the learning rate and that should be all.
So we have nothing left actually.
So that's fine.
So there we go.
We've defined all these different ranges.
So we are going to search in this range, search in this range search and this range.
Search in this range, Search in this range.
Now notice that we this is actually this actually discrete value.
So it's not like we're going to pick between this value and this value.
We're just going to basically pick this value or this value or this value or this other value.
And now to perform this great search, we are going to go for number of units, one in this range specified
here.
So we're going to have in HP non units one there we go, that domain dot values.
Here we have values.
There we go.
And then we're going to do the same for this.
So we have four NUM units, two in HP non units, non units, two dot domain dot values and then for
the dropout rate and then the regularization rate in the regularization domain learning rate and the
learning rate domain, we are now going to create this h params dictionary.
So we have h params right here which is going to be our dictionary and it's going to contain the different
values here.
So we'll start with this HP NOM units one and then what we get or what we're going to put in here is
this value we have here.
So the value we pick from this domain is what we're going to pass in here.
So we have NUM units, one, one, there we go.
We now repeat the same process.
So we just have your two and then the rest.
And there we go, we have the remaining parameters.
And then from here we are going to create a file writer.
So for each run, because here we're going to have different runs for the different values.
So for each run, we're going to define the file writer.
So here we have the file writer to summary create file writer, just as we've seen already.
So here we have the file writer and then we specify this directory.
Now we want the directory to have different names based on the exact run number.
So yes, we want to have run number here, which we initialize to zero.
And then we have this.
So let's have this lock's slash.
Let's just put in the run number in your string.
We have the run number.
So that's it.
We have that and then.
After each and every run, we want to increment this run number.
So yeah, we're going to increment the run number, run number, equal run plus equal one plus equal
one.
Okay, so that's it.
Now, we've defined this file writer.
That looks fine.
Next thing we want to do is with this file writer.
As default.
So we have this default as default will now log into current hyper parameters.
So we have HP dot h params and then we pass in this current hyper parameters recall, we are going to
sweep through all these different values and then for a particular round one to know exactly what we're
passing in and that's what we're going to pass in here.
So we have h params, that's fine.
And then once we've passed, once we've notified tensor bot that this is the parameters or the hyper
parameters we're working with.
The next thing we want to do is to pass this hyper parameters in this model tune method right here.
So here we're going to pass in.
Now this params, this parameter which we just defined here, which is a function of this different
values we are going to be sweeping through.
So yeah, we're going to have model turn, model, turn.
Which is going to take into pyramids and then is going to output like from yours is going to output
this accuracy.
So yeah, we're going to have the accuracy.
So we have the accuracy, which is that then once we get this accuracy, we are going to log in as value.
So we have TF summary scaler, so log in the scalar accuracy, accuracy, and then we have the accuracy
value locked in.
So that looks fine.
We have this run number.
Why do we have a red here?
That's actually okay now, so let's get back to this now.
We could run this cell.
We run the cell.
Everything looks fine.
And then we now go straight into this hyper parameter tuning where we are going to peak out the best
values for our hyper parameters, which actually give us the best accuracy.
So for each step now we want to say, print out this, so we'll print out the Rams.
So our H per Rams is so we have H per Rams.
There we go.
And then we also want to print out the current run number.
So what we're going to have here is the H per Rams or for the run let's specify the run number H per
Rams.
Is that so our parameters this and then we format.
So we have that and then we have the run number.
There we go.
So let's have this year and then I'll run this cell.
So yes, what we get, we also modified this code here to contain each and every value in this params.
So we get this kind of output and we, we make this run for like let's count this for like 287 times.
So we have 287 different runs.
And then from here we run the cell and then run this two.
So from here we have the killers.
You are the different accuracies we get.
Anyway.
We're not very much interested in this for now because we're interested in looking at this h params
here.
Now let's start with this table view.
You can see here that with this table view we could see the different values, the different hyper parameters
we make use of, or we, we tuned throughout this process.
So you could see that scroll up a little here.
You see we have the regularization rate number of units drop out rate, and then let's scroll down again
and then go this way.
You have okay, so we have a number of units to learning rate and then the accuracy.
So you see here that the accuracy changes based on the different values we have for this different hyper
parameters here.
Now, another better way of looking at this is it's parallel causing this view.
As you can see in this parallel, according to this view, you get to like pick out the the highest
accuracy value.
And then if you follow this path, you get to see that taking a learning rate of 0.0001.
And then your let's just look at this part.
So you see notice this red part here which you can actually turn to green by clicking on it.
So you see it turns to green.
You have 0.0001 and then the number of units two, it's like around 32.
Here we have drop our rate 0.2 number of units, 132 regularization rate 0.01.
So these are the best hyper parameter values we have and we can now make use of this information to
better create our model.
So now instead of having a model like this, so now we will modify this.
And then instead of having let's get back to what we had up here.
So what we're saying is, instead of having this year where we had ten, now we could take this to 32
and then right here it's 32, then we could pick the regularization rate to 0.01.
So we have the 0.01 drop out rate.
We didn't take.
We didn't know that drop heart rate.
Let's get back to that.
The drop out rate, 0.1.
No, no, no, 0.2 actually.
So drop out rate 0.2, that looks fine.
Learning rate 0.0001.
So that's it.
So we now see how to.
Better pick those values thanks to this hyper parameter tuning which we could do easily with ten support.
So let's get back to this and then we have 0.2 right here.
Now, it should be noted that the method with you so far is great search.
That is, we've specified the different ranges and then with some sort of search through the different
ranges.
And then we've gotten the best possible parameters which.
Maximize the accuracy value.
Now, another very popular technique is the random search with a random search.
We'll just define a range and then pick our random values in that range.
So unlike the grid search, where we have some defined or predefined values which we have to search
through with a random search, we kind of like pick values at random.
But then they both have the advantages and disadvantages.
And one thing you could take out from this is that with a great search, since you have a fixed range,
so you have a fixed range of values which you can pick from, it means if this fixed range of values
is very large, it becomes problematic as this process of searching is going to take a very long time.
So you may take a very long time before finding your best hyper parameter values which reconnect.
Whereas for the random search, since we are not going in a particular order, it happens that we may
even get the best hyper parameter values after just one step.
So after just going through or after just one run or a few runs, we may already get the best hyper
parameter values, just like we may get these values after 200 different runs.
So if you have enough computational power, you could make use of the grid search.
Since you are more sure of the different values you're going to be picking from.
And if not, the random search will be a better option.
Another very important aspect of terms of board is tensor boards profiler with tensor bars.
Profiler We are able to evaluate the TensorFlow code and based on this evaluation, modify this code
to ensure that it runs as efficiently as possible.
That said, to make use of the profiler, we're going to start by installing this test about profiler
plug in so we have your people install this.
We run the so the profile plugin now installed.
We now go ahead and run this cells right here.
We're going to have this profile badge.
So notice that we have this profile badge right here and this value specified.
Your has to do with the range of badges to be profiled.
And so here we could take say 100 to, let's say 132.
Okay, so that's it.
So we have that set.
We are now run the cell tool looks fine and then we can start with the training.
We have the stance about callback which has been added here.
So we run this just out with the training train now complete, we go ahead and run this two cells just
as expected.
We have the skills, we have the graphs, we have this distributions, we have this histograms time
series.
And then in here we have this profile.