Hello everyone, and welcome to the session in which we are going to treat hyper parameter tuning with
weights and biases.
There are three methods available for us to implement hyper parameter tuning in weights and biases.
That is the random search, the grid search and the bias and search method.
At the end of the session, you'll be able to search for the most suitable parameter values which optimize
the accuracy.
Previously we saw how to implement hyper parameter tuning with Tensorboard.
We created this model tuning method which takes as argument h params as hyper parameters which are actually
going to be tuned as you could see here.
And then we have an output of this model tuning method, the accuracy.
So what we're trying to do here is we're trying to tune or we're trying to obtain the optimal values
for this different hyper parameters which maximize the accuracy of the model.
The exact method we used was a grid search with a grid search method.
We actually go through each and every option or each and every possibility, and for each possibility
we are going to log its accuracy as we did right here.
Now, with weights and biases, we are going to see how to redo this, Bert, in an easier and more
reliable manner.
So let's check out the documentation right here.
You see, we have this documentation here.
We have hyper parameter tuning.
We can click on the sweeps quick start, which shows us globally what hyperparameter tuning with weights
and biases entails.
We set up weights and biases.
We configure the sweep, we initialize a sweep, we launch agents, we visualize the results and then
finally stop the agent.
From here we will see how to run this sweeps in Jupiter.
So the weights and biases sweeps allow you to easily try out a large number of hyper parameters while
tracking model performance and logging all the information you need to reproduce experiments.
Now we're going to focus on this pure Python method where the sweep configurations are in the form of
a dictionary like this.
The way this sweeps work or the way weights and biases sweep work is we have a central sweep server
that is this one here, the central sweep server.
And then we have this different agents right here.
We can have many more agents.
Then once we set the configurations, the sweep configurations in this central sweep server right here,
the agents now take over and do the actual hyper parameter tuning that is the search for the best hyper
parameters which help in optimizing the model and this parallel method of implementing sweeps in weights
and biases helps make the hyper parameter tuning process even more efficient.
As we could see here, this could be done in just two steps.
We initialize the sweep and put all necessary information or give all necessary information for this
central or to this central sweep server.
And then we run this different agents right here.
Now that said, we have this sweep configuration which looks similar to what we had with when we're
working with Tensorboard.
So here we had this configuration right here where we specified for each hyper parameter the values
you can take.
Given that we're having a grid search, we're making use of grid search algorithm, or we're specifying
simply all those different values.
And that was it.
And now coming back to weights and biases you see here, you just need to specify, for example, the
hyper parameter, the number of epochs give the values learning rate, for example, you give the minimum
and maximum value and so on and so forth.
But then note that here this as in this example, given the documentation, the method you're using
is not the grid search that is then going through each and every possibility in the list of values.
What you're doing here is actually a random search which happens to be more efficient than the grid
search algorithm, since it's possible in a shorter period of time to get very optimal values of the
hyper parameters.
As compared to the grid search algorithm where you need to go through each and every value.
Now let's get back.
Here we have the name, we have the method, and we have the parameters to understand where all this
come from.
Check out this sweep configuration right here.
So clicking on the sweep configuration, we have the structure of a sweep configuration.
Then here we have the different keys and the descriptions.
You will notice this method which we had seen here.
Let's open this in another tab, open a new tab and have that.
So let's have it this way.
There we go.
We have this name, method and parameters which we could see here.
We have name method parameters.
And then for each of these configuration keys, like those method, for example, there is this more
explicit description.
So here we have the method we're going to use.
If it's a grid grid search iterates over all possible combinations of parameter values.
The random search chooses a random set of values on each iteration, and weights and biases also comes
with this other method, which is the bias method.
This bias in hyper parameter search method uses a Gaussian process to model relationship between the
parameters and the model metric and chooses parameters to optimize the probability of improvement.
This strategy requires the metric key to be specified.
So right here we see now we have this three different methods and that is why your when we wanted to
work with the random, it suffices to just put this out here.
Either you put it random or you put grid or you put bias.
Now we move to the next we have the parameters.
Oh yeah, we have the parameters.
Anyway, the name just you could put the name for the sweep.
That depends on you.
Like here's my sweep, then the parameters here, it gets a little bit more tricky because here you
have this year, this key, and then the value is a dictionary, which itself is having its own keys
and values.
So we have this parameters.
Let's get back to parameters here.
We have the parameters where we we have parameters and then we have this different possible values and
the descriptions.
So getting back here, you'll notice that this hyper parameter epochs takes values in this list that
is either ten, 20 or 50, while the learning rate can be chosen between 0.0010.0001 and the maximum
0.1.
So each hyper parameter has its own distinct way of describing the values it can take.
Now, getting back here we have okay, we have the different we have this parameters.
And then for those values, you see, you can take you can say values, you can say a value.
So sometimes, like here we may decide and say, okay, one does epoch to take only just one value,
then we just have value.
And then in the case where once several values you see specifies all valid values for this hyper parameter
compatible with grid and all of that.
Now here in some cases you have a distribution and this select a distribution from the distribution
table below.
So you could see here the specifies how values will be distributed if they are selected randomly, e.g.
with a random or bias method.
So when we when working with the random or bias methods, you may want to select your values based on
a given distribution and here are the list of distributions you can use.
You have constant categorical int uniform uniform distribution or uniform log uniform q log uniform,
normal distribution or normal log normal and log normal.
Then from this distribution we now move to Min max Min.
Max is where you actually saw your here.
You had this min max.
So here you're simply saying you want your learning rate to fall or to have a minimum value of this
and the maximum value of that.
So we randomly pick values between this in this range and that's it.
You have mu mu min parameter for normal or log normal distributed hyper parameters.
So here you're having a normally distributed hyper parameters and you're specifying the mean.
While here you're specifying the standard deviation.
Then for cure you have the quantization step size for quantized hyper parameter.
Our next key is the metric.
And with the metric we have to define the name of the metric, the goal and the target.
Now here you could have, for example, like your validation loss, you have this metric validation
loss.
You could say you want to optimize your parameters or you want to choose the hyper parameters.
Which minimize the validation loss.
Now, you could also change this into an accuracy, let's say, validation accuracy or adjust the accuracy
or just the train accuracy.
In that case, you want to choose the hyper parameters which maximize the accuracy and hence here at
the level of the goal, you either minimize or maximize.
The default is minimize.
So if you have a validation, loss is needless specifying the goal because by default it's minimized.
And for now, as in this documentation, there is really no automatic way of deciding whether it's minimize
or maximize anyway.
You can always do that manually by specifying.
Now the next we have is the target.
So here with the target, as you can see here, for example, 0.95, it means that if you happen to
get a set of hyper parameters which permit you get this validation accuracy to this target value, then
at that point you will stop the process of searching for the.
Optimal hyper parameters.
And what happens exactly is all agents with active runs will finish their jobs, but no new runs will
be launched in the sweep since we've already attained the objective.
From here we have early Terminate, which is an optional feature that's a piece of hyper parameter search.
By stopping poorly performing runs, we get back here, copy out this code.
Just copy it.
Okay, copy it.
We get back now to our hyper parameter tuning.
We paste out the code here, and then we're just going to try to replicate what we have done already
with Tensorboard.
So here we paste this out here and then we have this NUM units one, we have this NUM units one this
year and have that no minus one values.
There we go.
Let's copy this out and paste here.
So we replace this values which we had already and then the next will be num units two.
It's kind of similar.
So we should not minus two.
Okay.
So here we should just copy this and paste right here.
Num units, two values.
Yeah, it's kind of similar.
Okay, we have that and then there we go.
So we have num minus one, no, minus two.
And then the next dropout rate we're going to have learning rate.
So let's just have the dropout rate here.
So we have the dropout rate and then this dropout rate, we take values of 0.120.3.
Now here, what we're going to use is like this min max.
So let's just copy this out here and then let's copy this out and then paste it out here.
Okay, so we have dropout rate, but here we're going to go from 0.1 to 0 point, let's say 0.4 and
this random actually not great.
So we have that.
And then here, okay, name, let's say name malaria Prediction sweep.
Okay, Method random parameters is fine.
So we're getting each and every parameter.
Now we have the dropout rate set.
We now move to regularization rate.
Regularization rate values between 0.001 and 0.1.
Okay.
So we have 0.010.1.
And then here, what we could do now is we make use of, let's say, distribution and then we have your
uniform.
So we specify that we want to make use of a uniform distribution and then here.
So we're going to use the same again here for the learning rate distribution.
Um, uniform.
There we go.
We have that uniform and that's fine.
But here we're going to go from this, let's say one E negative for mean and then Max one E negative
two, one E, negative two.
Okay, so looks good.
Let's take this off now.
Let's take off our set here and then we have that.
So from this we'll be able to create now this sweep ID, we'll get our sweep ID by running this one
sweep and passing in the sweep configuration which to run an agent.
The first step is we're going to define a function to run the training based on those hyper parameters.
And then we're going to pass that function with the sweep ID here in this one DB agent method.
So now let's copy this code out and then paste it out here, paste this out here, and then you'll also
notice that this is kind of like similar to what we had done here, because here we had defined this
method which takes in the hyper parameters which we're trying to tune, and then we go through this
method here for each and every sweep.
Now let's get back here.
We have the model.
We could simply make use of this model.
So let's copy out some part.
Let's, let's just copy out this here.
Let's copy that out and paste this here, paste it out and we have model tune, um, net model which
we create here, hyper parameters and all of that.
And then instead of what we had here, where we passed, we had the compile and the feed method, what
will return here will be just this model.
So we'll return the net model right here.
So let's have this net model, let's take out this part of the code and that's fine.
Okay, so we have this model tune here.
Um, instead of make model, we have model tune and we'll pass the configuration in here.
We add.
The project and the entity in our init method.
Then we've modified this keys right here to match with those of our one DB configurations which we have
seen already.
Let's get back here.
Let's just copy this and put right here so you could see that.
So as you could see, instead of number of units, one we have number of dense one and here we have
number of dense two.
So that's it.
We have the different values you could take.
We have the dropout, the regularization rate and the learning rate.
Now, right here in this model, this whole model we should call model tune, we shall make use of this
one.
DB configurations, which we've already passed in here.
And so here we have config and config and we're going to do the same for all this other different hyper
parameters.
Like here we'll have number of filters as your config number of filters.
And then you should note that in this example we are not going to tune all the hyper parameters.
So the hyper parameters which we want to tune should be in this parameter or parameters dictionary.
In our sweep configuration.
The method is random and the metric is the accuracy with the goal of maximizing it.
So if you want to tune any parameter or hyper parameter, you put that in here.
For the rest, we're just going to use this configurations which we've set already.
So you just have some fixed values.
So that said, we would modify all this and there we go.
So we now have all this different modifications.
We run this, run this, and then back here on this train, we could now replace all this with the compilation
and the fit method.
Now again, here we would have this learning rate.
So we have learning rate and then, um, we could modify this number of epochs to say three or actually
the config.
So we have config number of epochs.
And you should note that we're carrying this only on the validation set.
So you could try this on the full training dataset that will take much more time.
And also you could feel free to carry out this hyper parameter tuning on more parameters.
So you could take up, say, the image size like this parameter here, you could add that up and then
see how this image size affects the model performance.
So that said, we have all this already.
We have our agent, which takes the sweep ID, which we've defined already.
We have the function, which is the string right here, and then we have this count, which is the number
of runs to execute before we proceed.
Note that we specify this configuration, which is essentially this configuration we have here.
So we could just take this off and say, this is our configuration.
Okay, so that's our configuration.
And now we're ready to let our agents do their job.
So let's run this now after 20 runs, here's what we get.
Start from here.
You see, that starts by picking the drop out rate, the learning rate, the number of dense one number
of dense two, and then the regularization rate, while obviously keeping the other parameters constant.
Here we have this loss and its corresponding accuracy.
And you could check out for the other sweeps down here.
You see we have 20 different runs and we could get right here, you see.
And we click on this to view the sweep.
And now on this page you could see the different runs we have here from one up to 20, the epochs or
rather the run, the accuracy and its corresponding loss.
And then, uh, we'll skip this tool for now.
Let's look at this.
We'll go ahead and check out the hyper parameters which produce the highest accuracy, which is this
here.
Let's highlight that we have this.
Well, let's, let's pick this out from here.
Um, there we go.
We have this.
We have this that comes down here, we have this and then we have this.
And so here we have the hyper parameter values which give us the highest accuracy score.
Then coming into this, let's take this off.
We could check out this different parameters.
In all parameter importance with respect to the accuracy.
So you see that the most important is the learning rate, followed by the runtime number of dense to
dropout rate, regularization rate, number of dense one.
And here we have this other hyper parameters which we did not modify.
Now let's click on this here and see what we have.
We told automatically show, um, shows the most useful parameters.
So let's check this out.
You see that after clicking on that, the other fixed hyper parameters and even the runtime is taken
off.
So we are left with only this hyper parameters which we had fixed, or better still, which we had put
in this, uh, sweep configuration.
Then one other point you could note from here is this correlation.
So apart from importance, we could check out the correlation.
And here we told that this learning rate has a negative correlation because you could see with the red
the red indicates negative correlation, while the green indicates positive correlation.
So you could check out the values.
See, this is 0.051 while this is -3.53.
Now what this simply means is the lower the learning rate, then the higher the accuracy.
And then in this plot to the left, we have the accuracies for each and every run.
So that's it for the section.
Thank you for getting up to this point and see you next time.