So in this lecture, we will be answering the question, how can I choose hyper parameters?
Please note that while I've discussed this in the course already, sometimes people forget.
So I'm making this lecture to explicitly remind you of this topic.
Now, this is often a point of frustration amongst beginners.
Beginners often feel like there must be some direct method of doing this.
They think because this is science, that there must be an answer, that there must be some formula
or some algorithm that will yield to the correct results.
Unfortunately, this is not the case.
And as a responsible and mature scientist, you would do well to get comfortable with this reality as
a side note.
Some have said that machine learning is more like alchemy than science.
I would tend to agree with this notion, and this is one of those areas that makes it apparent.
So before we continue on, I want to mention that this applies to all the hype parameters discussed
in this course.
For example, in the early parts of the course, our main concern is how do we choose the learning rate?
What is the best optimizer in the Aon and section?
Our concerns may be how do we choose the number of hidden layers?
How do we choose the number of hidden units?
How do we choose the best activation?
What dropout probability should I use in the CNN section?
Our concerns may be what is the best filter size?
What is the best number of feature maps?
And these general themes repeat throughout as you learn about origins in LP recommenders, gans and
so forth.
So please keep in mind that this is not actually a separate question for each of these items.
These are all actually the same question.
Specifically, it's the question of how to choose hyper parameters.
So the answer to this question, although many of you may find this disappointing, is my famous rule.
Machine learning is experimentation, not philosophy.
The problem is, many beginners start to look for a deep philosophical reasons for choosing this number
of hidden layers or that number of hitting units.
In fact, the correct answer is to simply try random values and see what result you get.
Obviously, the final choice should be the one that gives you the best results.
The basic idea is that you must do experiments.
There is simply no other option.
One obvious thing you can do if you want to be sure you're at least in the right range is to look up
the papers in your field to see what they used.
This is assuming that your data sets are similar, so that should give you a reasonable starting point
for your random search.
But do keep in mind that this adds a bias and could potentially prevent you from searching closer to
the optimal point.
This will also help you get a sense for the scales at which to try random values.
For example, learning rates in many other parameters are usually chosen on a log scale.
For example, zero point one zero point zero one zero point zero zero one and so forth.
If you're interested in seeing more specifics, including actual code, to run a random search, I've
already covered this in my in-depth series of deep learning courses, specifically modern deep learning
in Python.
Do note, however, that most of it is just common sense.
For example, if a value of 10 is bad and 100 is worse and one thousand is even worse, you probably
won't continue one in that direction.
It may help if you made a plot of your model's performance versus the hyper parameter value, but again,
this type of thing is just common sense.
Now, many beginners still object to this, they say, but lazy programmer, there must be some rules
of thumb.
Why can't you just give me some simple rules of thumb?
And the answer to that is this is simply a bad approach.
I'll illustrate this with several examples.
As you recall, we've noted that Adam is often the default choice for optimization method.
However, there was a paper that came out showing that plain SGMD performed better.
Thus, if you're the type of student that tries to follow only rules of thumb without actually doing
the requisite work, you may have landed at a suboptimal result.
This is what happens when you fail to experiment.
Here's another very recent example from OpenAI, released in October 2021.
In this paper called Rocking, the authors explore an interesting phenomenon regarding the ability of
neural networks to generalize.
Now the idea behind this paper is not that relevant for this discussion, but what is relevant is what
the authors tried.
Specifically, they tried different approaches, such as full versus stochastic gradient descent.
They tried different learning rates.
They tried, dropped out, and they even tried to wait to K, which is a method that has somewhat fallen
out of favor in the modern era of deep learning.
So even researchers at one of the top eight companies in the world still must do these experiments and
even on simple things like weight decay.
Furthermore, I lied a little bit when I said the idea behind this paper is not relevant for this discussion.
In fact, what the paper demonstrates is that reality disagrees with some long-held notions about statistical
learning.
Specifically, they show that neural networks can in fact generalize well.
If you keep training beyond the point, it has over, fit it.
So again, if you're the type of student that just wants to memorize some rules of thumb, you would
again be proven wrong by those who did the actual experiments.
Importantly, note that this paper came out several years after the release of this course.
You should expect things to constantly change.
You should expect any rules of thumb you've memorized to soon become irrelevant.
Put more simply, memorization bad.
Experimentation good.