‫In this lecture, we will discuss about the hyper parameters of what neural network architecture . In
‫the lectors till now.
‫You have seen that there are so many hyper parameters in neural networks, and these hyper parameters
‫give us the flexibility of creating several types of architectures.
‫But the flexibility of neural networks is also one of their main drawbacks.
‫We have to decide on so many hyper parameters in our model.
‫Not only can we use any imaginable network architecture, but even in our simple, multi-level Perceptron,
‫you can change the number of layers, the number of neurons per layer the type of activation function
‫to use in each layer the weight, initialization logic and many more.
‫Although still a lot of exciting research is going on in the field of hyper parameter tuning for neural
‫networks.
‫It will still help to have an idea of what values are reasonable for each hyper parameter.
‫So you can build a quick prototype and restrict the search space.
‫Here are a few guidelines for choosing the number of hidden layers and neurons in a multilevel Perceptron.
‫Let's first discuss about the number of hidden layers.
‫For most of the problems, you can just begin with a single hidden layer and you will get reasonable results.
‫It has actually been shown that an MLP with just one hidden layer can model even the most complex functions,
‫provided it has enough neurons for a long time.
‫These facts convinced researchers that there is no need to investigate any deep neural network.
‫But it was later found that deep networks have a much higher parameter efficiency than shallow ones.
‫They can model complex functions using exponentially fewer neurons than shallowness, allowing them
‫to reach much better performance with the same amount of training data to understand why this happens.
‫Suppose you are asked to draw a forest using some drawing software, but you are forbidden to use copy
‫paste.
‫You have brought each tree individually branch pee6 branch leaf per leaf if you could instead draw one
‫leaf copy paste it to draw the branch, then copy paste the branches to create the tree and finally copy
‫paste this tree to make a forest.
‫You would be finished in no time.
‫Real world data is often structured in such a hierarchical way.
‫And deep neural networks automatically take advantage of this fact.
‫Lower hidden and layers, modrl lower level structures, intermediate hidden layers.
‫Combine these lower level structures to model intermediate level structures and highest hidden layers
‫and the output layer.
‫Combine these intermediate structures to model high level structures.
‫Not only does this hierarchical structure help deep neural networks converts faster, it also improves
‫Their ability to generalize to new dataset
‫for example, if you've already trained a model to recognize faces in a picture and you now,
‫want to train a new neural network model to recognize hairstyles, then you can kick start training
‫by reusing the lower levels of the first network instead of randomly initializing the weights and biases of
‫the first few layers of the new neural network.
‫You can initialize them to the value of the weights and biases of the lower layers of the first network.
‫This way, the network will not have to learn from scratch and it will only have to learn the higher
‫level structures.
‫This is called transfer learning.
‫So in summary, for most problems, you can start with just one or two hidden layers and it will work
‫just fine for more complex problems.
‫You can gradually ramp up the number of hidden layers until you start overfitting the training data.
‫Next, we discuss the number of neurons, per hidden layer.
‫Obviously, the number of neurons in the input and output layers is determined by the type of input
‫and output.
‫Your task requires, for example, the mnist
‫fashion dataset that we used that required 24 into 24.
‫That is 784 input neurons and 10 output neurons.
‫As for the hidden layers, earlier
‫It was a common practice to size them to form a
‫Pyramid, that is the first layer had the most number of neurons.
‫So, for example, in the mnist fashion dataset with three hidden layers, you can have 300 neurons in
‫the first hidden layer, 200 neurons in the second one and hundred in the third the rationale being
‫that many low level features can coalesce into far fewer higher level features.
‫However, this practice has been largely abandoned now, as it seems that simply using these same number
‫of neurons in all hidden layers performs just as well in most cases, or maybe even better.
‫Also, it has the advantage of having only one hyper parameter to tune instead of one per layer.
‫So instead of having three hundred, two hundred and one hundred neurons in the tree hidden layers, you
‫can have 150 neurons in all three of them.
‫If you think that the problem at hand is really complex, then you can try increasing the number of
‫neurons gradually until the network starts overfitting in general, increasing the depth of the network
‫has better result on the accuracy, then increasing the number of neurons.
‫Per layer.
‫Another approach could be to pick a model with a large number of layers and large number of neurons
‫Per hidden layer and then use early stopping to prevent that model from overfitting.
‫Next, hyper parameter
‫We are going to discuss is learning rate. Learning
‫Rate is arguably the most important hyper parameter.
‫In general, the optimal learning rate is about half of the maximum learning rate.
‫That is the learning rate about which the training algorithm diverges.
‫So a simple approach for tuning the learning rate is to start with large value.
‫That makes the algorithm diverge.
‫Then divide this value by three.
‫And try again and repeat this until the training algorithm stops diverging.
‫At that point, you generally won't be too far from the optimal learning rate.
‫Then there is batch size for batch size
‫The general rule of thumb, try to keep batch size lower than 32 because a small bad
‫size ensures that each training iteration is very fast and on the lower end.
‫Try to keep a batch size more than 20 because this helps take advantage of the hardware and software optimizations,
‫in particular for matrix multiplications.
‫So this will also help in speeding up training.
‫So a good range is between twenty to thirty two.
‫Lastly, there is another hyper parameter called epoch.
‫That is the number of training iterations that we will do instead of tuning it.
‫We would suggest that you use a large number for epochs and use the early stopping technique to prevent
‫overfitting.
‫So that's all about selecting hyper parameters.