‫In this lecture we will discuss about the hyper parameters of our neural network architecture and the
‫lectures in law you have seen that there are so many hybrid parameters in neural networks and these
‫hyper parameters give us the flexibility of creating similar types of architectures but the flexibility
‫of neural networks is also one of their main drawbacks.
‫We have to decide on so many hybrid parameters in our model.
‫Not only can we use any imaginable network architecture but even in a simple multilevel perception you
‫can change the number of layers.
‫The number of neurons per layer the type of activation function to use in each layer the rate initialization
‫logic and many more although still a lot of exciting research is going on in the field of hyper parameter
‫tuning for neural networks.
‫It will still help to have an idea of what values are reasonable for each hyper parameter so you can
‫build a quick prototype and restrict the search space.
‫Here are a few guidelines for choosing the number of the layers and neurons in a multilevel perception.
‫Let's first discuss about the number of hidden layers for most of the problems.
‫You can just begin with a single hidden layer and you will get reasonable results.
‫It has actually been shown that an MLP with just one hidden layer can model even the most complex functions
‫provided it has enough neurons for a long time.
‫These fact convince these voters that there is no need to investigate any deep neural network but it
‫was later found that deep networks have a much higher parameter efficiency than shallow ones.
‫They can model complex functions using exponentially fewer neurons than shallowness allowing them to
‫read much better performance with the same amount of trading data to understand why this happens.
‫Suppose you are asked to draw a forest using some drawing software but you are forbidden to use copy
‫paste.
‫You have to draw each tree individually branch by branch leaf belief.
‫If you could instead draw one leaf copy pasted to a broader branch then copy paste the branches to create
‫the tree and finally copy paste this tree to make a forest.
‫You would be finished in no time.
‫Real world data is often structured in such a hierarchical way and deep neural networks automatically
‫take advantage of this fact.
‫Lower hidden layers model lower level structures intermediate hidden layers.
‫Combine these lower level structures to model intermediate level structures and higher state layers
‫and the output layer.
‫Combine these intermediate structures to model high level structures.
‫Not only does this hierarchical structure help deep neural networks converge faster it also improves
‫the ability to generalize to new data.
‫For example if you have already trained a model to recognize faces in a picture and you know want to
‫train a new neural network model to recognize hair styles then you can kick start training by reusing
‫the lower levels of the first network instead of randomly initializing the weights and biases of the
‫first few layers of the new neural network.
‫You can initialize them to the value of the weights and biases of the lower layers of the first network.
‫This way the network will not have to learn from scratch and it will only have to learn the higher level
‫structures.
‫This is called transfer learning.
‫So in summary for most problems you can start with just one or two hidden layers and it will work just
‫fine for more complex problems.
‫You can gradually ramp up the number of hidden layers until you start all fitting the training data.
‫Next we discuss the number of neurons but you then layer obviously the number of neurons in the input
‫and output layers is determined by the type of input and output.
‫Your task requires for example the M NASD fashion dataset that we used that required 24 into 24.
‫That is 780 for input neurons and then output neurons.
‫As for the hidden layers earlier it was a common practice to size them to form a pyramid.
‫That is the first layer had the most number of neurons.
‫For example in the M NASD fashion dataset with three layers.
‫You can have 300 neurons in the first data layer two hundred neurons and the second one and hundred
‫and deterred the rationale being that many low level features can coalesce into far fewer higher level
‫features however this practice has been largely abandoned now as it seems that simply using these same
‫number of neurons in all hidden layers performs just as well.
‫In most cases or maybe even better
‫also it has the advantage of having only one hyper parameter to tune instead of one.
‫But earlier so instead of having three hundred two hundred and 100 neurons in the tree and layers you
‫can have 150 neurons in all three of them if you think that the problem at hand is really complex then
‫you can try increasing the number of neurons gradually until the network starts over putting in general
‫increasing the depth of the network has better results on the accuracy then increasing the number of
‫neurons.
‫Butler another approach could be to pick a model with large number of layers and large number of neurons
‫per head and there and then use at least hoping to prevent that model from all fitting
‫next type of barometer.
‫We are going to discuss learning great learning that is arguably the most important hyper parameter.
‫In general the optimal learning rate is about half of the maximum learning rate that is in learning
‫grade about which the training algorithm diverges
‫so a simple approach for tuning the learning rate is to start with large value that makes the algorithm
‫they would then divide this value by three and try again and repeat this until the training algorithm
‫stops diverging at that point.
‫You generally won't be too far from the optimal learning grade.
‫Then there is bad sites for bad essays.
‫The general rule general rules of thumb try to keep bad sides lower than 32 because a small bad size
‫ensures that each training ideation is very fast and on the lower end.
‫Try to keep a bad size more than 20 because this helps take advantage of the hardware and software optimizations
‫in particular for matrix multiplications.
‫So this will also help in speeding up training so a good range is between 20 to 32.
‫Lastly there is another hyper parameter called epoch that is the number of training ideations that we
‫will do instead of tuning it.
‫We would suggest that you use a large number for epochs and use the early stopping technique to prevent
‫or predict that's all about selecting high but parameters.