At this point, you are probably sick of me saying machine learning is nothing but a geometry problem,
but here is where it gets really important.
The question we want to answer right now is why are neural networks so important?
Why can't we just use a single neuron?
I mean, it seems to be a pretty good model.
It's nice and interpretable.
Large weights means the corresponding input is important and a very small or zero weights mean the input
is not important.
Unfortunately, this type of model only gets us so far.
You'll recall that when we discussed linear regression and logistic regression, I said there are two
different ways we can make things more complicated.
The first way we can make things more complicated is to add multiple inputs.
That's something we encountered already.
This is still pretty simplistic because you can always picture a plane in your head, a straight, non-trivial
surface that separates data between classes.
The second way we can make things more complicated is that the decision boundary or regression function
we are looking for is not a straight line or.
This is what we are interested in when we discuss neural networks.
The reason the equation transpose X plus B equals zero gives us A hydroplaned is because this is the
actual definition of A hydroplaned.
There's no way around this.
There's no possible setting of W and B that could give you a curved surface.
So how can we get a curved surface so that we can solve more complicated problems?
One method you might have thought of is to use feature engineering, for example, let's take linear
regression.
Suppose that for whatever reason, salary is proportional to the square of the years of experience.
Or put another way, that salary is a quadratic function of years of experience.
Then we could say that Y hat equals the X squared plus B plus C, believe it or not, this is still
just linear regression in disguise.
It's easy to see this by doing the following take x the years of experience and call that the input
feature X1 then take X squared the years of experience squared and call that the input feature x2.
Then my white hat becomes an equation of the form Y hat equals one X one plus W two X two plus B.
Of course we've already learned that this is just a linear regression.
The problem with Fisher Engineering is that there are many different possible features squaring the
inputs, his comment, but so is combining the inputs.
So if I have the inputs x1 next to that, I could make one of the features X one at times X two.
We call these interaction terms.
But what happens if I have three input features?
Now I have to consider X one squared, X two squared and X three squared.
But we also have to consider X1 acts to X1 X three attacks to X three in total.
There are six quadratic terms as an exercise.
You might want to list out all the possible combinations we could have if we had four inputs or five
inputs.
You should discover that the number of turns we have to consider grows quite fast and thus feature engineering,
even in this simple form, seems to be a somewhat clumsy solution.
But now, let's remember on Iran, if we consider just the first layer of a neural network, we can
see that there are multiple neurons.
Importantly, there are multiple different neurons.
Each neuron is a different nonlinear feature derived from all the inputs because we've applied the sigmoid
activation function.
The feature is not just a simple, linear combination of input features.
So you can think of these as a feature one, a feature to feature three all the way up to feature M.
Now, a lot of people wonder, how does this make a neural network non-linear?
Remember that a linear function always takes the form W, transpose X plus B, our neural network takes
the form of the equation you see here for a two layer neural network.
This is what we get if we drop this item and just make the whole thing into one equation, to transpose
Samoyed of W one transpose X plus B1 and then plus B to the important thing to notice about this is
you cannot simplify this into a linear function.
If you could, then we would have a linear decision boundary.
So let's see what would happen if we had no sigmoidal at all, let's suppose we have a regression network
with no sigmoid, then it's just to transpose times W one transpose X plus B1 plus B2.
Of course, using basic arithmetic, we can expand the product due to the fact that there's no sigmoid.
It's just straightforward multiplication.
Then we just get W prime transpose X plus B prime.
We have substituted W prime equals W two transpose times W one transpose and B prime equals two transpose
times B1 plus B2.
And so you can see that if we have no sigmoid, this just reduces to a linear equation.
There is an added bonus to using neuron's for our features, if you recall, all the weights and biases
are randomly initialized and found using an iterative algorithm called gradient descent when we call
the model fit function.
What this means is that instead of having to do manual feature engineering to try and guess what features
might be good, a neural network will do this automatically.
In the olden days of machine learning, feature engineering at times used to be the only method of really
applying machine learning successfully.
And usually this would require domain knowledge.
So if you wanted to build a good image classifier for, say, lung cancer detection, you would have
to be very knowledgeable about different computer vision techniques.
But not only that, you would also have to be very good at looking at X-rays of lung cancer.
By the way, note that I'm not a doctor, so I don't know if they actually look at X-rays.
But you get the idea if you wanted to build a good stock predictor, you would have to be very knowledgeable
about finance.
If you wanted to build a good fraud detector, you would have to be very knowledgeable about the insurance
industry.
And as you may have heard, deep learning is so popular because it allows people who are not domain
experts to build very good machine learning models.
Nowadays, we have people who are not expert radiologists building state of the art image classifiers
for medical diagnosis.
All they are experts in is machine learning itself.
That is pretty powerful.
One tool I really like is the tensor flow playground.
You can find it at playground to flow, dawg.
Basically this lets you train a neural network on some synthetic data right in your browser.
It lets you see for yourself that the known that work is learning a nonlinear decision boundary.
So if you go there, it looks like the default is the donut problem, where you have one circle inside
another circle, and of course the decision boundary should be a circle.
Your inputs are only the Raw Data X1 and X2, but you could also use feature engineering if you so choose.
So you could select X one squared, X two squared X1 X two and even the sign of X one, X two.
But we're not going to do that because we want to show that neural networks are able to learn nonlinear
features using only the original inputs.
So you never have to manually create things like this yourself.
And so by default, we have two hidden layer neuron that work with four hidden units.
So there's one, two, three, four.
So let's just run this and see what happens.
All right, so as you can see, it learns the nonlinear decision boundary quite efficiently without
the need for any manual feature engineering.
You're encouraged to play around with this yourself.
So try switching data sets so you can pick, let's say, this one and then run it again.
Also, try different numbers of hidden layers and different numbers of units per head in there.
This should help give you a stronger intuition for how neural networks work and lets you observe for
yourself that they are finding non-linear decision boundaries.