At this point you're probably sick of me saying machine learning is nothing but a geometry problem.
But here is where it gets really important.
The question we want to answer right now is why our neuron that work so important why can't we just
use a single neuron.
I mean it seems to be a pretty good model.
It's nice and interpretable.
Large weights means the corresponding input is important and a very small or zero weights mean the input
is not important.
Unfortunately this type of model only gets us so far
you'll recall that when we discuss linear regression and logistic regression I said there are two different
ways we can make things more complicated.
The first way we can make things more complicated is to add multiple inputs.
That's something we encountered already.
This is still pretty simplistic because you can always picture a plane in your head a straight non curving
surface that separates data between classes.
The second way we can make things more complicated is that the decision boundary or regression function
we are looking for is not a straight line or plane.
This is what we are interested in when we discuss neural networks.
The reason the equation w transpose x plus B equals zero gives us a higher plane is because this is
the actual definition of a hyper plane.
There is no way around this.
There's no possible setting of W and B that could give you a curved surface.
So how can we get a curved surface so that we can solve more complicated problems
one method you might have thought of is to use feature engineering.
For example let's take linear regression.
Suppose that for whatever reason salary is proportional to the square of the years of experience.
Or put another way that salary is a quadratic function of years of experience.
Then we could say that y hat equals X squared plus b x plus C Believe it or not.
This is still just linear regression in disguise.
It's easy to see this by doing the following.
Take X the years of experience and call that the input feature x 1 then take X squared.
The years of experience squared and call that the input feature x 2.
Then my y hat becomes an equation of the form y hat equals w 1 x 1 plus W2 x 2 plus B.
Of course we've already learned that this is just a linear regression
the problem with feature engineering is that there are many different possible features squaring the
inputs as comment but so is combining the inputs.
So if I have the inputs X1 and X2 than I could make one of the features x1 at times x 2 We call these
interaction terms but what happens if I have 3 input features.
Now I have to consider X1 squared X 2 squared and x 3 squared.
But we also have to consider x1 x 2 x1 x 3 and x 2 x 3 in total.
There are 6 quadratic terms as an exercise you might want to list out all the possible combinations
we could have if we had 4 inputs or 5 inputs.
You should discover that the number of turns we have to consider grows quite fast and thus feature engineering.
Even in this simple form seems to be a somewhat clumsy solution.
But now let's remember our neuron.
If we consider just the first layer of a neuron that work we can see that there are multiple neurons.
Importantly they are all multiple different neurons.
Each neuron is a different non-linear feature derived from all the inputs because we've applied the
sigmoid activation function.
The feature is not just a simple linear combination of input features so you can think of these as feature
1 a feature to feature 3 all the way of the feature m.
Now a lot of people wonder how does this make a neuron that work nonlinear.
Remember that a linear function always takes the form w transpose x plus B are known that work takes
the form of the equation you see here for a two layer neuron that we're this is what we get if we drop
the Z term and just make the whole thing into one equation W2 transpose sigmoid of w 1 transpose x plus
B1 and then plus B2.
The important thing to notice about this is you can not simplify this into a linear function.
If you could then we would have a linear decision boundary
so let's see what would happen if we had no sigmoid at all.
Let's suppose we have a regression network with no sigmoid then it's just W2 transpose times w 1 transpose
x plus B1 plus B2 of course using basic arithmetic.
We can expand the product due to the fact that there is no sigma.
It's just straightforward multiplication.
Then we just get w prime transpose x plus b prime where I've substituted w prime equals W2 transpose
times w 1 transpose and b prime equals w 2 transpose times B1 plus B2.
And so you can see that if we have no sigmoid this just reduces to a linear equation.
There is an added bonus to using neurons for our features.
If you recall all the weights and biases are randomly initialized and found using an iterative algorithm
called gradient descent.
When we call the model dot fit function.
What this means is that instead of having to do manual feature engineering to try and guess what features
might be good a neuron that work will do this automatically.
In the olden days of machine learning feature engineering at times used to be the only method of really
applying machine learning successfully and usually this would require domain knowledge.
So if you wanted to build a good image classifier for say lung cancer detection you would have to be
very knowledgeable about different computer vision techniques but not only that you would also have
to be very good at looking at x rays of lung cancer.
By the way.
Note that I'm not a doctor so I don't know if they actually look at x rays but you get the idea.
If you wanted to build a good stock predictor you would have to be very knowledgeable about finance
if you wanted to build a good fraud detector you would have to be very knowledgeable about the insurance
industry and as you may have heard deep learning is so popular because it allows people who are not
domain experts to build very good machine learning models.
Nowadays we have people who are not expert radiologists building state of the art image classifiers
for medical diagnosis all they are experts in is machine learning itself.
That is pretty powerful.
One tool I really like is the tensor flow playground.
You can find it at playground dot tensor flow dot org basically this lets you train a neural network
on some synthetic data right in your browser.
It lets you see for yourself that the known that work is learning a nonlinear decision battery
so if you go there it looks like the default is the doughnut problem where you have one circle inside
another circle.
And of course the the senior mandarins should be a circle.
Your inputs are only the raw data x1 and x2 but you could also use feature engineering if you so choose.
So you could select X1 squared X two squared x1 x 2 and even the sign of x 1 x 2 but we're not going
to do that because we want to show that neural networks are able to learn non-linear features using
only the original inputs.
So you never have to manually create things like this yourself and so by default we have a two hidden
layer neuron that work with four hidden units so there's one two three four so let's just run this and
see what happens
all right.
So as you can see it learns the nonlinear decision boundary quite efficiently without the need for any
manual feature engineering you're encouraged to play around with this yourself.
So try switching datasets so you can pick let's say this one and then run it again
also try different numbers of hidden layers and different numbers of units per head layer.
This should help give you a stronger intuition for how neural networks work and lets you observe for
yourself that they are finding nonlinear decision boundaries.