In this lecture we are going to look at a notebook which demonstrates the use of a feed for a neural
network for regression.
This lecture is going to walk you through a prepared at coal lab notebook.
Although a very good exercise which I always recommend is once you know how this is done to try and
recreate it yourself with as few references as possible as usual you can look at the title of the notebook
to determine what notebook we are currently looking at.
The important thing about this notebook is that unlike the other examples we did this one is going to
involve synthetic data.
That means data that we are going to create rather than data we've collected from an experiment or a
database.
Now you might ask why should we do this.
Is it even practical.
Is this just the lazy programmer of being a science geek.
And of course if you think any of these things you are wrong using synthetic data is very important
for understanding the behavior and quirks of any machine learning algorithm you want to be able to see
where it succeeds and where it fails.
The important keyword there is.
We'll be creating a dataset which we can actually visualize and look at.
So we can observe that the neuron network will create a prediction surface that corresponds to the true
function that we generated the data from of course.
Don't take my word for it that this is important.
One great example of this is in the field of clustering.
So if I go over to psychic learns web page on different clustering algorithms you can see a ton of examples
of this.
In particular it shows a set of some pretty contrived two dimensional data and then it compares how
each algorithm clusters this data.
By doing this we can see where each algorithm is successful and where it is not.
In fact this is the same thing you saw in the tensor for playground without being able to see the decision
boundary and the corresponding data with your own eyes.
You can imagine that this website would be much less engaging.
Okay so now you know why visualizing all synthetic data is important onto the code as usual.
We're going to start by inputting pi torch name pi and map LA Live.
We'll also need this 3D thing to help us make three dimensional plots.
Next we're going to create the dataset.
We'll start by creating the inputs X which will just be two dimensional data points uniformly distributed
between minus three and plus three.
As you know the random function returns data uniformly distributed between 0 and 1.
So if we multiply by six and subtract three the scales the data to be between minus three and plus three.
Next we calculate the targets Y which is just the cosine of two times the first feature plus the cosine
of three times the second feature.
Now you might ask Why am I using this function.
Is it some kind of special function.
The answer is no.
I just wanted a non linear function with a few bumps and curves and the cosine does just that.
Next we're going to plot this data on a 3D scatter plot so that you can see what it looks like.
So let's run this
now one thing I don't like about notebook compared to running a real python script is that you can't
play around with the plots.
So for stuff like this I like to run the actual python script on my own computer which shows you the
plot in a separate window.
Then you can zoom into the plot rotated and so forth.
This makes it a lot easier to get a good feel for the data.
So if you run this on your own machine you'll be able to rotate the plot and look at it from different
angles.
So you can really see what the underlying function should look like.
So next we're going to build the model since the data is in the form and by D where d equals 2.
There's no real pre processing to do so this is almost the same architecture as the previous example.
We have one hidden layer with 128 hidden units and a real new activation.
The difference is now we have two inputs and one output instead of seven hundred eighty four inputs
and 10 outputs.
And since this is regression there is no activation function and we're regressing on a single scalar
value.
So the output dimension is 1
Next we're going to create the loss and optimize optimizer since we're doing regression.
We'll use the MSE loss.
You'll see that instead of using the default atom I've set the learning rate to 0.01.
Now you might be wondering where can I learn about all these different optimizer objects and what their
arguments are.
The answer is as always the PI torch documentation.
Your next question might be how do I know that a lending rate of 0.01 is good.
And remember that this is because I actually sat down and tried it.
This is the only way.
Again if you're ever wondering how many epochs or what learning rate should I use.
The answer is to try it and see
next we have our training loop which I've put into a function.
This may or may not be useful in the future for copying and pasting.
In any case I just wanted to show you an alternative way of writing this other than the fact that the
loop is a function.
Nothing else here is new.
Next we're going to plot the last pre iteration to confirm that the training process converge nicely
if it doesn't.
That means you have to go back and modify your hyper parameters so this appears to look OK.
Now we have the next step which is to make predictions using the data since this data is two dimensional
and visualizing will we have a very nice situation we can actually plot the entire prediction surface
as a function along with the original data points.
This will confirm to us that the neural network even though it's just a bunch of linear equations with
real use can approximate this sum of coastlines.
That's pretty amazing if you think about it.
We can approximate a cosine equation without having any cosine.
Now some of you might be wondering how we actually make this 3D surface plot so let's go through that
the first thing we need to do is create a mesh grid we can start by choosing points along the x axis
and the x 2 axis which I've said are just fifty evenly spaced points between minus three and plus three.
So that's what land space does.
Next we call the mesh grid function which basically does the cross product between these two sets of
points.
So every X1 and every x 2 paired up with each other we assign these to the variables x x and y y which
represent the first axis and the second axis respectively unfortunately this data is in the right format
for our machine learning model which expects and ended by 2 array as input in order to convert this
into an end by 2 array.
We're going to flatten x x and y y pass it into the V stack function which stacks these 2 arrays vertically
and then transpose the result you can do some simpler examples on smaller arrays to confirm to yourself
that this does the right thing.
Next we use our model to make a prediction on X grid and then flatten the result since the result is
end by 1.
Finally we call the plot tracer function which plots the 3D surface.
The first three arguments are the X1 axis the x 2 axis and then the function value which is why the
other arguments are not so important.
But you can play around them if you want.
So let's run this
Okay so we can see that the neural network manages to approximate this function quite well.
As mentioned previously you get much more out of being able to manipulate the plot by rotating and zooming
in so as an exercise you might want to try exploiting this script as a python file.
So you do that as file download dot pi and then run this on your local machine so I'm not finished running
this on my machine.
So here is the loss
here's the predictions along with the original data points so it really helps to be able to rotated
and spin it around
all right.
And the last thing we're going to do here is try to test out if the neural network can extrapolate so
we know that the cosine function is going to repeat periodically from minus infinity to plus infinity.
So you might wonder can the neural network figure that out.
So what I've done here is I've changed little space to go from minus five to plus five.
So just to make it a little bigger.
And if we run this
we see that it doesn't quite work.
So which is assumes that the pattern keeps going in the direction that was going at the edges
and of course this is because the neural network doesn't use a periodic activation function.
So we wouldn't expect it to be periodic with respect to the inputs.