Okay, so this week you've been looking at machine learning for
predicting the next value in a sequence.
You've learned how to break your data down into window chunks that you could train
on, and then you saw a simple single layer neural network
that gave you what was effectively a linear regression.
Now let's take that to the next step with a DNN to see if we can improve
our model accuracy.
It's not that much different from the linear regression model we saw earlier.
And this is a relatively simple deep neural network that has three layers.
So let's unpack it line by line.
First we'll have to get a data set which will generate by passing in x_train data,
along with the desired window size, batch size, and shuffle buffer size.
We'll then define the model.
Let's keep it simple with three layers of 10, 10, and 1 neurons.
The input shape is the size of the window and
we'll activate each layer using a relu.
We'll then compile the model as before with a mean squared error loss function
and stochastic gradient descent optimizer.
Finally, we'll fit the model over 100 epochs, and
after a few seconds of training, we'll see results that look like this.
It's pretty good still.
And when we calculate the mean absolute error, we're lower than we were earlier,
so it's a step in the right direction.
But it's also a somewhat a stab in the dark,
particularly with the optimizer function.
Wouldn't it be nice if we could pick the optimal learning rate instead of the one
that we chose?
We might learn more efficiently and build a better model.
Now let's look at a technique for
that that uses callbacks that you used way back in the first course.
So here's a code for the previous neural network.
But I've added a callback to tweak the learning rate
using a learning rate scheduler.
You can see that code here.
This will be called at the callback at the end of each epoch.
What it will do is change the learning rates to a value
based on the epoch number.
So in epoch 1, it is 1 times 10 to the -8 times 10 to the power of 1 over 20.
And by the time we reach the 100 epoch,
it'll be 1 times 10 to the -8 times 10 to the power of 5, and that's 100 over 20.
This will happen on each callback because we set it
in the callbacks parameter of modeled outfit.
After training with this, we can then plot the last per epoch against
the learning rate per epoch by using this code, and we'll see a chart like this.
The y-axis shows us the loss for that epoch and
the x-axis shows us the learning rate.
We can then try to pick the lowest point of the curve where it's still relatively
stable like this, and that's right around 7 times 10 to the -6.
So let's set that to be our learning rate and then we'll retrain.
So here's the same neural network code, and
we've updated the learning rate, so we'll also train it for a bit longer.
Let's check the results after training for 500 epochs.
Here's the codes to plot out the loss that was calculated during the training, and
it will give us a chart like this.
Which upon first inspection looks like we're probably wasting our time
training beyond maybe only 10 epochs, but
it's somewhat skewed by the fact that the earlier losses were so high.
If we cropped them off and plot the loss for epochs after number 10 with code like
this, then the chart will tell us a different story.
We can see that the loss was continuing to decrease even after 500 epochs.
And that shows that our network is learning very well indeed.
And the results of the predictions overlaid against the originals looks
like this.
And the mean absolute error across the results
is significantly lower than earlier.
I'll take you through a screencast of this code in action in the next video.
Using a very simple DNN, we've improved our results very nicely.
But it's still just a DNN, there's no sequencing taken into account, and
in a time series like this, the values that are immediately before a value
are more likely to impact it than those further in the past.
And that's the perfect set up to use RNS like we had in the natural language
course.
Now we'll look at that next week, but first, let's dig into this code.