So we've looked at the sunspot data using
a standard DNN like
way back at the beginning of this course.
We've also been working a lot with RNNs and a
little with convolutional neural networks.
So what would happen if we put them all
together to see if we can predict the sunspot activity?
This is a difficult dataset
because like we've seen already,
while it's seasonal the period is really long,
around 11 years, and it's not
perfectly seasonable during that period.
So let's take a look at using all the tools we have to
see if we can build a decent
prediction using machine learning.
So here's the first piece of code we can try.
I've gone a little crazy here,
so let's break it down piece by piece.
First of all I'm setting the batch size to
64 and the window size to 60.
Then we'll start with a 1D convolution
that we'll learn 32 filters.
This will output to a couple of LSTMs with 32 cells
each before feeding into
a DNN similar to what we saw earlier,
30 neurons, then 10, and one.
Finally, as our numbers are in the 1-400 range,
there is a Lambda layer that multiplies out our X by 400.
With the first test run to establish
the best learning, rate we get this chart.
This suggests the best learning rate for
this network will be around 10 to the minus 5.
So when I trained for 500 epochs
with this setup, here's my results.
It's pretty good with a nice low MAE.
But when I look at my loss function during training,
I can see that there's a lot of noise
which tells me that I can certainly optimize it a bit,
and as we saw from earlier videos,
one of the best things to look at in
these circumstances is the batch size.
So I'll increase it to 256 and retrain.
After 500 epochs, my predictions have
improved a little which is a step in the right direction.
But look at my training noise.
Particularly towards the end of the training is really
noisy but it's a very regular looking wave.
This suggests that my larger batch size was good,
but maybe a little off.
It's not catastrophic because as you
can see the fluctuations are really
small but it would be very nice
if we could regularize this loss a bit more,
which then brings me to another thing to try.
My training data has 3,000 data points in it.
So why are things like my window size
and batch size powers of
two that aren't necessarily evenly divisible into 3,000?
What would happen if I were to
change my parameters to suit,
and not just the window and batch size,
how about changing the filters too?
So what if I set that to 60,
and the LSTMs to 60 instead of 32 or 64?
My DNN already look good, so I won't change them.
So after training this for 500 epochs,
my scores improved again albeit slightly.
So it shows we're heading in the right direction.
What's interesting is that the noise and
the loss function actually increased the bits,
and that made me want to
experiment with the batch size again.
So I reduced it to just 100 and I got these results.
Now here my MAE has actually gone up a little.
The projections are doing
much better in the higher peaks than
earlier but the overall accuracy has gone down,
and the loss has smoothed
out except for a couple of large blips.
Experimenting with hyperparameters like this is
a great way to learn the ins
and outs of machine learning,
not just with sequences but with anything.
I thoroughly recommend spending time on
it and seeing if you can improve on this model.
In addition, you should accompany
that work with looking deeper into how
all of these things in machine learning work
and Andrews courses are terrific for that.
I strongly recommend them
if you haven't done them already.