In this lecture we are going to talk about how to write code in PI talk for a simple Orient in
this is in preparation for the next lecture where we are going to do the same forecasting exercise that
we previously did but we're going to replace the auto regressive linear model with a simple or an n
by the way.
Remember that I'm not saying simple r and n to mean that simple is an adjective to describe the complexity
of this kind of Aren n rather this is the actual name we use for this kind of an N.
In other words we call this a simple or an N because that is its name.
And I'm not saying it's simple as opposed to being complex because in fact what we are doing now is
actually quite complex
so let's begin again by recalling the basic steps we are going to do in our script.
Luckily we already encountered most of this in the previous script so most of your focus in this lecture
is on how to make an art in as usual.
Step number one is to load in the data we are going to use our synthetically generated dataset once
again so that remains the same.
The only difference is that it is not going to be the right shape for an aunt in Step number two is
to instantiate our model.
This is the focus of this lecture because we're learning about a new kind of neuron that we're.
Step number three is to train the model.
Luckily this is very simple due to my rule all machine learning interfaces are the same.
Step number four is to evaluate the model which again stays the same.
Step number five is to make predictions using the model.
This will be somewhat tricky again because of the shapes.
Have I mentioned how important it is to keep track of the shapes in an ornate
alright.
So let's review step number one.
Basically is the same setup as before.
We're going to create a sine wave with and without noise then we're going to create a supervised learning
dataset with both inputs and targets the input being two steps of the sequence and the target being
the next step after those T steps.
Remember that we count from zero up to lend series minus Big T because we want the final target to be
land series minus 1 which is of course the final value of our series.
The difference between this script and the previous script is that linear regression expects a 2D array
as input so we passed in an end by t array.
However as you know and Arnon expects an end by t by D array as input.
Therefore we need to add a superfluous one dimension at the end to make it end by t by 1.
Step two is to instantiate the model.
Luckily I've already shown you at the most crucial part of this which is how to create custom models
in PI torch.
Now you understand why it was so important to do it early in a simpler environment.
To recap we know that we're going to have to subclass and end up module.
The next step is to set up our layers.
But the question is what layers do we need
in PI talk.
The simple or an end layer is simply called RNA.
And the reason we need to use a custom model is because Arnon layers don't work like the previous layers
we've seen in fact with most deep learning libraries aren't ends is where things start to get complicated
with linear models and ends and CNN is everything is pretty straightforward and the interface to each
layer follows the same kind of pattern.
Specifically a feed forward pattern.
You have one input and you get one output Arnolds are tricky because they have this feedback loop.
We know that we want to implement this equation which looks kind of like a logistic regression neuron
and we know that the origin module does just that but it seems to have a lot of interesting arguments.
So let's discuss them one by one.
First we have the input side in the head and size which should be pretty easy to understand linear layers
and convolution layers have similar arguments.
We have the input dimensionality and the output dimensionality but as you can see and have a few more
arguments that we have to specify the next one is num layers unlike aliens and CNN.
You don't need to add new layers manually as you recall.
If you want more dense layers in an N.M. you would add more layers of type and end up linear.
If you wanted more convolutions you would add more convolution layers but with aren't ends.
If you want more R and then layers you don't need to declare more than one on an object.
Instead you just pass in a different number for the argument num layers the next argument is the non
linearity where you can pass in something like the real you or 10 age.
Again this is different from an ends and CNN is where if you want a nonlinear activation function you
would specify that manually as a separate object or function with Arnold's ends.
It's simply an argument also on ns and Pi talks only except to the real you or 10 page activation functions.
The basic idea is because Arnolds are tricky to implement if you need to stack multiple or an end layers.
It's more efficient to have them implemented like this rather than gluing multiple R and ends together
and doing the computations one at a time.
Finally we set the argument batch first decoded true which tells the RNA layer that our sequence data
will be a shape end by t by D instead of T by end by the
so the full constructor looks like this.
First we inherit from an end up module inside the constructor we call the parent classes constructor.
Next we assign some instance variables including the number of inputs the number of hidden units the
number of hidden layers and the number of outputs.
Next we declare are an N layer or layers.
And finally we declare our final dense layer which is an N and the linear
the forward function is similarly tricky because unlike a and ends and CNS it's not just a matter of
passing the data through one layer after another.
Instead we have to pay special attention to the interface of the R9 layer unlike in an N or a CNN layer
where you only have one input and a set of weights with an orange layer.
You have two inputs and the set of weights specifically the output depends not only on the input x but
also on the initial head and state h not therefore in the forward function before we even called on
in.
We have to first define the initial hidden state.
As mentioned typically we just said it's all zeros in Pittsburgh.
This is done using the torch that zeros function which is analogous to the number right zeros function.
We also have to remember to move all the values to the GP you.
One important thing to keep in mind is that due to the interface of the Arnon module it may actually
comprise multiple are and end layers.
If you recall we store the number of layers in an attribute called El and because each are an N layer
requires its own initial state.
We need to define the initial hit and states to be a three dimensional tensor rather than just a one
dimensional vector and not only can we have separate initial hit in states per layer but also percentile.
Because of that the initial hidden state is three dimensional l by n by M where l is the number of R9
layers and is the number of samples or the batch size and m is the number of hidden features as you
recall from earlier we can obtain the bad size by calling X that size of zero.
Once we have our initial hit state each night we can pass it into our or an end module.
So the first argument is X which is the data and the second argument is h not which is the initial hidden
state you can see that the output interface of the RNA module is also a little different.
It seems to return it to outputs rather than just one
output from the PI torch or an end module is very strange if you haven't worked with it before.
Just to remind you the Arnon module in Pittsburgh is not just a single Arnon layer but rather it could
be a stack of and layers.
If you think about all the possible hidden states we can calculate.
We would have quite a few dimensions first we have N which is the number of samples of the bad size.
This one we expect because it's always there so small and it tells us which sample next we have L which
tells us the number of layers.
So the small l index tells us which layer next we have t which tells us the number of timestamps or
the sequence length.
Therefore a little T index is the timestamp.
Finally we have little J which as usual index is the feature dimension.
So overall we have four possible ways to index all the hidden states.
If we conceptualize them as being one giant array for each sample each layer each time step and each
feature
in Pike's works the return values are broken up in kind of a strange way.
The first output gives us the hidden states for the final layer at each time step by default.
Its shape is t by n by M where t is the sequence length and is the batch size and m is the number of
hidden features.
But since we passed in the argument batch first equals true earlier that means it is reshaped to be
n by T by M the second output gives us the hidden states over all hidden layers but only at the final
timestamp.
So by default a shape is L by n by M where l is the number of layers and is the batch size and m is
the number of hidden features curiously most common ends make use of the first output rather than the
second.
So in this course we'll be ignoring the second output.
However I did want to give you an explanation for why is there and not just ignore it without telling
you what it is.
The last thing we have to do in the forward function is pass the and stay through the final dense layer
since the hidden state we've obtained from the R9 module is end by T by M.
It's not necessarily the right shape for our use case remember that this gives us all the hidden states
for every timestamp but sometimes like for the next example we only want the head and state at the final
time step because that takes into account all the previous sequence data in order to obtain the hidden
state at the final timestamp.
We do exactly the same thing we would do if it were a num pi array if the RNA an output is called out
then I can index it with a call in then a minus one and then another call in the first column means
grab all the samples in the first dimension the minus 1 means grab only the last item from the second
dimension and the final column means grab all the features in the final dimension after indexing and
this way the output will be of size and by M which is the same shape we have when we work with and ends.
So what happens after this should now be trivial for you
now that we've done all the hard work the rest of the code will mostly be business as usual.
First we instantiate the model we just created.
We also move the model to the GP you we create the loss and optimizer as before we'll be using the means
squared error with the atom optimizer we create the inputs and targets for both the train and test sets.
We move the data to the GP you.
We do full gradient descent and we plot the last per iteration to make sure the laws converges
the last step.
Making our predictions might be a little complicated since we have to pay special attention to the shapes.
Remember that in general our input is of shape and by T by D where n is the number of samples t is the
sequence length and D the number of features the output is of shape and by K or K is the number of output
nodes.
In our case this is just one because we are doing scalar regression.
Importantly we recognize that a single input which is the time series is just technically a one dimensional
array of length T.
So in order to make this into an appropriate input for our own n we actually have to reshape it to one
by t by 1.
That's because the number of samples is 1 and the number of feature dimensions is also 1.
Then when we get our output it's going to be one by one since again and is 1 and k is 1.
Therefore in order to get this value as a scalar we need to index it as 0 twice you saw this already
in our linear regression script so it shouldn't be too difficult by now.