In this lecture I'm going to show you the code for how to create an orange n without embedding layer
so that it can take text as input rather than a feature vectors as input to be more specific.
We'll assume that this text is already encoded as a sequence of integers that can be used to index and
embedding.
We haven't discussed how to create such sequences of integers yet but I assume you have some idea in
your mind what is cause how exactly that will work in a later lecture.
But for now we'll just focus on the model itself.
So let's start with the constructor as you might expect.
It's pretty similar to a regular Arnon but just with one additional embedding layer because we have
this additional embedding layer we'll also need additional hyper parameters specifically the size of
the embedding dimension which will denote as deep as a hyper parameter.
Do we want the word vectors to be of length 20 or length 50 or maybe length 300.
This is a hyper parameter choice to be made by you.
We'll also need to know the size of the vocabulary which will denote as V.
So why do we need to know V and B well this specifies the size of the embedding matrix which will be
of shape V by D.
It has V rows and these columns.
Why is that is because each row gives us back a D size vector corresponding to a word.
This is what we discussed in the previous lecture.
So in total we have five different numbers that we pass into the constructor V which is the vocabulary
size D which is the embedding dimension M which is the number of hidden units L which is the number
of hidden layers and K which is the number of outputs.
Then we instantiate three modules the embedding module the Alice module and the final linear module.
We already know how the Elysium and linear layer work.
So I assume this requires no further elaboration.
Next we have the forward function in the forward function that we take in some kind of x and then we
pass it through each of our layers that we defined earlier since our model includes an Alice to m.
We know that we need to instantiate an initial head and state a night and an initial cell state scene
I as usual.
The important thing to pay attention to is shapes.
First what is the shape of Vex that we expect as input since each sentence is a sequence of word indexes
which are integers.
We expect X to be an array of shape n by t containing integers.
Each row is a sequence of length t in each of the T columns contains a word index after passing this
through an embedding layer.
It's going to map each word index to a desired vector.
Therefore we get back and end by t by the tensor.
We should recognize this as the usual shape that we pass through and aren't in therefore after this.
Nothing else really changes when we pass in an end by t by D through an Alice M we get back in array
of size and by t by M if we do a global Max pool or we simply take the final hit and state we get back
a 2D array of size end by M.
Finally after passing this through the final dense layer we get back in array of size n by K.