So in this lecture, we'll be implementing Cibo words, Vivek.
Let's begin by importing Jensen, which has a nice interface to the data set.
We'll be using for this notebook.
The next step is to load in our data set, which is called tech state.
The next step is to check the type of our data set.
As you can see, this is a data set of objects.
Now it's not my objective in this notebook to get into the details of what this object is, but feel
free to read the documentation if you like.
So all you really need to know is that you can loop through the data set, the next step is to show
that the data set is an admirable meaning that you can do it for loop over it to get it right.
I've commented this out since essentially this will go so fast with so much data that it will overwhelm
CoLab and you'll get an error.
The next step is to do what we wanted to do above.
But just for the first 10 items so that we won't have any issues with the output.
So as you can see, each item is a list of words.
Thus, this is a data set that is already tokenized.
The next step is to count how many samples we have in our data set to get a sense of its size.
As you can see, we have about 7500 documents.
The next step is to get a sense of how many tokens are contained in each document.
So in this loop, I'm getting the length of each document and saving it to a list.
The next step is to plot a histogram of the document lengths.
As you can see, a great majority of the documents have about 10000 words.
It appears to have a few documents with less down to about 5000.
The next step is to check the mean and standard deviation of the document, Lex.
As you can see, the average length is about 10000 with a deviation of about 100.
The next step is to import the Keros tokenize her.
The next step is to fit the tokenize year to our data set and transform the data into sequences of integers,
as you recall, the variable sequences will be a list of lists of integers.
Note that have set the vocab size limit to be twenty thousand.
OK, so let's check the length of our sequences list.
As expected, it still contains one thousand seven hundred one items.
But also check the length of the first document.
So our first document has about 10000 words as expected.
Let's now call the number attribute to see how many words are tokenize are contains.
As you can see, this is set to 20000 as expected.
The next step is to check the length of our word to index map.
As you can see, this contains about 250000 words, much more than no words.
This is because this dictionary stores all the words it encountered, not just the words it keeps.
The next step is to print out the word to index mapping just to see what's in it.
So as you can see, it's a dictionary containing words as keys and integers as values.
Note that they appear to be sorted by frequency, which makes sense since that's how we decide which
words to keep.
So we have the of and and so forth.
Note that the tokenize are also contains the reverse mapping, which is useful if we want to know what
word our neural network has predicted.
So as you can see, this is a dictionary which maps integer back to word.
The next step is to do our imports for the rest of the script.
I've also imported random and said a few random seeds so that you can replicate these results.
The next step is to build our model to start.
We'll need to decide what we want to set as the context size as well as the embedding dimension.
Note that these are hyper parameters which can be tuned to optimize the results.
I've chosen these values somewhat arbitrarily.
So please feel free to test your own.
In fact, that would be a great way to exercise what you've learned in this part of the course.
Note that the context size is for both sides of the middle word, so this means five words on the left
and five words on the right.
In hindsight, it probably would have been better to simply set a variable for one side and then multiplied
by two.
The next step is to create our neural network.
As you can see, this involves four layers the input, the embedding, a lambda layer and a final dance.
Most of this is probably trivial, except for the lambda.
Basically, the lambda layer allows you to specify your own function, but as a carris layer, this
is helpful if you have some custom function you want to define and you want to incorporate it into a
neural network.
So you can see here that all it does is call the TensorFlow reduce mean function on axis one.
As you recall, the output of the embedding has the shape and by T by D so axes zero corresponds to
n axis, one corresponds to T and access to corresponds to D.
We want to get rid of the taxes because we want to take the mean of each embedding vector in the sequence
of context vectors.
Because of this, the output will have the shape and by D, which is what we normally have for tabular
data sets.
Also notice that the final layer sets use bias to false.
OK, so the next step is to call model that summary to see the structure of our model.
As you can see, that shows us our four layers.
Note that the input layer and lambda layer have no parameters as expected.
On the other hand, both the embedding layer and the dense layer have the same number of parameters
one million.
This is because both of these store a weight matrix of opposite size.
So the embedding matrix has the shape 20000 by 50 in the final dense layer, matrix has the shape 50
by 20000.
Also make note of the output shapes.
Our input has the shape and by T where T is 10, which is our sequence length or the length of our context.
The output of the embedding layer as the shape and by T by D where D is 50, which is the size of our
embedding vectors.
The output of the lambda layer is of size and by D since this takes the mean along the T dimension.
And finally, the output of the final layer is NY V, where V is the vocab size since we need to have
an output for every possible word.
The next big block of code is used to build our data generator.
This is perhaps the most complex part of this script.
Basically, this is going to randomly generate context windows from our corpus to feed into the network.
The reason we want to do this is because if we were to convert every possible context and target pair
out of our data set, there would be many redundant entries since all the context windows would overlap.
Therefore, it's more memory efficient to generate inputs and targets on the fly.
So we begin by setting half the context size, which is just the context size divided by two.
Next, we define a data generator function, which takes in a list of sequences as input inside the
function, we pre allocate expansion y batch as an umpire raise, which we will eventually yield from
this function.
The next step is to compute the number of batches, which is the number of sequences divided by the
batch size rounded up.
We round up since if we don't divide evenly, the final batch will simply contain fewer samples.
Next, we enter an infinite loop inside a loop, we shuffle the list of sequences.
This is because we don't want to encounter the sequences in the same order on each epoch, which could
bias our results.
The next step is to live through our badges for this script.
Well, the final one epoch to be one pass through each sample for each sample will choose only a single
context window.
So inside this inner loop, we first grab the relevant sequences for the current batch.
The next step is to compute the size of this current patch.
As you may recall, this may be less than the actual batch size if we are at the final batch and it
doesn't divide evenly.
The next step is to live through each sequence in our batch.
Inside this loop, we first grab the current sequence.
The next step is to choose a random position in the sequence, making space for the context.
The next step is to select the context, so X1 represents everything to the left of the middle word,
and X2 represents everything to the right of the middle word.
You can see that I've commented out some code, which is probably less efficient, but describes what
we are doing conceptually.
So conceptually what we are doing is concatenating x one and X two, which will become the input context.
However, we've already pre allocated our expansion umpire Ray, which is easy to index using Nampai
syntax, so we can simply set X one and X two separately without doing any concatenation.
The next step is to get the target why, which is the middle word and to store that in a wide batch.
Once we finish looping through each sequence in our batch, we then yield expansion y back up to the
current batch size.
The next step is to call the compile method, which you've seen before.
The next step is to call a fit method.
Note that this is a bit different because our data is not in the form of an umpire raise.
Luckily, the TensorFlow Fit function is pretty flexible, so it accepts data generators as well.
Note that when we do this, we also have to tell the fit method how many steps are any cheaper?
This is just the number of batches, as we computed above.
Note that this code takes about 45 minutes to run.
So you may want to simply run it locally on your own machine.
Now you might be wondering why I've chosen 10000 epochs.
As you recall, each document in our dataset contains about 10000 words, and each epoch takes only
one context window from each document.
Therefore, on average, we would need approximately 10000 epochs to select all possible context windows.
The next step is to plot the loss per epoch.
Noted the loss per epoch is pretty stochastic.
This is because, as you recall, predicting a word in a sentence, given other words in a sentence
has some variation.
That's why we can build things like language models that generate different text every time we run it.
But overall, the loss is decreasing, which is good.
The next step is to plot the accuracy per epoch.
Again, we see that it's pretty stochastic, but overall it increases.
The next step is to get our embedding matrix, which is stored in the layer at index one.
The next step is to create an instance of Saikia Learns Nearest Neighbors object.
Basically, this uses special algorithms to find the closest vectors to a query vector, just like we
did with TFI Taf.
We're going to create this object to look for five neighbors using an algorithm called a ball tree.
We also need to call the fit method passing in our matrix of embeddings.
So basically, what we're going to do later is passing a query vector, and then this neighbors object
will tell us which of these embedding vectors are the closest.
In particular, it will choose the five closest vectors.
The next step is to demonstrate how to use what we've created to find the closest neighbors to a given
word to start, I've chosen the word queen.
The first step is to get the integer index.
For this word, which is stored in our tokenize are the next step is to select the corresponding embedding
vector for this word.
Note that because I can learn models, expect a 2D input array, I've indexed the embeddings at Queen
RDX to Queen RDX plus one, which gives us back a one by D matrix.
The next step is to call neighbors dot k neighbors passing in our query vector.
This will return both the indices of the closest vectors and their corresponding distances.
We don't need the distances, but I've printed the indices.
OK, so as expected, it gives us back a list of integers.
Luckily, our tokenize are also has a mapping from index back to word, so we can see what words these
indices represent.
So as you can see, we get back.
Queen Elizabeth, King Mary and Princess, which all makes sense.
Note that the same word as the query will always be in the first position because of the distance from
a vector to itself is zero.
The next step is to create a function, to encapsulate what we just wrote in order to check the neighbors
for other words.
So let's try the word, uncle.
As you can see, the results other than uncle or cousin, grandfather, mentor and father.
These make sense because they are mostly family relationships.
Let's now try Paris.
As you can see, the results other than Paris or Venice, Vienna, Florence and Milan, which all makes
sense.
These are all places in Europe.
Now, let's try Japan.
So we get Taiwan, Turkey, Singapore and Pakistan, which makes sense.
These are all countries.
Let's now try election.
So we get presidential elections.
Vote in candidate, which all makes sense.
Let's now try California.
So we get Texas, Florida, Illinois and Michigan, which makes sense since they are all U.S. states.
The next step is to try an analogy, basically, if we think of this like an equation, we can isolate
queen to get king of minus man, a plus woman.
This gives us a vector and we want to find the closest neighbors of this vector, which we hope will
be queen.
So let's try this out.
OK, so as you can see, Queen appears in the list of top five neighbors know that by convention, we
would typically ignore any words that were part of the query.
So in this instance, we would have returned the queen.
Let's now try this again.
But for England is too English, as Australia is to Australian.
So as you can see, Australia appears as the first neighbor, which is correct.