So in this lecture, we'll be looking at how to do parts of speech tagging in TensorFlow.
Now, just in case you don't know what this task is, all you have to know is the concept of nouns,
verbs, adjectives and so forth.
If you've never heard of those, take a few minutes to look it up, as it's pretty simple to understand.
So the major difference between this task and the previous task is the type of Arnon and we need to
use.
For the previous example, we wanted to classify a document into a single class.
This is called a many to one task because we have many items in our sequence of inputs, but only one
item as output.
This task is different.
This task is called a many to many task because there are many items in the sequence of inputs and many
items in the corresponding sequence of outputs.
For each input word, we should have a corresponding part of speech tag.
Note that this is still a classification task.
It's simply that we now have a sequence of classes as the target.
So let's begin by importing a few modules from Nanotech, since it conveniently keeps a copy of a parts
of speech data set.
We'll be using a dataset called the Brown Corpus.
The next step is to download the data using analytics download method.
The next step is to retrieve the corpse from the module we imported by calling the tagged sense function
note that we specify the tag set as universal, which will give us slightly different targets than default.
The next step is to simply print the data set so that you know what it looks like.
So as you can see, it's essentially a list of lists of tuples, each list within this list is a sentence
represented as a list of tables in each tuple contains two items of a word and the corresponding tag.
The next step is to put our data set into a format that's easier to work with by separating the inputs
and targets, the output of this will be two lists which basically have the same structure as what we
just saw for the inputs will have a list of lists of words for the targets.
Will have a list of lists of tags.
The next step is to do some additional imports specifically for TensorFlow, Nampai and so forth.
The next step is to split our data into train and test.
The next step is to convert our sentences into sequences of integers for this data set.
We'll keep the max vocab size as none so that we get a unique tag for each word.
I've also said lower to false, which means that we will not lowercase the words.
This might be important since a capitalized word might have a different meaning than a lowercase word.
As an example, consider the word bill.
This might be a name which is a noun, or it may be a verb like I'll bill you for the service.
Now, there's a very important argument here, which is that out of vocabulary token, I've set this
string to UNC, which obviously stands for unknown.
So why is this so important?
Well, remember that this is a many to many task.
Recall the default behavior of the tokenize or object if it encounters any word that is not in its vocabulary.
It simply ignores that word and doesn't convert it to an integer.
In this case, that would mean that some words showed up in the test set that wasn't in the train set.
But what would be the result?
Well, being a many to many task, we're going to also have a sequence of tags as the target.
The problem is that if we ignore unknown words, the input sequence length will end up being shorter
than the target sequence length.
Now what's really bad is that this bug is completely silent.
You could run this code and get no errors, but your performance would be bad.
The reason it won't throw any errors is because we'll be padding our sequences.
So even if the inputs and targets have different lengths after we had padding, they will all have the
same length.
Of course, that won't lead to runtime errors, but it will mean that your inputs are no longer aligned
with the corresponding targets.
In other words, some of those inputs will be assigned to the wrong tag, so it's important to keep
all words, even those that are unknown.
The next step is to get our word to index mapping and our vocab size.
The next step is to define a flattened function, which will convert a list of list into a single list.
We'll be using this in the next step.
The next step is to basically ensure that all the targets appear in both the training test sets so well,
first to flatten the train targets into a single list and then convert that list to a set.
OK, so this is our set of train targets.
Now, let's do the same thing with the test targets.
So this is our list of test targets.
They appear to be the same.
Now, just to be very sure, let's check whether or not these sets are equal.
OK, so these sets are equal, meaning that all targets show up in both the train set and the test set.
The next step is to convert the targets to sequences of integers as well.
Luckily, we already have a tool that can do this, which is the tokenize a class.
One small disadvantage of using this is that the class zero will just never be used.
So there is one extra unnecessary class note that there's no need to consider options such as max vocab
size, down casing or out of vocabulary tokens.
Since our set of targets is consistent and also pretty small.
The next step is to pat our sequences.
Well, first wants to find the maximum length of both the training test sets and a sign that to T..
Now you might think that this is cheating, but in fact, this is totally OK.
This is because an Aunt N can actually work with arbitrary sequence length and thus in the real world,
truncating any sequences would not be necessary.
The next step is to pad the train inputs.
The next step is to pad the test inputs.
The next step is to pad the train targets.
The next step is to pad the test targets.
The next step is to get the number of classes.
Note that this is the number of tags plus one since zero must be counted, even though it will not be
used.
The next step is to create our model.
Now this is a relatively straightforward model with just a few lines of code.
Like everything else we've seen.
However, there are again some important points to consider.
Firstly, one minor detail is that we're using a bi directional list here.
This might sound fancy, but it's actually very simple.
Recall that context may be used to help predict the part of speech tag for a word.
Again, consider words like bill or milk that can have different tags, depending on their meaning.
It may be the case that this useful context could come before or after in the sentence, as you've seen,
are it ends only go in one direction from the first input to the last in order.
So that would only incorporate past context, but not the future.
Imagine instead that we created two LSD ms for one of the LSD arms we pass in the sequence as normal
for the other LSHTM.
We read the sequence backwards.
In this way, we know about both the past and future.
Once we get the head in states for both the forward and backward Elysium's, we can simply concatenate
them together into a single vector.
So that's what a bi directional LSD is, as you can see.
It's a very simple code modification where we just add bi directional around the LSD unit.
Also note that we set return sequences to true, since we want all the hit in states, one for each
input soak in, each of these hidden states will then be passed through the final dance layer to get
a prediction for each input.
Now, that was just minor stuff, but I said that there was a change here.
That's very important.
If you look at the embedding, you can see that there's one additional argument here which says mask
zero equals true.
So what is this for?
Well, there is a problem if we use our network as is.
As you recall, this is a many to many network and we patted both our inputs and our targets so that
they all have the same length.
We know that this padding is just a bunch of zeros, but these zeros now appear in the target, which
means we are trying to predict.
Then of course, there's really no way around this because we must have the target be the same length
as the input.
What does matter is how we treat these zeros, and we are currently not treating them correctly.
Effectively, what we would like to do is build our laws function such that it ignores any entries where
the target is zero in terms of the built in cross entropy method and terrorists.
There is no way to do this.
What you could do is build a custom lost function, which I have done in other courses, but unfortunately
the skill level required for that goes beyond the scope of this course.
Luckily, there is a built in method to deal with this increase, which is to set this argument mask
zero equals true.
This will automatically make it so that any inputs which are zero will be ignored throughout the network,
including the lost function.
This effectively does what we want without us having to build a custom loss.
Now, while this is convenient, there is a downside.
The downside is that this is very slow.
I've tested both methods and using the custom lost function I wrote is much faster.
But whether you use mass zero or a custom loss, you must do one of these in order for the results to
be correct.
Unfortunately, you setting mask zero to true will make the fitting process go much slower than it would
have had you done it incorrectly.
Unfortunately, this is unnecessary evil if you want correct results.
Even worse is that this will go slower on the GPU than the CPU.
It takes about 30 minutes per epoch on CPU, but only five minutes per epoch on CPU.
And both of these are much slower than if you had not used this argument at all.
The blame for this essentially falls to TensorFlow, and I've seen at least one person say that the
solution to this is to simply use PyTorch.
In any case, in my opinion, this is the best solution for this class because I can still get the idea
across while keeping the code as simple as possible.
Otherwise, we'd have to write a custom Lawson that would take a lot of work.
The next step is to call the pilot fit, which you've seen before.
The next step is to plot our loss per epoch.
So the last report looks good.
The next step is to plot the accuracy per epoch.
So the accuracy pre-pack looks good.
Now, just as a sanity check, we are going to compute the true accuracy of our model to ensure that
the results we got above are actually correct and to ensure that Mask Zero is actually correct.
The first step here is to simply get the length of each sequence for both the train and test sets.
The next step is to get our models trained predictions.
Recall that the model returns probabilities or logins, so the output shape will be by T by Kay, which
includes padding in order to get the predictions without padding.
We're going to live through the model outputs and sequence lengths in corresponding order.
Inside the loop, we'll grab only the first part of the sequence that is not padded, and we use the
sequence length to do that.
The next step is to take the AMAX to get the class integer for each time step.
Finally, we appended these predictions to our list of predictions.
The final step after the above loop is complete is to flatten both the predictions and unpaid targets.
This is because the methods to compute various metrics in the socket learn do not account for nested
lists.
So it's simpler for us to turn them into one dimensional lists so that we can use the functions inside
it.
Learn.
The next step is to do the same thing for the test set.
The next step is to compute the accuracy and F1 for both the train, its assets.
So as you can see, our model does pretty well.
As expected, the F1 is a bit less optimistic than the accuracy, since our classes are probably imbalanced.
So you want to keep note of the above scores, since the next step is to compare our model to a baseline,
the results we got are meaningless if we haven't confirmed that they perform better than something else.
For all we know, the baseline could be 100 percent and our model is worse.
To establish our baseline, we're going to create essentially the simplest model possible.
Essentially, what we want to do is for each word in the train set.
Keep track of all the possible tags it was assigned, then just map it to the most common tag.
Now, I won't bore you with the code because it's pretty similar to what we did above.
So let's go down to the results.
So as you can see, the baseline model is pretty good, which makes sense for most words.
They only have one possible tag, and if they don't, then it's likely that those words are more commonly
used one way than another.
Even so, our model does perform better than the baseline, which demonstrates that accounting for context,
is useful and better than simply memorizing a tag for each word.
Now, there are several exercises you should do after you've gone through this notebook.
As usual, you should try different settings, so try the grew simple and try different numbers of head
units, different numbers of hidden layers and so forth.
Additionally, you should repeat the exercise with the CNN instead of an art end.
The second exercise is easier, but in my opinion, more important for this exercise, you're going
to take this notebook and you're actually going to implement the two major mistakes I mentioned in this
lecture.
As a reminder, these are the two mistakes.
Mistake number one is not using an out of vocabulary token in the input organizer, as you recall.
This will make it so that the input sequence becomes misaligned with the target sequence.
Since unknown words are simply ignored.
So the wrong tags are assigned it to the input words.
Mistake number two is not ignoring padding in the loss.
This will lead you to thinking that zeros are targets that were meant to be predicted, even though
they do not represent actual parts of speech tags and have nothing to do with the input text.
You should observe that even when you make both of these mistakes, the code doesn't break.
Instead, you'll get 99 percent accuracy even though your code is incorrect.
As a further exercise, consider why the accuracy is so high and discuss this on the Q&A with your peers.
As a final note for this lecture, I want to discuss why these mistakes happen.
I'm pretty confident that every blog and Kaggle notebook I came across made these mistakes.
These mistakes essentially come from a trusting libraries too much.
You think you can just use the tokenize and you think you can just use the embedding because that's
what you saw in the past.
Unfortunately, these default values don't work, and the only way to realize this is to think about
what this code actually does.
In other words, the lesson is that you have to think, and sometimes there is no way around understanding
how things were.
If you just use libraries and you don't have any understanding, you will make mistakes and build models
that don't actually work in the real world.