In this lecture I want to introduce you to the concepts behind tax pre processing in the previous lecture
we learn that obviously you can't multiply words by numbers which doesn't make any sense.
In order to use deep learning which really means neural networks we must have numbers because a neural
network is basically just a series of multiplications and additions.
So the question now becomes how do I turn my text into numbers specifically.
We need the kind of numbers that can be passed into an embedding layer which means we need integers
that correspond to rows in an embedding matrix
let's start from basic and hopefully pretty obvious points.
First how is text stored.
The answer is text files to be clear I mean in plain text files and not Microsoft Word or anything like
that.
Hopefully you've done enough Python so that you know how to open a text file read its contents line
by line and so forth.
Hopefully you know that Python files are themselves text files so text files are a fundamental ingredient
of computer programming.
You should love them.
OK so let's say I read in a plain text file using Python.
How are those file contents represented in Python code.
Well the answer is as a string.
But wait.
If the whole file is a string.
How can I get the individual words and sentences.
Remember that in order to convert my words into word vectors those words must be represented individually
as integers but starting out.
I have nothing but a humungous string full of multiple sentences and each sentence containing multiple
words.
No good
OK.
I'll make it a little easier on you.
Suppose we impose some structure on our tax file.
Let's make it a CSP.
We've all seen C views before.
So I hope there's no need to explain what these are.
Let's suppose since this is what we're going to see later anyway that this is a two columns ISV where
the first column is some string representing an email or a tweet that we want to classify.
And the second column is its corresponding label.
This makes things slightly easier because each line of the file consists of only two parts an email
or a tweet and some corresponding label as a side note.
Sometimes people including myself will refer to these emails or tweets simply as documents that may
be confusing to you as a beginner because you might think of a document as an entire file.
Like a book or a report.
But in A.P. when we say document that probably just refers to a single sample which could be a sentence
or maybe a few sentences.
It could possibly be longer but that depends on the task you're doing so be aware that the word document
is kind of an overloaded term and an LP.
It doesn't mean a document as an entire file like a word document.
It just means a single sample in your dataset.
So from here on out when I say the word document I'm going to assume you understand what I'm talking
about.
Now that our data is in the form of a CSP maybe we can use pandas to load it into a data frame.
But the problem is each of the cells in the first column of this data frame are still documents they
contain multiple words when what we really want is individual words in the next portion of this lecture.
I want to discuss how we might process these documents in pure Python code to give you an understanding
of how this works and how you might implement it yourself in the end we're going to end up using a library
that's part of the pie to ecosystem but it's quite immature and limited in its capabilities so it's
entirely likely that in the future it may not fit your needs and you may need to write your own textbook
processing code so it's good to have some idea of how this works in my more advanced and in-depth and
AP courses.
We actually go over how to do textbook processing in Python.
So if you're interested in getting practice with that you can ask me about it on the Q and A
in any case let's start with what we have a document which contains multiple words and possibly multiple
sentences.
We know that we want to split this up into individual words.
This is a process called tokenization.
If you wanted to take the simplest possible approach you could achieve this by taking your document
which is a string and just calling the split function.
This is probably fine and a lot of cases but there are some downsides to this too such as that.
It doesn't take care of punctuation consider the document.
Today I walk my dog my dog chased a rabbit we see here that although the word dog appears twice the
two times it appears are different because the first time there is a period appended at the end.
So if you wanted to say assign a unique integer to each word that appears in your dataset these would
appear as two distinct words.
That's possibly not good.
So you may want to have some code that removes punctuation.
You might also want to down case all the letters as well although there have been models that kept the
original casing.
Let's suppose that we've made it to the next stage where we've successfully token eyes all our documents
so that instead of just single strings they now appear as lists of strings where each string as an individual
word.
As you know the next step is to convert these words as strings into an integer representation.
Clearly each word must have a uniquely assigned integer for this to work.
So the word a might be a sign that Integer 0 AA might be assigned one and so on until we get to the
last word in the dictionary Zeitung which might be assigned 1 million.
The question is how do you write this in code.
Again this is all just conceptual for now.
But here are some ideas.
First you define some counter called current index and you make that zero.
You also instantiate a set which keeps track of every word you've seen so far.
Then you live through each word and your corpus by the way Corpus is just a fancy term for your text
data set.
So each time you encounter a word that you haven't seen before.
You do two things.
First you add this word and the current index which will be assigned to this word to a dictionary.
We'll call this the word to index dictionary since the key is a word and the value is its index.
Second we increment the current index by 1 so that the next time we encounter a new word it will be
assigned a new index value.
Pretty simple I hope
one caveat to the previous pseudocode is that usually we don't want to start at the index 0.
Instead I propose that we start at the index too.
So why would I want to do that.
Well if you recall each sentence in our dataset might be of different length.
So we need padding padding is usually done with zeros so zero is typically a reserve number for that
use case.
But why start at 2 and not 1.
Well remember that we're going to have a train set and a test set.
What if our test set contains words that didn't exist in the train set then these should be considered
unknown words because we couldn't have learned anything about them during the training process.
And so for unknown words we should assign that next one and therefore we have two special indices zero
represents padding and 1 represents unknown words as a side note in the library we'll use later called
talks text.
It actually does the opposite but obviously that doesn't change the point.
So in actuality zero is unknown and one is padding.
And of course this is only the case if you use towards text and you don't write this yourself.
OK.
So what's the next step.
Remember that we would like to have constant like sequences or at least constant Length Sequences per
batch.
So let's suppose we have a batch of sentences each of different length the first sentence is nine three
one hundred twenty seven and six.
The second sentence is 14 21 and the third sentence is ninety nine five hundred one eighty seven and
to remember at this point we've converted each sentence into a list of tokens which we then converted
into a list of integers what we can see that the length of these sentences are five two and four respectively
since the longest sentence has length five.
The two shorter sentences would have to be padded to be of length five.
So the second sentence becomes 14 21 0 0 0 and the third sentence becomes ninety nine five hundred one
eighty seven two and zero.
And so just a reminder we're using the convention that zero equals patting.
Even though this might not be the case in the code
but here's an interesting idea.
Instead of patting at the end why not Pat at the beginning.
So the second sentence becomes 0 0 0 14 20 one and the third sentence becomes zero ninety nine.
Five hundred to one eighty seven and two in the first case we call that post padding because the padding
goes at the end of each sentence and in the case we're considering now we call it pre padding because
the padding goes at the beginning of each sentence as a quiz.
Think about why we might want to choose one over the other which one might be better for tax classification.
Using our own ends and why.
I'll give you a minute to think about it or you can pause the video and come back when you're ready.
All right.
So hopefully you thought about your answer to the question which might be better for tax classification.
Using Arnold's pre padding or post padding the answer is padding in our case the Arnold will read the
sequence from left to right.
Or equivalently from beginning to end.
That's not a problem in and of itself but recall what challenges do Arnold's face.
The main challenge for our own ends is learning long term dependencies due to the vanishing gradient
problem.
So if we put padding at the end what happens.
That means if there was any pattern to be learned from the sentence it's going to appear at the beginning
and if there's tons of padding say 50 or 100 zeroes then there's a good chance that by the end of all
those zeros the Arnon will have completely forgotten what it saw earlier and due to the vanishing gradient
problem it's going to be very hard for the aren't in to learn to capture those patterns that appeared
at the beginning of the sentence.
Therefore it's better if you're doing a many to one task in the left to right direction that you put
your padding at the beginning.
Okay so let's summarize all the conceptual steps that need to happen in a text pre processing pipeline.
First your data might arrive in some weird format like for example and maybe be a collection of web
page else or a database table or you might be using the Twitter API to download tweets as Jason documents
for reasons you'll understand very soon it'll be convenient if you can format these into a CSP.
Second when you load in your data whether it's a CSP or not this data won't be in the form of one string
per word in order to convert each document into a list of strings where each string contains only a
single word or token.
We need to perform a process called tokenization.
Third once you have your documents encoded as lists of strings you need to convert each of those tokens
into an integer representation.
That means you need to assign or map each token a word to a unique integer and in addition you'll need
to convert each sentence from a list of strings into a list of corresponding integers.
At this point your dataset consists of nothing but lists of integers finally.
You'll want to perform padding so that your data set or at least each batch has the same sequence.
I hope that none of these steps seems particularly difficult to write by yourself in Python code.
As an exercise I would strongly recommend writing your own Python code to perform each of these steps.
This isn't strictly required for the upcoming example but it's an extremely useful skill to have especially
if you plan on doing something unique and not covered by the basic use cases in existing libraries.