So in this lecture, we are going to discuss how to do text preprocessing.
Now, you might notice this lecture has a funny name.
I've called it beginner blues, pi torture NLP version.
So what do I mean by this?
Well, let's begin with the fact that this is a brand new updated lecture, which did not exist when
I first released this course.
In fact, there have been a few free updates to this course over the years.
So why does this lecture exist?
Well, you may have noticed if you went through the Legacy and Alpay lectures, they use the library
called torture text.
And at the current time, the torture text library has been updated such that the previous code does
not work with the latest versions.
Many beginners got stuck because of this change.
Now, the reason I wanted to make this lecture is because over the years I've seen many beginner mistakes
being made and many misconceptions that need to be addressed.
So if you are one of the students that got caught wondering how can I do an LP?
If torture text change their API, then this lecture is for you.
So let's start this lecture by saying that this is not something anyone who is a student of mine should
have gotten stuck with, basically I taught the skills to deal with this sort of thing approximately
five years ago.
So this is not new technology by any means, merely a change in the API.
So that's one of the major downsides of being dependent on libraries.
If you haven't studied my in-depth series of courses and you depend on libraries to get things done,
then you will pretty much get stuck whenever anything changes.
On the other hand, students who have taken my in-depth series of courses tend not to get stuck because
they've been taught the skills such that a change like this is not even an issue.
So in this lecture and the coming lectures, we'll be borrowing a bit of content from my in-depth series,
specifically deep in AP, Arnon's and HMD, where I first taught this technique many years ago.
Now, you might think I'm just being too hard on beginners, but here's the reality, what I'm about
to teach you in this lecture is a very practical skill used very often in the real world.
In fact, it's so useful.
I use this as an interview question for data science employees.
Basically, the question is to perform the textbook processing steps in Python, which in this course
we had previously done using torture text.
The hard truth is I would never hire someone who could not write the code to solve this problem.
In fact, my colleagues loved this question as well.
And so they are also using this question in their interviews.
So don't be surprised if you go to your data science interview and they ask you this question.
We use code like this every day.
And so if you can't write this kind of code, we will not give you the job.
Thus, if you were one of the beginners who saw this and just gave up or got frustrated because the
library changed on, you realize that this is not a good attitude for a data scientist to have.
If this is your attitude, you are not getting the job.
The main theme of this lecture is basically how you can be a builder and a leader instead of a follower
and someone who only commits to rote memorization.
Now, I do want to mention that most of you did not get stuck.
For the most part, I find that the students of my VIP courses tend to be more mature and well-rounded,
not always, but to a greater degree.
So this is a good thing.
If you were one of the students that did not get stuck, then congratulations.
You are on the right track.
Another falsehood I want to address before getting to the main content is that beginners often think
they should always be using the latest version.
In fact, I have a YouTube video about this called Why Bad Programmers Always Need the latest version.
The reality is this is not the way things work in the real world.
You see, as a business, you really only have two choices.
Number one, do the work that your clients have asked for to produce real meaningful output and get
paid real money or no to do behind the scenes work, such as updating libraries, which does not produce
any change in output for your clients in which you do not get paid for.
So as a business, we tend to choose number one rather than number two.
Beginners often feel like everything has to use the latest library because they haven't yet had the
opportunity to work on real world projects.
Many times we even skip versions.
So today we might be using version five and we might not update until version 10.
Beginners who think they have to update their code every month whenever a new version comes out are
not very practically minded.
This is not how we operate in a real business.
So for the rest of this section, I'm actually going to leave the legacy lectures up since you'll need
to watch them in order to complete the following, exercise the exercises to write your own code, to
replicate the text preprocessing steps, that is, write the code to replace the target text functionality
that we need.
Now, you might wonder, how can I do this if I haven't yet taken lazy programmers deep in AP course
is true.
This exercise will be a little easier if you've taken my in-depth series of courses.
But it's not true that you can't complete it.
If you have not taken my in-depth series of courses to complete this exercise, you only need basic
python knowledge.
And actually this is why it surprised me so much that so many people got stuck.
So to summarize the exercise, the steps are roughly as follows.
Number one, tokenized each document, you can use any tokenized you like, but the simplest method
is to simply call string that split number to map each token, a two a word index.
As you recall, these will be used as indices to the embedding matrix.
Note also that you'll need special indices for padding since each batch needs to have the same length.
Number three, once you have your word to index mapping, convert all your documents into integer format.
Number four, use your integer formatted documents as input into your neural network.
Of course, the last step you already know how to do if you've completed the previous sections of this
course as a bonus, you should also create your own data generator that produces batches of neural network
inputs.
Remember to also return these as Tahj Tensas.
As a further bonus, the sequence length of each batch should only be as long as the longest sequence
in the batch, otherwise the sequence length can simply be the maximum document length in the whole
data set.
OK, so as you'll see, this is nothing but a review of what is already shown in the legacy text preprocessing
lecture.
So watch that lecture to see an example of how your code should behave and then write your own code
to reproduce that behavior.
Now, after all this, you might be wondering why not simply use the latest version of torture text
and to these students, I say you must watch this whole lecture again because the purpose has evaded
you.
You see, your approach of using libraries is the whole reason why you are stuck here in the first place.
Of course, if you learn today's library, you're just going to be stuck again three months from now.
And then six months later, when it changes again, you're going to be stuck again.
So hopefully you see why this approach is a bad idea.
Furthermore, the torture text API is just not that nice, which is, in my estimation, why they decided
to change it in the first place.
Unfortunately, the latest version is also not that nice, in my opinion.
Not really worth learning.
In fact, I've seen several courses that simply use the CARUS text module instead because it has a much
nicer API instead, a better and more powerful approach is to actually learn how these things work.
Getting back to the real world, often when we're hiring people, we don't care which APIs you know
how to use, we care that you can write code when we're hiring people.
We don't want people who are going to get stuck every time a library changes.
Those employees tend to need too much handholding to get things done.
And as such, my recommendation is to not be that kind of employee.
The main reason why beginners were getting stuck is because they were just trying to memorize syntax
without any understanding of what it was doing.
So the approach of rote memorization is very bad in programming.
So not only is the approach in this exercise nicer, it's also conducive to improving your understanding
of how things work.
Another question you may have is, isn't it always better to use a tried and true solution instead of
rolling your own?
And for this type of question, it's obvious you have to apply some critical thinking and context.
Obviously, you're not going to build your own database or a Web server framework.
That would be absurd.
You won't build your own SVM or random forest unless, of course, you're taking a course on SVM or
random forest, in which case that would make sense.
If you're doing a Kagle contest, then of course, you're not building your own SVM for the contest.
So it's all about context.
Normally, big complex code bases like a database or a web server would not be built from scratch.
Simple code like looping through a log file to pass out the important fields would be built from scratch.
So there's always a spectrum in the case of text preprocessing.
What you'll see is that this actually falls on the simple side instead of the complex side.
In fact, another reason, aside from all the good reasons already given, is performance in the section
on recommender systems.
We'll see that building your own data loader is actually more performing than using the built in utilities
and pie torch.