So a few students have asked me over the years, how can I apply what I've learned in NLP to other languages?
Now you may have noticed this whole courses in English, but does that mean what you learn in this course
can only be applied to English text?
Luckily, the answer is no.
You can, in fact, apply everything you learned in this course to any language.
In fact, this is an especially relevant topic because of this course has students from all around the
world.
In these situations, I like to remind students that many NLP techniques can even be applied to bioinformatics
and genomics.
In other words, strings of DNA, RNA, amino acids and so forth.
If these techniques can be applied even to languages created by nature, then surely they are capable
of handling languages created by humans.
So in this lecture, I hope to show you that it would be very easy to apply what you learn in this course
to other languages should you find the need to do so.
So let's begin by recalling the high level steps of an A.P. analysis, like the ones we do in this course.
The first step is to simply recognize what our data would look like.
Our data will be text.
This will be the case no matter the language.
The next step is to tokenize the text.
Perhaps we might remove stop words and do some stemming in limitation in order to simplify the tokens.
Once we have these tokens, we need to map each token to an integer.
Let's review why this has to be done.
As you recall, machine learning works with numbers in tabular machine learning.
Our data is represented as a table, with rows corresponding to documents in columns corresponding to
tokens, since each column corresponds to a different token.
And we need to know which token we must have a mapping from token to column index.
Therefore, we must map tokens to integers.
Once we have this mapping, we can then convert our data set into account vectors or 240fps.
Once we have this data matrix of document vectors, we can then apply any machine learning technique
we've studied.
So, for instance, we can create a recommender system or perform spam detection.
We can summarize a document or we can build a topic model to find latent topics.
Alternatively, you can use tokens directly and build a Markov model like we do in this course.
Now there's a reason we're reviewing all these steps.
What I want you to notice is that these are the steps you need to do no matter what language your text
is in.
The only thing that changes is how you do your tokenization.
And I hope to convince you in this lecture that this is not a task of machine learning, but a task
of learning a language.
So, for example, if you want to work with Japanese, it's not a matter of learning different NLP techniques
for Japanese.
Instead, it's actually a task of learning the language of Japanese.
The NLP parts are still the same as what you learned in this course.
To illustrate this, let's consider some Japanese text.
Here are some immediate questions that should come to mind.
Firstly, what is a word in Japanese is a word considered to be one of these symbols or multiple contiguous
symbols.
Secondly, what is a sentence in Japanese?
Does Japanese use the same punctuation as English so that we can find the boundaries between different
sentences?
This also reminds us to think about the boundaries between words.
If we want to tokenized Japanese text into words, we need to think not only of the words themselves,
but also how to split the words.
Another question to consider is what are the stop words in Japanese?
Just like in English, these would be words that are so common that they won't be useful in our analysis.
Now think about the common theme behind these questions.
Are these questions a matter of learning more NLP, or are they a matter of understanding the Japanese
language itself?
I hope you can see that it's not a matter of learning NLP, it's a matter of learning Japanese.
And of course, if you want to learn Japanese, this would require you to take a course on Japanese.
Not, of course, on an LP.
Now, I want to point out another fact, which is that it's not a matter of this course being English
centric, but rather the whole machine learning ecosystem being English centric.
For instance, if you use the count victories or insight, you learn this will not tokenized Japanese
correctly.
So although technically the count victimizer can do tokenization for you, this is limited only to languages
that can be passed by its token isare, which probably assumes English and other languages with similar
syntax.
But this brings us back to the practical world in particular, if psyche learns count, victimizer won't
tokenize my Japanese text, what should I do?
Well, the answer is to simply build a tokenize of yourself using your knowledge of the Japanese language
or to find an open source tokenize are made by someone else.
In fact, I was able to find one within seconds after simply doing a search for how to tokenized Japanese
text.
In other words, this is not very difficult at all.
Now, both of these approaches have pros and cons.
Firstly, if you build your own Japanese tokenize, this requires you to actually know Japanese.
Unfortunately, learning a new language is pretty difficult.
So if you're starting from scratch, this may not be for you.
However, if you're working with Japanese because you live in Japan and you already know Japanese,
then this is a non-issue.
The second downside to this is that building things takes time and effort.
So what about finding a tokenize are built by someone else?
This would be ideal if it works.
It means that not only do you not have to spend time and effort on a pretty low level task, you also
don't have to worry about learning a new language.
But there are downsides as well.
Firstly, if no one has bothered to make its organiser in your language of choice, then this isn't
an option and you're stuck.
Secondly, you have to trust that it works.
So using this method, it would still be advantageous to know the language such that you can properly
judge whether or not the tokenize her is performing as expected.
Ultimately, if you're going to be working with a particular language, you should probably know that
language so that you can analyze the results.
This applies not just to low level tasks like tokenization, but also higher level tasks like sentiment
analysis.
For instance, if you don't know the language, then you won't be able to analyze why your model is
making mistakes.
As an example, you might have texting this movie is good, but marked as negative sentiment.
Suppose your model predicts positive, which is technically incorrect because it doesn't match the label.
In this case, you can diagnose the issue as being an incorrect target, but you can only do this if
you actually know the language.
So the lesson is, if you're going to be working with a different language, then it's probably a good
idea to learn that language.
And again, this is in the realm of a language course, not an NLP course.
So to summarize, this lecture is really just a long winded way to say that most student concerns about
working with other languages are misplaced in terms of the NLP side.
The solution is actually very simple.
Most of what you learned in this course remains the same.
The only part which is different is in how you tokenized the text.
Tokenizing text in other languages really has two solutions.
Number one, learn the language and build one yourself or a number to do a search and find one that
has already been built.
I tried this myself and found some within seconds.
Now, whether or not they actually work is a different story.
But as you can see, this isn't really the kind of thing you expect to learn inside, and I'll be course.
And it's certainly not a machine learning task.
It's simply a matter of doing a search and finding an existing tool or in the case that it doesn't exist.
Building one yourself.
Either way, it's strongly recommended that you actually learn the language you're working with so that
you can understand the results and see whether they're good or bad.
So another question I get sometimes is about word embeddings, which are word vectors we use in neural
networks.
Occasionally, students inquire about word embeddings in other languages.
Again, it really comes down to the same strategy.
You do a search and find out if what you're looking for has been uploaded by someone else.
Unfortunately, there's not much I can do to help you since I would just take the same approach.
So anything I would find you would find as well.
Alternatively, if the word embeddings you need it do not exist.
Well, again, the same strategy applies, simply build them yourself.
Again, this involves tokenizing the text, which we already discussed and then running some algorithm
like words of ECHR glove.
Note that both of these are open source, so you don't actually have to write any code yourself.
Another question I received in the recent past was concerning multilingual models, that is models that
understand the multiple languages at once.
Note that this is an active area of research, so you're unlikely to find any well-established models
that can handle any language of your choice.
One of the main ideas is to make the word embeddings consistent across languages.
So, for example, the word cat in English should give you the same vector as cat in French.
An example of a model that is multilingual is GPT three.
Although do note that it's skill in each language has been said to be inconsistent.