So in this lecture, we're going to discuss the problem of spam detection, as with the other sections
of this course, which are focused on applications.
We will split this up into two distinct lectures.
This lecture will focus on describing the problem without discussing any solution that is.
Our only goal in this lecture is to understand what spam detection is and why you would want to do it.
The next lecture will focus on one specific solution, and after that we will look at how to implement
that solution in Python.
Now, one surprising fact I learned in view, one of this course is that not everybody knows what spam
detection is.
I personally found this very strange, but I guess my audience is more diverse than I assumed.
So let me turn this over to you, the students.
If you have never heard of spam detection before, please let me know either using the Q&A or emailing
me directly from my website.
Lazy programmer.
Me if after watching this lecture, you still do not know what spam detection is.
Then I invite you to tell me which parts you were having the most trouble understanding.
That being said, let's move on to discussing spam detection.
OK, so what is spam detection?
Well, in order to best understand spam detection, you must have some experience using email, text
message or some other kind of messaging service.
So this lecture will assume you have such experience.
If you do not, then it would be best to perhaps just spend some more time around your computer and
experience these services for yourself.
OK, so what happens when you have some experience with email, text and other messengers?
Well, you learn that not every message you get is legitimate.
A majority of the messages you receive should be from friends, family, colleagues and other people
you expect to hear from.
But sometimes you will get strange messages from unknown people.
Perhaps they are trying to sell you something.
Perhaps they were trying to install malware on your machine, but getting you to click a mysterious
link.
Perhaps they are trying to steal your credentials by pretending they are your bank or some other service
you frequently use, such as Facebook.
One classic scam is called the Nigerian Prince Scam, where the sender pretends to be a Nigerian prince.
They tell you they have a large sum of money, which they need to move urgently out of their country,
and for some reason, they need your help.
They might ask for your bank details so that they can deposit their money into your bank account.
Or they might ask you for a small advance payment in exchange for your help.
They will offer to let you keep a small part of their fortune.
Of course, there is no money to be made because none exists since, as mentioned, it is a scam.
OK, so I hope the premise of a scam is clear.
These are messages that you receive, but you do not want.
Typically, they are from people who are trying to scam you or get something from you.
So now that we know what spam is, I hope it's clear why we would want to detect spam, although would
constitute spam seems obvious.
Remember that the reason that spammers and scammers even bother to send these messages is because some
people do fall for it.
Because of that, companies who run email services and email clients try to filter spam from your inbox.
Of course, another obvious reason is that we simply do not want to see spam, even though we can detect
it ourselves.
We would rather not be bothered by it in the first place.
And just as a side note, this is an excellent example of how machine learning is used for automation.
Sure, you could delete all your spam messages by yourself, but imagine how much time that would take.
It would be better if a machine did that for us, and today that largely is the case.
So at this point, I want to point out another common beginner mistake, which I saw in view one of
this course.
So some students for some reason were confused about where spam detection fits in to an email application.
Remember that the goal of this section is not to build a whole email service like Gmail or a whole email
client like Thunderbird.
That would be a pretty monumental task.
You would need a whole team of people and probably months or years of time to do that with any success.
Building an email service requires lots of other work, which is not relevant to us in a machine learning
course, such as building the user interface, designing the database and so forth.
Basically, what you have to be comfortable with is building only a small part of a big computer program
and knowing where it fits into that program without having to build the whole thing yourself.
Of course, we do that every day in the real world if you work as a software engineer.
But I realize some of you may not have that experience.
So as always, if there is something you don't understand here, please use the Q&A to get that sorted.
So for this class, what you can do is pretend you are a software engineer at Gmail.
Your job is to write a function.
Remember that this is just a function.
You don't have to write all of Gmail yourself.
There are other people at Google working on Gmail with you.
The function as a very simple interface.
The function is called detect spam.
The input into this function is a document which represents the text of an email, SMS message or any
other kind of message.
The output from this function is just a binary value.
Suppose we return one if the input was spam and zero if it was not spam.
In the rest of this section, we will look at how to write such a function and remember every other
part of the Gmail program, like the user interface, the web server and so forth will be written by
your team members.
So your job is only to write this one small function, which is part of a bigger code base.