So in this video, we are going to discuss one of the most popular and powerful machine learning methods,
the random forest, the random forest and other methods like you are today's go to method when it comes
to machine learning for most typical supervised tabular data.
The random forest tends to work very well without much tuning.
This is unlike methods such as deep neural networks, which usually take some manual tuning to get just
right.
So in order to understand the random forest, you first need to understand the decision tree.
So what's the intuition behind the trees?
Well, imagine that instead of machine learning, you wanted to build your own computer program to classify
tabular data.
One common construct we use for building computer programs is conditional expressions.
In other words, if then else.
So let's consider our favorite example, predicting exam grades from the number of hours studied and
the number of hours.
Let's suppose you were to write such a program by hand.
You might do something like if our study is greater than five than pass otherwise fail.
But of course we could make use of the other data point too.
So you might have a nested if statement.
If our study is greater than five and number of hours slept is greater than seven, then pass otherwise
fail and so on and so forth.
Of course, the more variables you have, the more complicated your program will get.
So how is any of this related to decision trees?
Well, let's recognize this fact, if statements are trees, if statements create branches, so if you
draw your logic on paper, you will get a tree.
Now, your next question might be sure, but how do you actually build a tree automatically from a data
set?
The answer to this question turns out to be pretty complex, but you're welcome to check out extra reading
DOT text if you want to know.
OK, so believe it or not, that is all there is to understanding the intuition behind decision trees.
They are simply nested if statements.
Knowing what we know so far, there are some interesting questions we can ask, as you recall, one
of the models you learned about in the section is machine learning is nothing but geometry.
So let's consider what a decision trees decision boundary actually looks like.
Note that at this point we are considering classification.
So here's an example of a tree.
This tree says if you studied more than five hours and you slept more than seven hours, you will pass
your exam, otherwise you will fail.
So you can see that this tree is characterized by horizontal and vertical lines.
This is because a decision tree node can only split based on one attribute at a time.
So every statement in the tree is going to be a greater than expression.
The way this translates to paper is that one side of the space goes one way and the other side of the
space goes the other way.
So you always get straight lines.
Are planes perpendicular to the axis whose variable is being split?
One important fact to note is that trees can be of arbitrary depth, so you can split on the same variable
more than once because of this, it's very easy to get good accuracy on your train set.
In fact, if none of the training points for opposing classes overlap, then you should get one hundred
percent accuracy.
Of course, this is clearly not good because this is what we call overfitting.
Now, since we'll be using Decision Trees four time series forecasting, we'll need to know how they
do regression.
Basically, this comes down to how do decision trees make decisions for classification?
If you imagine that we split up the space of X factors, the decision made by the tree is simply the
most common point in the partition.
So, for example, in the space where that our study is greater than five and that our slept is greater
than seven, most of those students pass the exam.
In other words, most of the points there are yellow.
Therefore the label assigned to that area would be pass.
So you can think of this like voting.
The majority vote wins, but for regression, if you imagine again that the spaces split up by the tree
voting no longer makes sense.
However, what would make sense is if you simply took the average of the points in the partition.
So suppose you're trying to approximate some function, but you only have samples from the function
since you only have a limited number of splits in each partition.
You'll just take the average value of the points in that partition.
What this ends up looking like is a bunch of horizontal lines, so hopefully that makes sense.
The reason they are horizontal is because you've simply taken the average of the points in the partition
defined by the tree.
And consider what will happen if you overfit or in other words, make the tree too deep.
If you overfit, then your tree will just put each data point into its own partition and the result
will be a very weird looking function.
OK, so now you know how decision trees work for both classification and regression, as you recall,
there is one problem with these models, which is that they easily overfill.
So the way to solve this problem is to combine multiple trees together.
So imagine that you take hundreds of trees which are all trained on a different subset of the train
set with a different subset of the input features.
Then you combine the predictions of these hundreds of trees.
This is basically what a random forest is intuitively.
Each tree will make perfect predictions according to their own subset of the Transat, but because they
are all using a slightly different TransFair, they will make slightly different predictions.
It turns out that if you average out those predictions, the result is a much smoother function.
So they smooth out all the unnecessary random variations due to noise.
On the other hand, because decision trees can be so accurate, they don't lose much accuracy when you
combine them.
It's only those tiny pockets of overfitting that go away.
OK, so that's the intuition behind the random forest, it's just a bunch of trees voting together.