Let's consider this example or this sample SAT prep um exam.
So let's suppose that you get into the exam hall and you realize that out of this, um, eight questions
which have been asked, four were already treated in class.
What this means is that this exam paper is not very great, or will not do a great job at evaluating
your performance as you've already, um, had to treat this, um, four questions in class.
Actually, your performance should be evaluated on this.
Four other questions.
Since you you've not, um, yet come across them in class or you've not yet come across them before.
And so after the exams, you would have, um, all the students getting a score of greater than, um,
let's get back greater than four on eight.
Or we could say, um, greater than, um, 50%.
So everybody in class is going to have, um, a score greater than 50% because the questions have already
been treated and everyone has revised these.
And so at the end of the evaluation, the instructor might think that all the students have grasped
the different calculus concepts, whereas that's not true.
They have just reproduced what they were already trained on.
That's what they already revised in class.
And so to avoid this kind of scenario with our model training, where we actually evaluate our model
on what is already seen, like in this case, well, we say, for example, we train on this data,
we train on this data, and then we evaluate on this same data.
And we have say a root mean square error of ten.
Then that would not be a correct way of evaluating the model because um, it's already seen this data.
So generally what we want to do is split our data up into the train set.
And then, uh, a validation set or let's say a train set and a test set.
Let's keep this one aside for now.
So we have our train.
So let's say we have this part here this first five.
We use this for training and then this other three we use this for testing.
So this is training.
And then here we have testing.
So what happens now is the model uh after being trained on this is now going to be evaluated on data
it has not seen.
Now the whole point of doing this is we want to be sure that the model is able to predict the right
price when given the inputs.
And so if we go ahead, um, or if we go around, um, testing or evaluating on the data it has already
been trained on, then we may overestimate our model's performance.
And so breaking this up like this makes sense because now we are testing on data the model has never
seen.
Now apart from breaking this up into these two parts, we also, um, might decide to break it up actually
into three parts.
That is the train validation and testing.
So now we could say this first four will be for training.
So this first four for training, the next two for our validation and then the last two for testing.
The reason we shall do this is because we want that during the training process.
We have an idea of how the model performs on data.
It has not seen as if we know how the model is performing on data it is yet to see.
Then we could make modifications at the level of the model, or even, um, a level of the optimizers
to see how we could, um, ameliorate our model's performance before we go ahead and test our model
on the test data set.
Now, in our case that we have a thousand samples, so we have 1000 samples, then we want to break
this up such that we have a considerable number of validation samples.
Because the number of samples we have two matters.
As, um, if we have a very small number of validation samples, then we may not be able to properly
evaluate how our model performs on data it's yet to see.
So in this kind of, uh, or in this kind of scenario, we could say we could take 60% for training,
we could take 20, um, for validation, and then we could take 20 for testing.
We could also say we we take 70.
We could say to take 70, uh, we could take 20.
And then here we could take ten.
But the point is, we want to have a pretty accurate idea of how the model performs on data.
It's not yet seen.
Now, if we have if we change this and now instead of 1000, we have say 10 million samples.
So instead of 1000 we have 10 million samples.
Then in that case we could, um, decide to say, okay, we want to have 90% for training, um, 10%
for validation and then another 10% for testing.
And the simple reason here is because we have so many samples, this 10% of those, um, 10 million
samples is enough for us to evaluate how well the model performs on data.
It's not yet seen, because the idea is to ensure that our model, um, is evaluated on data.
It's not yet seen.
Um, operations like the normalization shouldn't involve the validation and testing as that will be,
um, data leakage.
So if, for example, you want to normalize your data and then you take x minus um mean divided by the
standard deviation.
And then this mean is the average of all the data that's even including the validation and test.
Then you would have leaked some information from the validation and testing into the training data set.
And so to avoid this, um, operations like normalization are carried out only here, uh, in our train
set such that we do not have anything to do with the validation and testing.
So these are completely separate.
And then from here we train our model, we obtain our trained model.
And then now we evaluate on the validation and test set.
Now that said we get back to the code.
We dive into data preparation.
So here we have our data which we were uh which we had prepared, yet we had X which is the full data
set.
And then before normalizing let's get let's create the cell below.
And we we have now our train ratio.
So let's say we have our train ratio which we will set to 0.8.
Take that off.
We have the validation ratio which will set to 0.1.
And then we have the test ratio which will also set to 0.1.
So we have validation training and testing.
And then we have our data set size.
Data set size which is simply the length of x which um obviously is a thousand.
You could print that out and check that it's uh, and verify that it's 1000.
So that's it.
We have train ratio, validation ratio and test ratio.
We are going to dive into splitting up our data.
So we have x train x x arm underscore train.
There we go.
We have x.
And then we take all the values from zero right up to 0.8 times the length of the data set.
Because if we if we have 0.8, if we have the split 0.8, 0.1 and 0.1, then what this means is here
we have 800, here we have 100.
And then here we would have 100 samples.
So we go from zero to um actually 800.
Now since it's zero we could take this off.
And then we have int um data set data set size which is 1000 times um 0.8.
So that is times the train ratio.
So 0.8 times 1800.
And that's it.
And then now we we have the Y.
So we repeat the same for the Y we have y.
And then take this off we have y.
So that's it.
We have our train ratio.
Um well we have y train and X train.
And now we could print out extreme X train um shape and then Y train shape.
There we go.
So we run that.
We create a new cell below.
And then we do the same for the validation.
So let's copy this.
And there we go.
Now note that for the validation we are not starting from zero.
We're starting from this um 800.
We go from 800 to 900.
So in this case we have um this 800.
And then we go, let's copy all this again.
So this is 800.
This is 800.
But we add the vowel ratio.
So vowel underscore ratio close that.
So this is now 0.8 plus 0.1 which is 0.9.
So we go from 0.8 to 0.9.
And then we repeat the same.
So let's just copy all this.
And we repeat that for the training for the Y.
So let's copy this.
Um take this off and there we go.
We have take that off and then we have that now for Y.
So let's print out x.
Um.
Train ship, or rather exile ship.
Let's get back here.
We have x vowel, x vowel, and then y vowel, and then x vowel ship and y vowel ship.
Then we do again the same for the testing.
So let's copy this and then print that out.
So you notice how we have 800 here.
And then here we have 100 samples.
So let's go ahead and do the same for the testing.
Take this off.
We have test.
We have test we have test.
And then now we're going from 900 to the end.
So let's just take all this off.
That's fine.
Take all this off and then we go right up to the end.
So run that again.
And you see we have 100 samples.
So that's how we split this up simply now the normalizer is going to be adapted to um on the train set.
So we adapt that to the train.
So the mean the standard deviation is going to be gotten from the train data and not from the full data
set.
Here we have normalizer train.
So train there we go.
So now we have set our we have our new normalizer.
You see we get back to the model.
We it's going to be the same.
Nothing's going to change.
But uh when we get to our model or rather when we want to train our model, we'll specify X train and
then Y train, and then we'll say here we have validation data.
So we specify that we have validation data.
And then we'll give it um x val and then y underscore val.
So that's it.
Let's um put this way so it's more clearer.
And we have this comma and we run that you see with training.
And you could um see we have loss.
We have root mean square error.
Um, we have validation loss.
And then we have root mean square error or we have validation root mean square error.
So once while you are training you have an idea of what's going on.
You don't need to stop finish training before testing you.
While you're training you, you you get your data validated.
And so you could even stop the training after, let's say, five epochs depending on how the model is
performing.
And then you could go and modify some parameters or change the model overall.
So that's it.
That's how we carry out um, validation in TensorFlow very easily.
Now let's get back to documentation.
Come here Keras models.
And then we'll check out the fit method.
We have this fit method.
And you see here that apart from the validation data or by apart from specifying the validation data,
you could also specify the validation split.
So you could have um, all your data set and then decide that your validation split is going to be say
0.2.
So let's say if you took in all x here, if you train it on all x and then you have y, you could instead
of having validation data you could just say validation split.
So so you don't have to split the the data up manually.
So you could just do this and that will be fine.
And then obviously you specify let's say 0.2 and that's it.
So you run that and you see that it still works just fine.
Now let's go ahead and modify the plot.
So such that we have both the validation and training plots.
So yeah, we have this plot.
We'll have our second plot.
Take that off.
And now we'll specify that as a validation.
So we have val loss um y label loss and epochs.
Then for the legend we have here val loss.
So we add that and then we do the same for the root mean square.
So we just add this up here.
Piece it out.
We have vowel root mean square, and then we also have um, vowel.
Let's take this off vowel root mean squared.
So run that again.
And you see from here that the model or the training performance is better than the validation performance,
as we have a lower value for the train loss as compared to that of the validation loss.
That's the same with the root mean square error.
And then now if you want to evaluate your model instead of putting all x, you could say val just on
the validation.
So you now evaluate that model only on validation data, you see.
And then you could also decide to evaluate the model only on the test data.
So let's take this off.
And we have um test data run that and there we go.
So you see it performs differently in the validation and test data splits.
After evaluating the model.
The next step will be to make use of this model to carry out predictions.
So let's say we have for example X test.
Let's take a simple uh a single sample X test zero.
And then we you see we have this eight different elements and it's of shape eight.
So we would um now take our model and do model.predict.
That's the predict method we shall be using and then pass in this X test.
So we have x test and run that and then see what we get.
Let's say this is y output.
Here we have y output.
You see we have this um output right here.
If we print out the y output here we have y output.
You, we have something like 2748.
Actually this has been gotten like this because I retrained again for over 1000 epochs.
So the model got much better.
Um, anyways, that's what we have.
We now dive to the next part.
In this next part, we shall be placing the model's predictions and the actual, um, prices or the
actual expected prices side by side.
So we'll do something like this for each and every sample we will have, uh, we'll plot out what the
model predicts and then what was expected.
And then we'll repeat for all the rest.
Again, here we shall be making use of matplotlib.
So we'll define our figure and then specify the figure size.
We have 40 by 20.
And then we shall um specify the width of each bar.
So let's take um 0.1.
And then we have plot bar.
And then here we have a range of values.
So let's let's just specify um well let's just put that in here TensorFlow range 100 numpy.
And then we have what the model predicts or y predicted.
What the what the model predicts.
We specify the width and we have the label.
So for our label here we have um model prediction.
Model prediction.
Let's take this off uh copy and paste that.
And then now again we repeat the same.
Because this, this 100 is picked because we have 100 test elements.
So we could just say length of test set.
So that's it.
We have what the model predicts.
And then we also have um we have the actual values.
So we'll call this y true or y.
We call this y actual.
Anyways here we have y true.
Keep that short.
Um we have the same width.
And then we have our label now for each and every um, for, for this range we have here because let's,
let's, let's copy this and, and paste our year around that.
As we're saying for this range of values, we want that each time we, we, we plot out the y predicted
to the right, we should plot out the actual, um, values or maybe to the left.
But anyway, we want that shift.
And so what we'll do is we will add a width.
So we'll add a width to this values.
So you see that if you add plus plus a width you find out instead of 0 or 1 we will have 0.1 and then
one.
Let's take numpy.
NumPy um on that.
That should be fine.
So here we have uh width is not defined or okay.
Width is not defined.
Let's say 0.10.1.
We run that again and that should be fine.
So while that's running we have um plot um x label.
And then we have actual actual versus the predicted price.
And then we have plot y label.
We have car.
Rice, uh, car price.
Okay, so we have plot, um, show.
There we go.
Plot show.
That's fine.
So now the next thing we'll do is actually get this y and y true to get the y true.
That's the actual values of the prices.
What we could do is we we would get the y test.
So y true y true is equal y test.
But since y test is of shape 100 by one, what we'll do is we'll just, um, take all and the rows we'll
take all, we'll pick all the rows and then um, and then pick this um, first column.
So let's take that off.
That's our y test.
We print out y true.
And there we go.
So you see we have this value.
We could take numpy take numpy.
And that's fine.
So we'll now have this simple list.
So these are all the different um values for all the actual prices for the cars from for the test set.
And then we'll do the same for the model's prediction.
So let's copy here and then paste out.
So for y test instead of y test now we have the model which takes in x.
So we have this test that is we have what we know.
This is what we know.
And now we're trying to compare it with what the model predicts.
So instead of y test we're taking x or big x.
So we have big x.
Um the model takes in all this um outputs um 100 by one.
And then we have all and then zero.
So let's print out y pred y predicted what the model predicts y pred.
And then here we have that shape.
There we go.
Let's run that and see what we get.
You see we have 100.
Let's just let's print all this out and let's see numpy.
NumPy.
And then here we have numpy.
So we could now take this off.
So we now have our y true and our y pred.
So this is what the the model predicts.
And then um just below we have what the model or what was supposed to be predicted.
So now we have all this.
Let's run this and then see what we get.
Let's take this off and we can now look at our bar chart.
You notice that, uh, we have one in blue and one in, um, orange.
Now, this here just looking at the values.
You see that the there's one, there's the y.
True.
The actual Y's, um, seem to be much larger than the what the model predicts.
So our model is, is clearly underperforming.
So you could see that from this chart.
See here in blue.
Let's let's maybe zoom in so you could see that clearer.
Uh you see take that up okay.
So you see just here in blue you have the what the model predicts.
And then in orange you have, um, what was expected of the model.
Let's get back.
And there we go.
So that's it.
Um, for the section, we have just evaluated and tested our model.
The next section, we'll look at corrective measures to make our model much more performant.