Hi there and welcome to this new section.
Previously, we saw that we could train a model such that it takes in input um output pairs.
That's this input output pairs.
And then later on when given new data is able to tell us, um, what the price of the car is based on
this new data.
And what this tells us is our model relies a lot on the data for performance.
And so in this section we shall be focusing on how to prepare our data, um, so that the model makes
use of it for training.
So we're going to head over to Kaggle.com.
And more specifically, uh, on Mayank Patel's Kaggle data set entitled Second Hand Cars Data Set.
Now this data set is what we'll be using in this section.
So here we have the different features.
You go from the ID to on road or on road now years kilometers ratings and so on and so forth.
You'll note that we are not going to make use of the on road hold and on road.
Now, as these two features aren't very clear.
Generally, in machine learning we want to have an idea or better still, know in detail what the features
we're working with.
Um, are.
So that's it.
You could, um, open your detail.
You have all the details of the different features.
So for the ID one, we have the, the on road oh five 535,651 on road.
Now we have number of years.
So this tells us that this car uh with ID one um, has spent three years.
The kilometers, the number of kilometers covered is 78,945.
The rating is one, the condition is two, and so on and so forth.
So we have all this information right up to the horsepower.
Then if we click on column here you have this um sort of summary.
So for every feature let's take the number of years.
For example, you have the number of valid, um samples we have in our data set.
We have a thousand valid samples.
The number of mismatches zero number of missing samples zero.
The mean is 4.056 and the standard deviation is 1.72.
Check out another feature.
Like the number of kilometers covered, you have a mean of 100,000.
Um while standard deviation is 29.1 thousand.
So what this simply means is that, uh, most of our values, most of the values we have, will lie
in the range of 1000 -29.1 and 1000 plus 29.1.
But our normal distribution curve will look like this, where we have the mean at 100, um thousand
and standard deviation, uh, which is 29.1, meaning that the values falling within the range 100,
because here we have 100, um, -29.
Obviously this isn't, um, in thousands since we have k, so 100 -29.
And here we have 100 plus 29.
That's essentially about 90 about 81.
It's about uh, not actually 81, 71.
So we're going from 71 to 129.
So what this means is most values, uh, fall within this range.
Or the fact that a value is in this range means that it has a higher probability of, um, appearing
in our data set.
At this point.
You could go ahead and click right here and download this data set.
So it's just 25kB.
You will have your train dot CSV file downloaded.
Then when you download and extract that file you could simply just drag this into your colab, um,
notebook.
So notice how here you have this drop files to upload them to session storage click you drag and uh
drop in here.
You should also note that once you disconnect from the session, you're going to lose this file.
So, um, it's only, uh, in the session that we have access to this file.
And so with the file, um, copied in here, we could start with importing.
So let's import um TensorFlow.
TensorFlow as tf.
Well this is for working with uh models.
So modeling or deep learning modeling then we have import pandas.
Pandas as PD.
This will be used for reading and processing processing uh data.
And then we have import seaborn as SNS.
This will be used for visual.
Asians.
Now we could run this and then click open our train.csv file.
You see we have this um, train dot csv file right here could show 100 samples.
You see that we have a 1 to 100 of 1000 entries.
So we have 1000 different, um, input output pairs.
Um, exactly the same as we had seen on Kaggle.
So here we have the ID we have on road or on road now, years, kilometers and so on and so forth.
Um, up to talk.
That's for the inputs.
And then now we have the output which is the current price.
So with that we close that.
And then now let's copy this path.
So we copy that path and then we do data pandas um read CSV we specify that path.
So let's let's just create um a file path here.
File path.
There we go.
We have our file path.
Now we specify our file file path.
And then we also specify the separator.
So here we have the CSV which stands for comma separated value separated values.
And here given this separator or comma simply tells us that um, each and every value separated by a
comma.
So that is that simple.
So let's run this and then we have data head and we see what we get.
Um open this train dot csv file with your text editor.
And one thing you will notice is the fact that you could see how all the elements in a given row are
separated with a comma.
So we have the idea one.
We have on road old 535,651, which is this.
Um, let's get back.
We have the on road now 798,000.
That's it here and so on and so forth.
So you have three, you have the kilometers.
You could scroll.
So you get to the current price.
We have the current price 351,318.
And this comma two separates the header rows, different columns like the vid on road or on road.
Now up to the current price.
If you happen to work with csvs which are separated by something other than the comma, like say for
example semicolon, then you would want to change the separator and have the semicolon.
Now with that said, let's get back to our um, csv file.
This is just an extract, um, of 21 samples.
Um, here you could see already that this CSV file contains our data set.
Looking at this, we could break up our data set into the two main parts.
There's the inputs and the outputs.
So we we have let's just take all this we have here the inputs right up to this point.
We have the inputs see.
And then we have um to the right we have the outputs.
So if we have 21 um samples like in this case then it means we have 21 rows.
So we have um a 2D tensor which is 21 by number of columns.
Here we have 123456789 1011.
So we have 11 columns.
While um on the side of the output we still again have 21 rows.
But this time around we have only a single, um, column.
Uh, now we could generalize this so we could, we could go from 21 to n.
So depending on number of samples we have, we could have n um and then we have 11.
And then here we have n by one.
So we have this 2D tensor which is uh n by 11.
And this other 2D tensor which is n by one.
After loading the data you could still check out the shape.
Now note that this shape contains both the inputs and the outputs.
So the n is maintained or true because we have 1000 samples.
But we now we have a combination of the inputs and the outputs.
So we have 11 plus one which is 12.
From this point we will make use of Seaborn to carry out some um visualizations.
So we'll make use of this um, pair plot method from Seaborn and compare the different um features.
So we have number of years, kilometers, radian, um condition up to talk for the inputs and then for
the output.
We have the current price.
Anyway, in this pair plot we are comparing each and every, um, feature with with the others.
So we'll compare years with all these other features and for example economy with all the other features.
So let's run this and see what we get here is the overall pair plot we're getting.
And if we copy this um, on our board, you see that um, for each and every.
One for each and every feature we have, um, a plot which shows how that feature is related to the
others.
So this plot here, for example, shows how years are related with a feature, years with itself.
And then this plot shows how years is related with, um, this other one kilometres.
So let's, let's just look at this.
Let's zoom in and take a look, a closer look at this year we have years.
And yeah, we have kilometers.
So, um, the way kilometers is related with the number of years tells us that whether a car has been
used for maybe, say, two years or six years, depending on the driver, we may have, um, a higher
value for the number of kilometers covered or a lower value.
So if we, we have a car which is new or brand new and that, um, is used, um, maybe to go for work
and get back home, then we could fall around this range here you see fewer kilometers covered.
But if you just bought a brand new car to be used as a taxi, then you would fall, um, in this range,
as you you are, you're basically using the car, um, for the whole day.
Also, you may have, uh, bought a second car or a third car.
So you have maybe three cars, and you, you, you rarely use a third car.
So even after seven years, the car hasn't run so many kilometers.
On the other hand, even with normal usage after seven years, we expect that the car has, um, run,
uh, many kilometers.
So actually, this shows us that the number of kilometers covered, um, depends on the usage of the
car.
And so whether the car is one year old or seven years old, the usage or the number of kilometers covered
may be different.
One other feature which we would like to visualize is, um, this one comparing the, um, the current
price and the number of kilometers covered.
You will notice that as you cover, as you increase, uh, as there is an increase in number of kilometers
covered, the price tends to drop.
And unlike this other features where we don't have a direct or inverse, um, correlation with the current
price, if you increase the number of kilometers or if the car has run through so many kilometers,
then the car's price tend to drop regardless of the usage of the car, regardless of the type of car
or any other factor.
So this particular feature is really important.
With that said, you could feel free to check out the Seaborn documentation, um, to learn more on
how to visualize data so that you could take, um, much better and quicker decisions.
Let's go ahead and add, um, this header right here.
So we would have data data preparation.
There we go.
We have that header before we go on to extract the data and create the separate inputs um x and output
y.
We are going to shuffle this data.
Now the reason why we're shuffling this data is because we want to avoid the model being biased based
on the order in which the data was collected.
So we shall simply create, uh, a random, uh, shuffle data.
So here we have shuffle data, um, TensorFlow random um, shuffle.
And then you pass in the data.
So that's it.
Now if we do shuffle um data head.
Now if we get the first five elements of the shuffled data, shuffled data, um, first five elements
and then we compare with the data first five, you would find that the ordering is going to be different.
So here unlike the data which is 12345 the shuffle data is 288 um 201 882, 557 and 313.
So this shows us that our data has been shuffled.
Then since we want to get x and y we'll simply have our x.
Now this x is going to contain the.
If you look at data let's just print.
Let us um print out Data.head here.
So we have data Data.head.
There we go.
Our X is going to contain just some parts of our data, of our shuffle data.
First of all it's going to contain all the rows.
Remember we had seen this already in the section on tensors and variables.
So we get all the rows.
And then once we get all these different rows, the next thing we'll do is to collect or select um the
specific um columns.
So here we want the this column.
This is the zeroth column 012.
And this is three.
Because we don't want on road now we don't want on road old and we don't want the ID, we want years
um up to the talk.
So we want to be three right up to the end.
And then for Y we want the shuffle data.
Shuffle data.
Um, and here we want everything again and just the end.
So let's run that print out X, and then we print out y.
We have your X which is of shape 1000 by eight.
And we have y which is of shape um 1000.
So one d tensor.
Let's just print out the shape.
So you could see that we have y shape and x shape.
Now we want our inputs.
Uh that's x and y to be 2D.
So we're going to go from this 1000 shape to um 1000 1000 by one.
So that's what we want to do.
And we're going to make use of the Expanddims method.
So let's get back here and we do TensorFlow expand dimensions.
And since we want to add this to the end we're just going to have negative one.
So we run that again.
Um and then let's take this off.
And then we print out the x shape and y shape in general before passing our input data into our model.
What we want to do is to normalize this data to speed up the training process.
And so we are given, um, this input data.
Uh, let's call that x.
We shall subtract each and every one of those values by a mean value and divide by a standard deviation.
And so what the model is going to get is actually an x prime which is equals the normalized um or which
is the normalized version of our x.
Now the way we could get the mean and standard deviation is by simply going through all the different
values, getting the mean and the standard deviation.
Now you should take note that we also don't want to include data, which the model will not see during
training.
So we only want to get this mean.
And the standard deviation from um, data.
The model is going to be trained on practically.
The way we could normalize our data is by making use of TensorFlow um, normalization layer.
So you see here we have the simple definition.
We have this arguments which are well explained.
And so you could always feel free to check this out.
And then you will um have some examples.
Now let's get back here.
And then we import we're going to import TensorFlow.
TensorFlow Keras layers um or rather from from TensorFlow Keras layers.
We're going to import the normalization normalization.
There we go.
So we import that.
Let's go down and then show how we could make use of that normalization layer to normalize some um random
data.
So here we have um our 1D um tensor let's say 34567.
There we go.
So we have this 1D tensor.
And then we shall define a normalizer.
We have our normalizer our normalization.
Normalization.
There we go.
And then what we do is we say normalizer.
Normalizer takes an A.
So let's print it out and see what we get.
Um let's print that out.
You can see that when we don't specify any value here.
Well we get returned the exact same input.
Now let's modify this and say we have five and four or let's, let's be more specific.
The mean is five and the variance is four.
Note that the standard deviation is the square root of the variance, the square root of the variance.
And so if you are doing um x minus the mean um divided by the standard deviation is really just x amount
minus the mean divided by the square root of the variance.
So let's put this brackets here.
So see that clearly x minus mean divided by square root of the variance.
And so um given that we specified the variance and the mean.
Now let's run this again see what we get.
You see we have negative one -0.50 0.5 and one.
Now if we try to test this out let's well let's get back.
Let's say we have um let's take three.
So here we have three three minus the mean.
The mean is five.
Then um divided by the square root of the variance which is um two.
So that gives us negative one.
And that's how you have this negative one right here.
So when you specify this mean um, and the variance when you specify the mean and the variance for each
and every value, we're going to run this computation.
And the values are now going to be normalized.
So that's it.
We could also test this out for 2D.
Um tensor.
Let's get back here.
We have this 2D.
Let's say we have six seven, eight.
Um, whether you say for four, five, six, seven and eight, close that, run that again.
See, we have as, uh, as we had this previously, but now for this other ones, we have um, -0.50,
0.51 and 1.5.
You could test out for eight, for example, if you have eight, eight, um, minus five is three divided
by the square root of the variance, which is two, three divided by two is 1.5.
That's how we have this 1.5 right here.
Now um, when working with data, we generally don't know the mean and the variance upfront.
And so because we don't know the mean and the variance upfront, we want to be able to calculate this
automatically.
And so what we could do is we could um allow our normalizer to adapt um its mean and variance based
on the data.
So let's get back here and say normalizer.
Normalizer um adapt to a.
So we want our normalizer now to adapt to a.
And so now we don't need to have this five and four again.
So let's run this.
You see we have these values negative ones and then ones.
Now let's explain why we have these values.
First of all getting back to the documentation you would find that we have um this axis set to negative
one by default.
So if we set the axis or if we change the the the axis value, if we say axis is zero, you, you would
find that we will have completely different values.
So having the axis um, or not putting any value is the same as having the axis to be negative one.
Now when for a 2D tensor like a, you specify that the axis is negative one, it means that you are
going to um, compute the mean and the variance along the columns.
Remember we have a a has shape um, two because we have two rows and then five columns, it has shape
two by five.
And so when you specify access to be negative one that's you're specifying the columns.
What you're saying is you want to compute the mean along those different columns.
So for this first column three and for our mean is 3.5 which is logical because if you do let's get
back here.
If we do three plus four um, divided by two you have 3.5.
And then the variance is going to be 0.5.
So what we're saying is if we have this two values that's three and four then our mean our mean here
would be 3.5.
Now um to the left another the the value we have is three.
And to the right the value we have is four.
Now what um, separates three and 3.5 is 0.5.
And what separates 3.5 and four is also 0.5.
And this happens to be our standard deviation.
So in this case our standard deviation is equals 0.5.
While the mean the mean here is 3.5.
And so now if we do three let's take this off.
If we do three -3.5, um, divided by -3.5, then all of that divided by 0.5, what we're going to have
is um, -0.5 divided by 0.5, which is negative one.
And then if you do let's just copy this.
If we do, um, four -3.5 divided by 0.5 you would have one.
So that's how we obtain these values negative one and one.
Now for the rest is uh similar situation because we have 4 or 5, the mean there is 4.5 and then we
have four.
And then the standard deviation is 0.5, because the gap between 4 and 5 is one.
And then when you take one half you have a standard deviation of um 0.5.
Now if you replace this by five you see again and 4.5 here we have negative one one.
So that's how we obtain um all this.
And now let's add another.
Let's get back and then add another um row.
So let's say 912 um four and then zero.
Run that again.
Uh we're getting this error.
Well let's take this off.
Run that again.
You see, now that we have, um, our, um, inputs, which is this which have now been normalized.
And so now we could go ahead and normalize our x.
So we're going to create x normalized.
We could just have this.
We have take that off.
We take a off.
And then we we have normalizer and we adapt our X.
And then we'll have this to be x normalized X normalized is simply normalizer.
Um and we pass in the x.
So that's it.
So take that off and then let's print out um x normalized x normalized.
There we go.
We now have the values of x which have been normalized.
We still have our 1000 by by eight shaped um tensor.
So that's it for this section on data preparation.
We are now going to move on to modeling.