In this session, we will discuss about dissolution of the used car price prediction problem, so this
Craigslist, which is the data center, which we have, is the world's largest collection of huge vehicles
for sales.
And this particular dataset includes every used vehicle for within the United States, which is present
on Craigslist.
So initially we would include the libraries, which are known by abundance, even as I live in invalidly
leadership and were of off.
So as you can see, this particularly date Kansas, the I.D., UARS region, region area and many other
object type data columns.
So we will be converting these data columns into medievalists and such data columns which don't really
have any relevance related to the price we would be dropping them.
So the dataset contains around five like thirty nine thousand rolls of data.
And these are the columns which are you as you can see, the columns you are in the region, you all
image you are in these columns have no relevance to us.
So we would be removing these columns and the other object types we will be analyzing and then seeing
if we will have to convert them into the dummy beads.
So the first thing which we can do is we can have a look at the value columns.
So if you see the Idee column has one as the value count, that means that these all are unique values.
So we will be getting rid of this one.
Similarly, the yuan is again unique so we can get rid of them.
Next comes the region.
So there are a total of four hundred and three regions.
So we will have to find out a few regions which we need to keep and get rid of all others.
So if you see the top five regions can be a high number of values, that is around four thousand, three
thousand and the ones below the earth towards the bottom, they can be around thirty eight or thirty
seven guys and their listings.
So we will be selecting only the top of him.
Next comes the region guaran so again, we will be getting rid of these region bureaus because we cannot
delay much from this next if we look at the price.
So you can see that for zero as a price, we have around forty four thousand barrels of data, which
cannot really be possible.
And towards the bottom, again, a lot of cars have a very low price.
So there are certain prices which have only one values available.
But again, these are not of our problem because these are integer type.
So the only thing which we need to worry about here is this particular dataset where we have zero as
the value of the price.
So that is something which is not possible.
So this is something which we need to be careful.
So why are you taking care of this?
What you can do is you can be a mean value or a median value of that particular region where it belongs.
Or you can drive on other different methods or find different patterns between the regions or the type
of cars and then find out the median or mean value accordingly and then put that here.
So that is completely up to you, how you want to use these values.
So because this is a large number you can automatically take on me.
So I would suggest you to take a mean of different regions and then infuse those values in these blank
values.
Next, we have here.
So these are the years when these cars were manufactured.
So here you can see that there are a few cars which are very old.
And most of and here we have a lot of cars which are buried under recent years.
So what you can do is for the cars, which have very old data, you can basically create a separate
column or maybe see older than 50 years or older than 40 years and create a single category out of it
instead of having.
These combined.
Similarly, here we have some of the fringe manufacturers of these cars, so you can see that there
are a few manufacturers which have a lot of cars listed here, while some manufacturers have a very
few number of cars listed like the Abdeslam, Harley Davidson, Alfa Romeo.
So what you can do is you can put these into other categories.
Only of these categories don't really make a huge impact on the prices.
But you can see that this actually makes an impact.
Like if we have Ferrari, Aston Martin, these cars are actually having a very high price.
So this will actually have a huge impact on the prices.
So we cannot really converted directly without having a good analysis on this.
So we will find out and divide these into different categories.
Or you can divide these manufacturers into different levels which are associated with the price.
So this is another thing that you can do.
Next, we have the models now, these models are actually very unique things, and these are very specific
needs for a particular God.
I'm here if you see here we have this Silverado fifteen hundred and this name has been split inside.
And I have another six thousand rows of the David fifteen hundred and another 5000 bottles of the Dovid
Silverado.
So which means that these are kind of replicative and it does not really have a good information present
here because the same guy has been listed in three different categories inside this.
So what we can do is we can actually decide if we need to keep this model or not or if we need to convert
this into a different type of category or how we need to deal with it.
This is something that we will have to find.
Next are the conditions, so because these are very lesser numbers, we can have all these conditions
and read them and convert them into an ordinary variable, which looks like the best way out, then
we have the number of cylinders which can directly be converted into numeric value.
Then we have FuelCell.
These can be converted into a categorical dummy variable and then we have all the which is statically
an integer type.
So we don't really need to think much about it.
Then comes the title.
So this needs to be converted into dummy variables.
Then we have the transmission again.
This would be converted into dummy variables.
After that, we have a van, which is basically the details about the car, which seem to be a lot unique,
as there are around one like eighty one thousand different values associated with it.
So the best way out would be getting rid of this particular column.
Next, we have the type of drive which can be directly converted into a dummy, the.
Size can again be converted into the type of the car can again be converted into a dummy variable,
then paint color.
We can maybe decide the top four colors and based on the top four colors, we can see if we need to
convert them into three variables.
And we can actually get rid of the view towards the bottom because they are a lot of colors.
We don't want to create a lot of columns out of just colors because that would hardly have any impact
on the pricing.
Next, we have image euro, which we can directly get rid of.
After that, we have this huge text column, which is description.
So there are two ways.
First would be you can create a model by getting rid of this column entirely.
And the second would be you can write the first method and additionally create counterfeiter's or Defitelio
vectors out of this description column and add that as an additional feature to your mom.
Next comes the state, and if we look at the state, there are a lot of values in the state, so we
will have to kind of get rid of this column because it does not have much information in it.
So we will have to find out if there is a specific pattern which is associated regarding priorities
for the state.
If not, then we can radically get rid of this.
Next comes the latitude and longitude, which we can directly get rid of, because that would again
be similar to the state.
So what we are doing here is we are dropping the idea you are in the region you are living in means
you are description, latitude, longitude, country, region, and this is something which I have done.
So I also told you about what all you can do other than what I am actually implementing here so you
can do it accordingly.
But this is just my implementation.
Next, this is the final list of columns that we have, so we have praise your manufacturer, modern
condition, cylinder fuel, odometer titles, distance mission, drive, size type being color and speed.
Now, let's describe this and then we see the.
So we have praise so you can see the minimum price is zero and then we have twenty five percent of the
values which are price around four.
Then we have eer of manufacturing.
After that we have the odometer values.
So first thing that we going be doing is handling outliers.
Now, as you have already seen, that the prices have a lot of outlier present because there are a lot
of old cars which are also present.
So what we will be doing is we will see the find out the outlier of the will for the standard variable
itself.
And the target variable here is price.
Right.
So we will be finding out the outliers that and we will remove the model which so that those particular
values could be accurate because these outliers present in the price itself, that is the target itself,
will bring in a lot of difference in the values of the mean and these deviation which we have in the
values.
So the first thing which we will be doing is removing outliers from the licensing.
So the difference between the 75 percent value and the maximum value is very large.
So we will leave the 10 percent values at the end of the distributions.
So what we are doing is we are finding out on day one what they do and then finding out the percentile
values.
And we will be dropping all the rules of the the list and these two values, which we have just found
out.
So now we will have a look and he ordered me to call you.
So this is a distorted values of the order values.
So here you can see the full value is zero and next comes to the eight thousand, which means that there
is a lot of difference between these values and there are a lot of not on numbers as well.
These are the different values which we have, so there are not a number of values and only one zero
value.
So what we will be doing is we will find out of whatever is having not a number.
So here we have around 80000 values which are having a land value, and we will create the scatterplot
for all the leaders.
So here you can see there is one outlier value and all of the values are actually scattered here near
zero.
So this is the maximum value which we have.
So what we are doing here is we will simply find out the values, we will drop this maximum value and
we will drop the column, the particular rule which has the value.
So we will basically be getting rid of this particular data point.
I'm the first data point which have value zero percent in this.
So now when us pulled us out of it.
So this is how the distribution comes out to me.
So regarding the values which have not.
No.
So for them, there are around above three hundred thousand.
These values can be considered as outliers because most of the values are present here.
These are the most of the values and these are the other values which are present here.
So we are considering the values above three hundred thousand as the outlier values and then we again
create the scatterplot.
So here you can see the distribution again.
So this is for more analysis, which has been done on the order with the value being done the year as
well.
So here we have the value of here and here we have the value of price.
So with the distribution of the price, you can see that there is not much relevance of year and price
in the lead up because it is kind of like of uniformly distributed price and your distribution.
So if you see here, we have the year nineteen forty below, which the prices are actually not really
much and there are a few, only a few datapoint also present.
So this looks like an outlier.
Otherwise all the data looks like uniformly distributed.
So what we will be doing is we will be getting rid of these particular rules of the year of manufacturing
is less than 19 for
next.
What we are doing is we will handle the other volumes which are present here.
So I am simply finding out the percentage of null values in each claim.
So here I'm just finding out the number of null values and I'm checking of which columns are present
then.
And I have simply found out the percentage.
So here we have the column name the Josiah's Conditions Cylinder Bangalor Drive, a manufacturing model
transmission's you legal status, then null values, the number of values which are present, which
is the value and the percentage value.
So here, if you see that this is column has it on 64 percent of values, which are actually not antico
column condition has five percent values, which the lender has 33 percent values, which are Benkler
has twenty four percent values.
The General Andreev has twenty one percent type, has 18 percent values which are not apart from that,
all other columns have very few number of values which are actually not.
So these are the values column for condition, which is again having a huge number of value counters.
So these are the different value columns.
You can see excellent has a very huge number of presents.
Then we have code, then we have like new, then we have and we have new and salvage.
Now, the missing values in the condition can be found using odometer as mileage affects the condition
of the car.
So what we are trying to do here is finding out the mean value of the odometer reading group by their
conditions.
So here we have the excellent Otomi, the good old Boomin, the like New or Boomin Salvacion, all the
mean and they're or doing so.
These are the mean values which we have just found out.
And these are the values of what, like new.
We have eighty seven thousand for excellent.
We have a little higher value for good.
We have then something, then we have and then we have salvage.
So these are the different mean values for the old.
Now using these values, we can actually include the odometer reading, which will have more value in
the condition.
So what we do here is we are simply checking that you so wherever the air is greater than 2090.
So there for the condition we are basically putting as new, then wherever the value is, make new or
less than make new the mean value of the like new.
We are simply putting as the like new that whatever the value is again greater than fair or don't mean,
then we are putting forward that it is greater than all the mean.
And this is basically a condition which you can actually analyze and find out what has been done here.
So based on that, we have simply rated excellent good in salvage.
Then what we are doing here is we are finding out the values again, the percentage values.
So here again, you can see this is 60 for which we have just seen.
So we are dropping all the values which have less than five percent value.
So here we have this manufacturer.
So below this all comes under the five percent.
So we are removing the null value rules from the title status, human transmission model and manufacture.
Next, we are also dropping the.
With more than 30 personal views, but cylinder's can be an important feature, so we are not dropping
the cylinder for you.
OK, so we have simply dog this size clip from this next Izadi Malverde.
So we are just finding the values again.
So cylinder has 33 percent null value.
We have been color drive by the null values now and on the column you don't really have any values.
So we are finding out the paint color and we have simply filling them up using the full method.
Similarly, we are finding out the values for drive and filling it using the Hollywood film method and
same in place to type and.
This leaves us with a cylinder having 11 values and drive, having two null values which we can actually
get rid of because that is a very small number.
Now, let's explore this later a little bit, and now we're importing sci fi and importing stats from
sci fi and they're creating a lot of.
If you see we have a price and we have here here, so you can see price and you don't really have much
association.
These are actually a uniform distribution present here.
So we can find out these different relationships which are present here.
So here we have the meter and all the window which comes out.
Seeing here and here again would not have much relevance, price and pricing and we wouldn't have much
relevance here.
And prices are simply distributed by when we talk about all the meter in price.
So as the price of basically as the odometer value increases, the price is also increasing.
So as the volume, the value is increasing, the prices actually.
Decreasing.
So this is the relationship which we have and these are the details which we are live with now, for
all the objects, we will simply be converting them into dummy columns.
So here you can see these are the different state values, which we have a lot in number and different
values associated as well.
So now we are comparing the condition with the price.
So these are different conditions and these are different prices.
So as you can see, for knew the prices high and for fear the price is the least.
So clearly the would condition new has highest price, as one would expect.
And now we will create a categorical plot between the cylinder and the price.
And this is the volume from which we are creating.
So this is the wilin plot.
So this shows the distribution of number of cylinders and the price.
So here you can see this has 12 cylinders, so this is the price, I mean, at 10000.
Then we have eight Lendell, which has the mean at a little greater than 10000.
Next, we are comparing the fuel with price.
So it's here.
You can see if the fuel type is diesel.
The price is a little higher.
When the fuel type is gas, the prices are lower than for hybrid fuel.
Again, the prices are a little lower.
So this shows that the price range would be for a majority of each type of guy is based on the fuel
itself.
So for gas, it is between five to 17 K, for diesel it is from 12 to 20, K for hybrid.
It is from seven to 15.
And for hybrid, it is from seven to 15.
K under it is 11 to 20.
An electric blanket, Waitiki.
So next of you are creating a categorical plot between the Dido's state design, the price, so for
P.M., you can see the price is a little higher.
Well, when we talk about the lean mean that I don't see this as having the highest price meet.
So these are the different distributions for the prices based on the title SIECUS.
Next, we have the distribution among the transmission in place, which clearly shows that the price
for automatic transmission are almost similar.
But when we talk about the other day of the mission, it has a very high price.
Next, we have forward the wars and different types of the drive, so there is not much difference between
both of these.
But the third one, this particular one is a little different.
These are a lot.
See, so what we can see for that is we are distinguishing between the type of vehicle and the price.
So here you can see different type of vehicle, have a lot of different people faces.
The price of truck and pickup is somewhat similar.
When we talk about sedan and the minivan, they are a lot similar.
So these are different type of distribution, which.
So the important observation can be obtained from the above the regarding the price brackets for each
day of the week.
So we can find out the price window for each week from this particular distribution.
And here you see the distribution for colors.
That is not much variation, but still we have a little price distribution, which we can get out of
this based on the quartile values from these.
Then we have a manufacturer which again gives a huge difference between these values.
So we can see manufacturer is actually giving a lot of difference in the prices which we have.
So next, what we are doing is we are creating the models or you can see the dummy variables are prepared.
So we are simply converting the labels into encoded values.
And next, we are simply having a string split of the data.
And we are applying the forest on top of it.
So when we apply the random forest, we are simply finding out the minoxidil then means when error and
group mean square error.
And what we get out of this is the mean absolute error of one six one zero, the mean square error of
around six fifty eight thousand, then root mean square error of two five, stubbled six.
And if you talk about the accuracy, it is actually eighty six percent, which is of decent accuracy
based on the model and the simplicity of the model which we have taken here.
So this is one implementation of this morning, another implementation, what you can do is you can
simply take the description column and convert them into the idea vectors, as it will provide a lot
more information about Neka, although the information might be present in the different columns.
So that is something which you need to be careful of because then it can introduce a little bias in
the data.
So you can try that as.