In this session, we will learn how to derive some relief from the numerical data.
So the first step would be seen involving the bonders and library.
Often, including the library, I would involve the fire.
So this is the fire which we will be using.
And this is the Ffion import, so I'm reading the file from the read CSFI function and this is the file
path and my file has a delimiters semicolon.
So I'm using the LAMIDO in this.
I will not use this delimiters.
So let me show what happens.
So if I had run this.
And.
Get the big head.
So here you can see the detail comes in a single column, but if I will, I'm in the middle.
Then it will be shown in different columns, it will be read properly, so please make sure that the
delimiter which you are selecting is correct.
Now, the next thing which we can do is we can describe the details of the numerical values.
So these are the numeric columns in my data.
So I can find out the ground, the mean value, the standard deviation, the minimum value, twenty
five percentile, 50 percentile, 75 percentile and the maximum value of my numerical columns.
The next thing which I can do is I can do it in full.
So this will give me the information about my Dufrene.
So it does me that my data frame has forty five thousand one hundred eleven entries ranging from zero
to five forty five thousand two hundred.
And then it tells me that the columns are both 17 and in this data columns, these are my column names.
These are the normal values, which means I do not have any null values in my data.
And it also tells me that the data of the columns.
So by this I can find out which values I need to modify.
So here you can see that the monthly column has value as object.
So let us see what my monthly column has.
So month's column has values as me, June, July like this.
So what we can do is we can create another mapping.
So instead of January we can put one.
In February, we can do so.
Similarly, we can change the values of the month to get the values in a numerical.
And other volumes are like Joel Madden, education, the federal housing loan.
And the outcome and why these are the values which we can then vote into now regarding the contract,
we will actually have, we'll see what kind of value it holds.
So if you see here, the contract holds object.
So we will have to find out what kind of value it holds and what number of values are having value as
unknown.
And then we will have to make our decision if we need to keep going back, forelimb or not.
And in case we aren't keeping the contract volume, then if there is any particular value which we can
impute or not.
So these are certain decisions which you will have to make.
Now, next, we have video which will allow us to get the name of the volume and we have a function,
Nuni, which tells us that what are the number of unique volumes for each and every column?
So for each volume, I have seventy seven unique numbers, but just fine.
And the job is a categorical volume.
I have 12 unique job types, so when I have 12 unique job types, it means that I have a wealth gap
that needs to be created.
Now, when I will be converting these 12 job tapes into categories, this will mean that I will be increasing
at least 11 volumes to this entire dataset.
Now, this is something which I will have before they decide if they are for a certain number of job
types which are important and sorting job things, which I can actually remove from my dataset.
So that is something which I will have to make a decision upon.
Now, again, we have this material value, which is fine.
We can create three dummy values for this.
But education, again, we can make for dummy values for default.
Again, the value can be created.
Housing loan gone back for all of these values can be created.
Then we have.
The day's campaign duration, so all of these values.
I'm numerical in nature, so this is fine, we can handle these accordingly in case we had any particular
volume which had a lot of categories.
So in that case, we would have had done what they're doing scattergories into certain levels or certain
groups, or maybe we would have selected only five or seven top groups so that we could reduce the number
of volumes.
But again, how we will do the feature selection is another topic, which we will discuss further.
But this is just to give you an insight into how we will be tackling the beat them.
Now we have visual display so we can use the display directly and I can use to describe also this would
just give me the details about my age column.
Now, what we can do is begin again.
Zibakalam means the day that I am the number of unique values together.
This will actually give me a broad insight if I want to convert a particular value from categorical
to numerical.
And I have one number of categories.
So here you can see that I have 12 categories, three categories.
We've got the two categories.
We are checking only for the object types.
For the details or are the objectives which we have to go and work for in the 60s for VIPs, we don't
have to do anything specific because they are already guilfoile.
Yet again, it is, too.
Three.
To El.
For.
And so this is perfectly fine, we can walk ahead with this now here we can use certain aggregate methods
to create values or to impute values.
So here we have the age Takamine and the age of median, which I can find out.
Using dot, dot, median function, so similarly, we have many aggregate functions available.
So these are the names of several aggregate functions.
They found some mean, mean, absolute deviation arithmetic, median minimum, maximum mode, absolute
value products, the deviation so we can use all of these values, any of the values, whichever it
is required.
Now, if you want to find out a particular number of value count.
So here we have the count which belong to each and every category in one particular feature or input
value or volume.
So this is a limited time job.
So in the job column, or we can also pilot feature or input value or attribute.
So in these we have these 12 values vagi blue collar management technician, admin services, retired,
self-employed, entrepreneur, unemployed, housemaid's student.
And now if see majority of the values are from the upper limit.
So we have a values, a few values which have more number of data present with them.
And there are certain categories which do not have much detail on them.
So what we can do is we can photobook now here.
We don't really get a view of how many values, how much percentage of data is actually presented by
the blue collar people or how many, how many, how much percentage of the data presented by students.
So what we can do is we can get the value count and along with it we can normalize the data.
So when we normalize the data, what happens is we get a percentage view of this.
So you can see that 21 percent of the danger presented by BP, 20 percent of data is represented by
management, BP, 16 percent, this technician, 11 percent is admin, nine percent is services, then
five percent is retired.
And then three percent, three percent.
These are self-employed, entrepreneur, unemployed.
So what we can do is whatever is less than, say, five percent or three percent or two percent, that
is your choice.
What is the value you want to apply by reducing the number of features so we can decide on top of that?
So let us say I want to keep only data which represents at least five percent of my population.
So what I can do is I can keep the retired services, I mean, technician management and blue collar
as my dummy columns.
And I can remove self-employed entrepreneur or unemployed housemate's student and unknown from minding
them so that the data which I have would have a good representation on you, because that is something
which is actually going to help out.
While I am actually trying to train my mind in some detail, which is actually less in number, will
not be able to give that much weightage to the presentation.
Now again, that is completely your choice and is something which is derived from the type of problem
that we have, a legacy that we have something very special for students.
We are talking about loans and we have a special category of student loans.
So in that case, someone who is a student becomes a very important person in our in the problem solving,
which we are doing.
So in that case, we will keep this student no matter how small the percentages.
So that is something that you will have to keep in mind and something which you will be having as a
good idea by deciding if you should be removing a particular column or if you should be removing that
particular category wise and wanting the job column in two different categories.
Or in the dumbness, so similar thing would be applied to different type of features so far, different
features like we have job, marital status, education, housing loans and so forth, these also same
thing would be applied.
So let's see, I have this column contact and.
Ninety nine percent of my data is be presented by ACORN that I see value X, then what I can do is I
can completely get rid of that one percent of my data because it doesn't really have very good representation.
And all the values are actually constant in nature.
Or like I say, there are two categories.
Yes, no.
And maybe and there is only one percent of people who are opting for.
Maybe then we can get rid of that maybe and don't keep it as a good idea.
It's a.
So we can do something like that.
Now, the next thing which we have is.
We can check if there is any value in our data.
So when I can simply say the need is not which, we'll find out if there is any null value.
And then I think the values of the number of null values.
So we get that there is no null value in this particular dataset.
So here what I'm doing is I'm adding certain null values in that each column.
And I'm adding certain land values in the violence problem and creating a new Dufrene.
So this is my new day offering, which has null values in violence, volume and age volume.
So now what I'm doing is I'm checking if certain volume is not.
Then I can find out the value found in violence.
So here I am finding out the value value found in violin study.
You can see that it is showing for forty thousand, fifty thousand, thirty thousand.
All these values are coming up, but it is not actually showing any value for not a number that we actually
have.
This not a number here, but the value found is not coming from.
So what we can do here is there is one option, which is to include drop in.
OK, so if we run this, we have this mighty violence, which is just the same thing, and we are getting
out the value vote along with the value guns we can select to have normalized so that we are able to
see a in this view of data and.
When we say that Albany will default, which is by default, too, so what will happen is when we say
Albany will default, then it will not remove the north no category.
So here it was dropping as to.
So that's the reason why it was not showing the null values here.
But now here we are keeping an equal default.
So it is showing the north the number and the percentage of.
So now here you can see that there are actually 10 percent of data, which is not a number.
So we will have to imbue these values and handle the according.
So in this, you will have an equal to do, then you will not forget that your picture of what data
you have.
So I always make sure that every penny is equal to is equal false, actually, when we are checking
if a particular value has to be dropped or not or if there is any particular value or not.
So these things can be tracked by having this drop.
Any defaults?
I always vote to eat in every column one by one, so that you have a better picture of what kind of
data you have, data transformation and feature selection.
And working with the data is a very long process.
In a normal project, it takes a lot of time, almost 60, 70 percent of the time goes into the detail
preparation.
And after that, almost 10, 20, 30 percent of time goes into the model.
So it is always recommended to put in a good effort with the data preparation and feature selection
and feature preparation so that the model that you will be training would be trained with.
Because it is always said that garbage in, garbage out, which means that if in your model you do not
have a good quality of data, then no matter how good your model is, stream, the output will not be
satisfactory.
So always make sure that the deed is done vainest.
So here we are.
Seeing this health on the value counts function so you can see what functions we have here.
And how we can move on, so if you want to look at any other function, you can simply apply his wonderful
for guidance and show you the entire documentation of the function.
Then we have volume, so let us check the volumes at the end of the day, doesn't.
I'm here, we have medical, so we have.
Twenty seven thousand values as measured, 12 percent as single and five.
As damaged now.
By visualizing the data, we are just visualizing these data as a single than right now, but what we
can do is we can also compare data based on multiple columns and analyze multiple columns simultaneous.
So how we can do is we can do that by creating a that.
So one meeting with your staff, what we can do is we can use feed or crosstab and we can put the column
names into it.
So I am creating a stand between default and housing, so this actually gives me the number of values
which I have.
So for default then the value is no, I'm for housing when the value is no.
There are nineteen thousand seven hundred of people.
When the fall is yes and housing is no, there are three hundred and eighty rows of people.
Then the fall in housing water.
Yes, there are four hundred and thirty five volumes of data.
So this gives a more clearer picture than actually there are more number of people who have the as no.
And there is a list number of people who have voted as yes.
And if we see somebody give me the people who have housing have more number of default in comparison
to the people who don't have housing.
Now, here we are creating another froster.
So this gives a more hybrid version of it, it just gives the addition of these two values, so it gives
nineteen thousand seven hundred and one plus twenty four thousand here.
And these two values are added here and these two values added here.
So it gives a complete value count to.
So we are just getting the body count and here select the day, it allows us to get a particular DNA
from the entire database.
So here I am selecting object.
So it will allow me to get all the object I need from here.
So I have doubled my education.
So all these volumes which have a database as objects have been selected.
Now, here, what I'm doing is I'm getting the categories.
Which of these I as objects, so I got all the following names where the data like is object.
And here I'm just pointing out the data and the value found for all the categorical data.
So this will actually allow me to visualize what all these volumes do I need to have what sort of value
should I have for removing the particular columns, for removing a particular activity by converting
them into dummies?
So all these things can be analyzed from this entire plot, which we have got.
Then we have this group by option, which we have seen.
So it allows us to group and view the mean value so we can apply any aggregate function on top of it.
So I'm applying mean here.
Yes, I mean.
Here I am again, applying mean and viewing only a few minutes of viewing all the problems, and here
what we can do is we can apply a different aggregate functions also by.
These are the different aggregate functions.
Next, we will have a look at data visualization.