High in this particular session, we will go over the logistic regression food, so first of all, we
will allow we will import the important libraries.
So the full library, which we will import is the brain faceplate.
And after that, we will import the number and find us as required.
And I have defined the record function as well, which we have used earlier also.
So this report will only generate the report of the greatest TV or Reinoso TV, which we will perform
and retrieve the best models out of it so we can simply put in the CSV, object to it and specify the
number of models which we want to obtain.
So based on the number which we will provide, it will generate the top those number of models.
If you provide five, we will get the top five models, a very expensive five hundred, then it will
retrieve the top hundred models out of it.
So without losing time.
So let us start with the model details.
So in this particular example.
I have taken up a dataset which has two partitions to work, so usually earlier what we were doing is
we had a data set, which was only dataset which we had in that particular dataset.
We have the data for training on testing also.
So what we used to do was we will have that particular data.
So we will split the data into two parts that this one training data set and another testing dataset.
Now, from the training data set, we will train the model and from the testing dataset, we then evaluate
the model.
Now, in case you don't really have that one data set, but you have two data sets explicitly provided.
So in this case, what we will have to do is because then we had only one data set, it was more convenient
for us.
We will simply do our data consummation.
We can simply perform entire data analysis on that single desk.
And once we were done doing the data transformation, then we will split the data now because we were
splitting the data from one particular data frame only.
We did not get any issues regarding the data consistency because the number of columns created will
be the same in both the dataset.
The transformations which we would have done, the dummy columns which we would have created always
will be same in that particular data.
But now, because we have two different datasets, there will be a lack of consistency.
Now, in this case, whatever changes we make to the false data frame, the same changes we will have
to make on the second detail frame that is in the training data set in the testing dataset, the changes
will have to be seen because if the changes are not seen, then let's say I have been my model on the
training data on which I have performed all the transformations, and now I want to test my model on
the testing data, see if they are not in sync with each other in the training and the testing dataset.
I'm not in sync with each other.
Then my model will not be able to understand which column are we discussing about if there is any more
number of columns or less number of columns in case of testing data, then it will ask for you that
why is there a column missing?
I have been trained on a particular dataset which have been Gollum's, and now you are giving me another
dataset which has of 17 or 20 columns.
So how am I able to understand this particular dataset?
And then it's been throwing it on through, so to solve this problem.
We will see how we can do that.
So for this particular situation, what we will do is we will combine both the training and testing
data set, do our transformations, do what analysis?
We will perform our detailed mission on the combine data set.
And then once we are done with the entire data formation and we are satisfied with the transformation,
we will split the data, set the back to the training data set and the testing dataset.
Now, once we have made the transformations and split the data frames back, now the data frames which
we have will be in sync with each other, will have similar type of columns in the dataset, and now
when the model will be trained on the training data set.
And will be actually evaluated on the test, does it?
It will understand what the columns mean.
Because now there is no inconsistency between the problems of the desert, so this will give you a picture
of how the whole nation needs to be done in case of any production data or when you are walking into
a life situation.
Because in case of a real life situation, you will have to make their transformations and the convergence
is such that both had in sync.
In the example questions or in different tutorials, if you use a simple dataset and then split it and
then evaluate the model, it will be very simple in nature.
But this is exactly what you will have going on in your life.
So you need to understand this very clearly.
So without wasting any time, let me go ahead and explain this to you, so we have taken the data,
set the first date as it is our dream dataset and the second data set is the best data set.
This R.G. stands for revenue growth.
Now, we will have a look at the data in some time.
So please be patient and let us read the files.
So we will be updating the first five by using please don't read CSFI and we don't read CSFI gives me
the data from Vitrine.
Similarly, I'll read those ESV for the test file and create a data frame.
B the underscore test.
Now, I will create a new policy.
Now, why am I read this column, so let us see the data first.
OK, so let us go to the head of the board.
Now, what do we have in the vitrine data set?
We have these columns reference no children, age, band status, occupation, occupation, partner,
home status, family income, self-employed, self-employed, partment, and a lot more columns.
And finally, we have the region and the revenue great value is to.
OK, this is the revenue, great value, and we don't have this data column yet.
This has not been ordered till next in the testing data.
We have all the problems, but the revenue grid is not would not be presented.
So what we will do is because in the training data said, we will have all the volumes, but in the
best data set, we will have the Fargood column missing because it is expecting us to find out the testing
values.
So the target will be missing here.
So what we do is we create a new grid column in the testing dataset and fill it with the null values,
we fill it with not a number of values.
And in the B Train the Avenue column, which is data in which we fill three and in every test again,
we create another column, name data and we fill test and.
After that, we will.
Get the column names from the train and put those columns which represent then maybe bring in the very
best, what it does is it will get all the column names from the train and it will fill that.
I would be best for all of these columns so that we don't have any extra columns in the B B bringing
this dataset.
Now, after that, we are concatenating the train and we need this data set by X is zero, which means
we will have all the train data set data on the top, followed by the test data.
Now let us see what it looks like.
And we have put this into video.
So what do we have in the train?
And we did test.
So in the big three, we have the entire data, which was present in between earlier and additionally,
we have created our data column, which has the label print written on it.
Apart from that, in the best, we have all the same columns which were already present.
We have created one revenue column, which has not a number of algebra's in it and our data column,
which has values best present in it.
Why have we done this?
Because we have combined the columns.
We have combined the did the data for baby brain and body test and we need to have an identifier on
which data belongs to b B train and which data belongs to.
We need this.
So whenever we will check the column data we will check for the value.
So if the value print is present in the data, that means that the data belongs to be debri.
And if the value test is present in the column data, it will mean that the data belongs to the best
data from.
And the value for revenue growth will be null in the big test by the values of revenue.
Great will have values to work on in the brain.
Now, let us have a look at the combined dataset.
So we be underscored all dark and.
So this is the head data, it contains all the columns and it has the same data set, if you will have
a look at the daily.
You can see that it has the.
Databased.
The best data, basically, the values, the rules for the train have been present at the top and below
that the data for the brain test have been pasted.
Now, what we will be checking is now you can see here we we are checking the revenue, great values.
So for all the values, for revenue, great, we are checking the value count.
So from the value count, we get to know that there are seven thousand and sixty one rolls which have
the value to Winnik and each sixty three values which have value one in.
Now we will find out the ratio so you can see that the class, too, has actually.
Eight point four one finds values in comparison to the last one.
So this means that it is up in balance, plus it is an unbalanced class.
So when it is an unbalanced class, then we will have to provide certain weights to these classes so
that these classes become balanced and we are able to get a better result out of it, because if we
do not balance out the classes, then what will happen is that the model will train and it will try
to predict everything as as to and because almost eight point four times values are presented as two.
So what will happen is you will get an eighty four percent accuracy, even if all the values which are
predicted are predicted as to save the model starts predicting everything as to then also you will get
eighty four percent accuracy from this one.
So we don't want that to happen.
That is the reason why we will balance out these classes.
So let us go further.
Now let's check about the data now, the learnings that you have done, then all will come into the
picture.
So you will be doing several things.
Now, what I will be doing would be some short good methods, but you via you will be implementing this
code of data of your own.
You will be implementing it with all the data analysis, with all the numbers, you can apply the most
profiling.
On top of it, you will find no different correlation values.
You will find out the columns which need to be removed based on the correlation matrix or VAW matrix.
So you will be applying all those things which we have known until now.
And while you are applying logistic regression, so we will get to know about how we can decide upon
the future importance, also like we have learned about the efficient.
In the case of linear regression, so similarly, the politicians can be obtained in case of logistic
regression.
So based on the confessions, you can actually decide which column you need to keep and which columns
you need to remove.
So we will be applying all the learnings here.
So whatever I am applying is a different thing, but you need to apply all the learning so that you
get a better picture.
And while these sessions, I will be using several different data set.
And I could have used the same data set for all the algorithms, but I decided to pick out different
data sets and applied different the methods for implementations so that you would get to know what all
you can do with this lone.
So try to implement all the code of your own with all the learnings which I have given you, with all
the things which I have taught you, so that you will actually learn a lot from this, and please know
that you will learn a lot if you will practice this thing until and unless you type in the code and
do everything on your own, it will be a little difficult to get ahold of all of this.
And it will be really fun once you start implementing these, because there is a lot of things to run
and a lot of things to learn from this and as you will get towards the end of the code.
You will actually see the accuracy's will actually see how well the models are performing, and then
you will want to actually improve the models and that is the fun of machine learning and the signs that
you will actually feel the fun of improving the models.
It does not even think that much effort.
All the models are already implemented.
All you need to do is just implement them, just type in the code and just select the model, they select
the type of scoring method you want to use and just implement those.
Right.
So it is really fun.
Just try it out, just go through it and you will really enjoy it.
So without losing all the time and without playing with everything, let us start with the code.
So here I am zipping all the column names with the data types and the number of unique values.
Now, what does this number of unique values give me?
It will give me the number of values which are present in each column.
So in case of a reference number of just as a guideline.
The reference, no application, no rule number of a person SSN, no, although these numbers, which
are basically the identifiers, basically something which prove the identity of a person.
Will not really impact how the model is walking.
If I change the reference number or let's say I change your loan application, I would literally impact
if the loan approval or not.
Not really rate the loan approval will depend on your salary, your number of dependents, how much
income you have.
Right.
Only the application number would not be back if the application number was some other number.
Still, your application would have been approved or rejected.
So we can completely get rid of this particular column because it has no significance in machine learning.
Next, we have these children called.
Now, the children column has the type of.
Now, this children column has dated, which rings a bell to me because children.
Has to be on numeric value.
Now let us get back to it.
So here, once I get to it, I see that all of the values are unlimited, but only this zero value is
causing this column to be of theta.
So what I will do is I will convert and replace this string zero with the numerical and can word this
entire column do numeric right.
Next is easy to find.
Now let us see that each band has a difference of five minute.
All the values are having a difference of five.
So what I can do is I can simply say, I mean each and create a new column which will have the mean
value of forty one and forty five.
Forty five and fifty.
So here I will have forty two point five.
Here I have fifty forty seven point five and so on.
They instead of having this age band, which is not mathematically appropriate, I can simply have the
mean value replaced.
Next is statis, which is what kind of values does it have?
It has the object and there are five different types of values present.
So again, we will analyze what proportion of values are present and the how we should convert that
into one more than pudding or dummy variable.
So that is something which we will have for now, again, occupation.
So again, there are nine values in occupation.
So we have to decide if we need to keep all the nine values or we can keep we need both five or built
for occupation types, then occupation partner goes the same way if you want to keep all nine or a few
of them.
The same thing applies to homes.
This family is showing us object but should be numeric.
We don't know ways of showing like that.
So you can see that we have several ranges in family.
So, again, what we can do is we can people I mean, family income here, instead of having the entire
range, let us keep Amien family in them.
Now, what is the range for this?
The family income is actually ranging from twenty seven thousand five hundred thirty thousand.
So it is almost two point five five thousand twenty five to two thousand five hundred here.
Here again, two thousand five hundred is there.
Here we have four thousand to eight thousand.
So here the gap is of around five thousand.
So now the gaps are inconsistent.
So because the gap is inconsistent, we cannot take a mean value, so what we will do, we will convert
these family income into family income in our family income tax.
This is one thing which we can do, another thing which we can do is we can keep a family income level
because these are different levels.
These are different levels of family income.
So maybe we can do something like we can keep eight thousand, four thousand to eight thousand as income
level one, then eight thousand do let us say at eleven thousand income level to eleven thousand to
fourteen thousand income level three and so on, so that we can decide what family income level should
be given.
We can convert this into an ordinal variables.
Right, so that is something which we can decide how we need to take care of it.
Or maybe you can create two or three different columns out of it, one being family income law, family
income tax, then another being family income mean.
Then we can give a family income level and then we will be finding out the coefficients using Lassalle.
We can actually determine which type of variable actually held us up.
So that is another thing which we can.
Right, then we have self-employed, it is a binary value, so it will be converted into zero and one
self-employed partner will be converted and go zero and one.
So like this, we can determine how we need to change the values.
Region value again will change into four different regions.
General will convert into binary data value will stay intact because we need this value to split the
data back.
So we need not touch this particular column.
So here we have the data, so here you can see there are different floating columns which are fine.
We only need to take care of the.
Object data Pullum so we can determine how the transformation needs to be done.
So this is the be all and for all regions we can see these other different value columns for the regions.
So from these region values, you can determine how many region types you want to consider.
Now, what we can do is reference number and post code and we'll still to see what this post code enforcement
area.
Postcode and coastal area has some values.
Post code.
Here we have postcode and coastal area, which are object, they have a lot of cardinality, it is has
it having a very high cardinality with respect to the object today.
So this means that it has a lot of irrelevant data.
We cannot convert ten thousand values into ten thousand dummy values.
Right.
We cannot create them thousand dummy columns so we can simply get rid of this particular column.
And the same thing applies to the pool.
Stadio.
So next thing is children.
So for children, we will convert zero two zero four plus to four and then convert this entire column
to numeric.
For each bank, we will convert this into dummies, for starters, occupation, occupation, partner,
home estate as family income, we will convert this into dummies.
Self-employed, Devora, Origin, gender, all will be converted into dams and a new breed of one will
be converted to one for Mazzitelli.
So the same thing we are doing here, so we are using prop function to drop these particular columns,
which we provided in a list for Axis one means we want to work with the respective columns.
So we are moving these columns and in place means that we want to remove these columns from the oil
completely.
Then be all children, so we are checking and not where, so wherever the children value is zero, we
are putting zero.
Otherwise we are putting the additional value of video on children wherever the value is one.
For all the values, the last value, we are changing forward to four other ways, we are keeping the
values children again for all children.
We are converting it to numeric and delineator scores so that if there is any missing value or null
value, it will be converted into a null value.
Next, we are having them and you get great value for them and Newbridge value equal to one.
We are converting it into the digital form.
So for value one, it will convert to one for value.
Two, it will give false and convert it into Siedel.
Next, we will be creating categorical variables.
We will get the categorical variables using select data types.
Giving needed life to be object, so this will give all the object data types to the categorical variables.
Then.
For the categorical variables, except for minus one, why are we giving it minus one?
Because we don't want to transform this data.
So this minus one will just consider all the categories from stopping the region and ignore this data
and then it will convert the dummy's.
All for all of these columns, so we are converting these age one status, occupation, occupation partner
who start this family income, self-employed, self-employed, partner to the area, gender and region
in two different columns.
And after conversion, we have ninety six columns.
And now we are checking if there is any null value, so in the children column, using DOT is not a
dot some we see that in the children column.
There are might be null values otherwise all there are no other null values.
So what we will do is wherever the value is null, so be all location, be all children.
It is checking wherever the children value is not.
The children column, so the Selecting the Children column from this data, wherever the children value
is.
And in that column, it is putting the.
A mean off the body already that train children, so wherever the data is, brain data type is strain.
So from that training data, it is picking the children column I'm thinking of.
So it is not considering the testing data for the mean of the values.
So it does not mean from all the entire data set, but only from the training dataset.
Because the testing data say it might have different outliers and different ideas, so we don't want
to get into that, we will simply take the.
Nine of the training that only.
Now, next ones, we have done all the transformation so they'll know what the transformation has been
done, now, what we will be doing is from in the train, we will be all all the columns, there be
all data column has the value print.
This will select all the rules which have the value print in the data column and put it into the print
dataset and it will delete the B train data column.
So it has created the train data frame and remove the data column from the data.
Now from the testing data frame, it will remove the revenue column also and data column also once it
has copied the.
Data from all wherever the data column values test.
So we have simply selected all the data wherever the value is drawn and quartered in the we.
Now, one more thing is, in case you are not able to understand any line of food, you can simply try
to run it separately.
Just try to run it piece by piece, try to run it piece by piece.
That would be the ordering data is equal to plain gives.
Now, what will be all three in you?
So then you will get to know that in the brain I'm creating this data frame, that I'm putting all these
values.
So in case you get stuck and you will destroy this method and do it step by step and then any piece
of code food, when you get stuck, just run the thing or try to create the logic step by step from
the very small piece of junk and then put the entire picture.
So that is the best practice which we have invited on any programming language, because initially you
would not know what you want to create.
Initially you would only know of the entire thing or the entirety of it.
But the logic building grows from the very minimal things.
So first of all, you will get what e then you will get B and once you have both and B, then only you
will add.
And so that is the practice which you will follow.
Now we have all this data now be the best and we have been generated.
So the next thing which we will be doing is we will import the largest regression model from the lineaments.
Now we will get the score from the Escalon documentary, and these are the barometers which we will
be having the fullest barometer last week.
Next is penalty, which means that Elvan is what is Ferlazzo.
So we are selecting Brembo to the original penalties and the best one will come out on its own.
So never think that always will give a better result than average or Lazreg will perform better than
Ribeau the scenarios and then see what actually performs with.
Then see the next alpha value, so from in the space, we will get divided from zero point zero one
two thousand.
We are getting 10 those then the such values and we are creating logistic regression model of the object
of logistic regression model and we are giving food in the setting will do to now.
Food in the septic will do through means that we want to get the beat done, not value.
Also we will be getting the old fashioned values.
Ba dum ba dum.
And we, that will be the three intensive means that we want to get to be done by mutual.
So for this, we will apply the great TV.
Now we will learn how to apply a random saltville until that is a different implementation.
So for now, we are working on digital TV.
In the next session, maybe you can learn about the dinosaur TV and then see the implementation.
Here we are applying to TV, so we give the name.
This is the modern name.
Then they give the barometer grade, which means that all of these bad images which we have, they give
the see the value that because we want to have five folds in the cross validation and they give the
sporting images to see if you see.
In the last week, because we already know that the glass is imbalanced, that is eight point one for
value ratio.
So in that case, we can also give all another value.
That is another set of values that it can take value from this to this also.
So the class can be given in the form of a dictionary so we can give another type as violence, none.
And the second one would be class level.
So we can simply say the label is one for one it.
One and four zettl.
The wait is eight point for one, so this is something that you can actually verify what the lead should
be and then accordingly give it.
So here, what do we have?
Here, the class zettl has a twin for one week and class one has a twin.
One as the week, so that is.
So we can debate accordingly, and after that, we will run Vetrini.
So here in extreme, we are dropping the revenue grid column from the B train dataset.
And in the very thing that is the very value, the way the Dufrene, which will have only the volume
that is the target volume.
So X will contain the features or attributes, or we can say X values and will contain all the values.
That is one single column which will have the values of the revenue.
Now.
We will do the grid search, which means that we are training the model on these X and Y values, and
after the footing is done, you will get something like this.
The entire design of the grid search from the grid search, you can use grid search, dot best estimate
to obtain the best model which has been generated, and you can get the detail into this new model named
which is global.
It will see if the details of the best model which we have created.
Now we can create the report using grid to TV design and we are giving the number of results we want.
So we want five results, so it will give us the five best ranking models.
So from here you can see that the main validation score or the accuracy that we are looking for, a
U.S. score is coming out with the zero point nine five four.
And here the sea value is zero point zero one class, which is balanced and the value is a one.
Now, next thing is, we have got the best estimate in the longer rate, this longer contains the details
of the best estimate that these values, that which model has actually given us the best result.
Now we will fit the model again, because now we have the research object, which contains a list of
thousands of models which we have think we don't want all those models.
We only want that one best model.
So we will take the best model here, the attributes of the best model here and set it on the training
the desert.
Now, after training, we will generate a lot of fun.
Now, how can we create value so for generating the value we have this particular logic which we have.
So this is we are focused upon creating a little space of zero point zero one zero point nine nine and
the engine revving ninety nine such values.
So this will give us values from zero point zero one two zero point nine.
Now, out of all of these values, what we are doing is we are making predictions using the model which
we have dreamed Logan has the model, which we have been great and predictable, but all the is the
function which gives the probability as a prediction, if you will have only predict, if you have only
predict, it will give the exact high class value zero or one.
But if you want to find out the probability as a prediction, then you will use predict probable.
So what will predict probable.
It will be the values of the existing values or the evaluation values.
Whatever they decide.
You have the X values you have for which you already have values in place, you will use those to actually
create all these things.
So I am using extranets and from it I am getting the predictions in school.
So this is the predictions which I am getting from the Logan model, which I have generated.
And Vytorin is the data which contain the actual value values.
Right from these actual values.
I create a new data frame which has the values, the actual values, which is named Azriel.
Now I am checking for brain for that is my training school value.
Whatever braining school value is greater than zero point two.
I am checking this particular condition.
So what will happen?
This has the probability values so probability values can range from zero to one.
So the probability value could have been zero point one or zero point towards zero point nine.
So whatever the probability value was, zero point two or zero point one nine.
For that, it has given proof and wherever the value was greater than zero point two, it has given
false.
So if whatever value was greater than zero point two, it has given to this is a big mistake.
So whatever the value is greater than zero point two, it has given through there.
And whenever the value is less than zero point two or equal to zero point two, it has given false.
Right.
So it will have maximum values as through, because the value which we have provided is zero point to.
Named in the title, value will be low in the title, value will be lower than this condition will result
in maximum values rate and indicative value will be higher than the desired maximum for those values.
Right now, this is a method which is this method which actually gives us or allows us to find out the
best value.
If we talk about the logistic regression or any other method, any other scale model, it will by default
assume the value to be zero point five.
But if you still want to decide upon a better value, you can use this for development.
So what what do we have here?
We have this Gesell.
This is a blank list, which we have, and we are running it for different values.
What are these values?
D.R.C., two point zero one zero point nine nine.
So for each of you, it will come back to the training school, which we have obtained, the predictions
which we have obtained and see it and the predicted value, so we will get either zero or one based
on these training school compared to the cutoff values which we have.
So the cutoff value will be zero point zero when most of the values will be to when the value will be
zero point nine nine.
Most of the values will be false.
Now, think about that.
We have these two positive through negative, false positive, false negative.
This is the implementation of the finding out of how many troops was there, how many negative are there
and how many which values of it.
So it is as simple as finding out the predicted value equal to true and real value equal to one.
So if both of these values are equal to one, then it gives up.
So it does add all these values then based on these conditions, it is just adding all the values I'm
getting you all these positive negatives, false positives, false negatives, the gives, the all the
positive values, which is equal to positive plus false negative.
You can force this and try to get to this slowly also.
So basically, whatever proof positive is present and whatever false negatives present, false negative
means that you have predicted it to be negative, but it was actually positive.
That is why it is a false prediction.
Right.
So it is a positive value.
Similarly, we have all the negative values.
Now, this is the.
Formula for Geass calculation, which is too positive, divided by the positive values minus false positive,
divided by the negative values, and it is just simply upending all the key values to this guest list
so that you have all the key values for all the kind of values.
And based on these gifts values, you can decide based on the maximum.
So it is checking the value of incase all is equal to maximum Gesell and getting the index of.
So the index gives me the value as zero point five, so the cutoff, which I will be considering is
zero point five.
Right now, Lowgar Dot Intercept gives me the value of the intercept, which is Zettl, I am zipping
the columns with the coefficient values, so it gives me all the coefficient values.
And you can see that there are several propositions which have got the value zettl.
So this means that these columns can actually be removed from the training of my future models.
So when I will be training my decision, three random forest or exergy most or any other more than I
can very easily get rid of these columns because these modules have no impact in the calculation of
my target class.
So I can easily get rid of these claims.
Now, let us get four billion and we can simply submit these values or let us say you want to participate
in any competition or do you want to create the output of the desire to make predictions.
So what you can do is simply use the word or predict Prova.
I give the input X values that is X data free and get the output values from it.
This is the Y values which have been predicted and you can simply create a new data frame of this,
values these values and put it into a CSFI using the ASV.
And this is the C ASV name which you have used.
Now in case you want to provide the high classes.
I'm not really the probability then you can simply compare it with the cost of value and put it as type
and so it will convert it into zero and one and then you can create a file out of it.
So in case you will be using a manual method for this particular model of introduction, moving this
moving this model into production.
So in that case, you can simply export the CSV files to get the output of the model which you have
created.
I'm getting the the output, all the predictions from the models.
So I hope you understood this in case you have not understood any of the things which I have told you,
please go through the video again and again, because this is a very important concept and I hope that
I will try to clarify these things again and again over the time in my videos for the.
So that is that is something that you would not have understood.
You can learn from those videos.
And in case you don't still don't understand, then you can get back to me and I can explain those again
to.
So I hope this would have helped you in the next session.
We will talk about different types of metrics which we have.
And so without using those metrics, you will get to know what metrics you can use and where you should
use what.
Thank you.