What do data scientists to?
As we have discussed, data science is the field of finding out solutions to a particular problem by
analyzing the data.
So in this context, what happens is that an organization is looking for solutions for its problems.
These solutions have to be fact based and they have to be realtime in nature.
These solutions could be something which is related to, let's say, an organization is facing problem
with its sales or it is facing problems with its marketing.
So the organization will reach out to a data scientist and then data scientist will take out and execute
a particular process.
So the tasks which would be assigned to a data scientist would be very first being the identification
of data analytics problem.
So although the organization has come to you for a problem seeing that there is some issue with the
SEALs, we are not making enough profit.
The marketing is not going right.
We still need to analyze the data to find out if actually the issue is with the sales or the marketing
or the pricing is not separate.
So what is the actual issue?
Is it because of some specific market fluctuation, which is constant for all of the products or all
of the organizations, or is it something which is very specific to this particular organization?
So all these things have to be analyzed and the data scientist will try to find out the actual underlying
problem.
So once the data scientist has the problem in hand, the question which he wants to solve in hand,
then the next task is to determine the correct details and the variables.
So let us see if the problem is with the sales and the sales, prices are not set.
Right.
Then there would be no use of taking the customer segments in consideration.
Or let us see if the problem is in customer segmentation, then the prices or the sales details would
not be much useful.
So the data scientist has to collect all the information which is present and find out which information
is actually relevant to the problem.
So the data scientist collect the data and find out different variables, find out what all data is
available for the solving of the problem.
Once this large data is collected, this data is a matter of raw form, so this data set could be present
in a structured form or an unstructured form.
A structured form would be something like a database or a complete diabolo file format or an Excel file.
While an unstructured data could be Audu with you or some images, all of these would be unstructured
data or some text.
All these would be unstructured data.
So all of these data sets would be collected from different sources and then this data would have to
be cleaned and validated to make sure that the data is accurate, complete and uniform in nature, in
case some data is missing, then the data scientist would try to collect the missing data or would try
to impute values as close to the missing data.
If the data scientist is not able to collect those information, then they would probably drop that
particular piece of information based on how much information they have.
After that, the data scientist were try to find out what more for the best for this particular problem.
So the data is stick to a particular model.
An algorithm is applied on top of this data.
And then the data is analyzed and different patterns and trends are found out of it, once these patterns
and trends are found out, the interpretation of the data to discover solutions and opportunities.
So once we have the patterns in hand, we will try to find out the different solutions, like let us
say that in some particular segment of customers, which is really very interested in the product,
then instead of doing marketing on all other customers, we can target that particular customer segment,
which we are expecting to be highly interested and save on resources which could be used in some of
the.
Then communicating these findings to stakeholders using visualization and other means is the last of
which a data scientist will do.
So all these things would be done by our data scientist from analyzing the problem to finding out the
data and then gleaning the data and validating the data and then applying different models to find out
patterns and trends and then interpreting and analyzing that entire Biden brain, then trying to find
out different solutions and finally communicating these solutions to the stakeholders is what a data
scientist does.
Now, here are a few implementations of machine learning like Netflix and Netflix.
You would have seen the recommendation system where the Netflix suggests different movies or shows which
you might be interested in.
Similarly, Amazon, Google Maps, elixir fraud detection, then see the different object.
Free speech recognition does the self driven cars and then of finding out the best picture and searching
from that based on the picture.
All these things are implementations of machine learning.
Now, what is machine learning exactly?
We have talked about the different implementations we have felt about what is does how he finds out
the problems, how he find the solution.
We have discussed about what is data science, why is the science required now?
Although we have all these things in mind, we still need to answer the question that what is machine
learning?
So although we have the problem in hand, we need to apply certain algorithms which wouldn't be helping
us to find the solution to the problem.
Now these algorithms, one thing could have been done was someone could have written a complete full
fledged code to have considered different cases, like I can have different Ifill's conditions to decide
if someone should prepare the Dinev or not or if someone should order dinner or the dinner.
So this thing would have been written by a simple Ifill's condition saying that, are you feeling hungry?
If yes, then we will check if we have proper ingredients of the food at home or not.
If we have food ingredients, then is the ingredient something which you would like to eat or not?
If this is something that you would like to eat, then are you really feeling like cooking or not?
And if you're feeling like cooking, then you would go for cooking.
Otherwise you'll order.
All right.
So this could be written.
Is that as an Ifill's condition also?
But we don't really want to do that.
We want to provide different scenarios of all the month before or an year before data with all the conditions
and all the values of are you feeling tired, are you feeling hungry?
Like the data would contain different columns with values, like if someone is feeling hungry or not.
Is someone feeling energetic or not to cook or not, then if someone wants to eat outside or not, if
someone has ingredients at home or not?
All these questions and those are the values for all of these would be provided in its ability and based
on these values, what decision that person made at each and every day would be given.
And we would let the machine analyze this entire decision and find out different patterns, that something
like if someone is feeling energetic and the ingredients are fine with the person and delayed that those
ingredients, then they broke, they would go for cooking.
This is one type in which we have found another pattern would be like, let's say someone is not feeling
energetic enough.
So they would probably order the food.
So these are the things.
These are the items which we don't want to give to the machine, but we want a machine to find out these
patterns from the data.
So machine learning is all about providing this data to the machine, I'm expecting the machine to launch
from this data to make its decisions in future.
So we will give it that one month data regarding if someone had the food or ordered online and based
on my data machine, would learn from that data, find out patterns from the data.
I mean, decisions and future.
OK, so machine learning is an application of artificial intelligence.
That provides systems, the ability to automatically loan and improve from experience without being
explicitly programmed, so without being explicitly programmed, without writing an explicit offense,
Lou.
I just want that machine to look at the data and learn different patterns from it and find out if someone
would be cooking the food or ordering only.
Machine learning focuses on the development of computer programs that can access the data and use it
to learn for themselves.
Now, where is machine learning used, the heavily hive self-driving car, the online recommendation
systems like Amazon and Netflix, knowing what the customer is saying about you on Twitter, Lizzi of
sentiment analysis, we can apply sentiment analysis on textual data, which is present in Facebook
or Twitter on the field or any other feedback forum, and find out what exactly our customers are feeling.
We can find out the good reviews.
We can find out about the bad reviews.
From the Bible reviews, you can find out what is the actual problem which the people are facing and
then try to solve them.
Then fraud detection, all of these are machine learning implementations.
Now, the next question is, what are different types of machine learning?
There are three types of machine learning problems.
One is supervised learning.
Second is unsupervised learning and thought is reinforcement learning.
Supervised learning is where we want to predict something.
We want to give certain values to the to the machine.
And we want to predict something for future.
Let us say we can give some weather related data to the machine.
We can provide the temperature, the season, the day of the month, all these details and the temperature
of that particular day.
And we can give, let's say, three past three years data.
And then based on the temperature and all these details, we want to use the machine to learn this data
and predict the temperature for tomorrow.
Or we can give a machine several images of different animals and let it know that this one is OK and
this one is a dog and which one is a little chimpanzee.
And we want the machine to recognize or predict which in case any unseen images provide to it.
And we wanted to predict that this image of a cat and this image is of a dog.
So in case of supervised learning, we want to either predict some continuous value, which is called
regression, like let's say we want to find out temperature, height, weight prices, all of these
things are continuous values.
So if we want to predict continuous values, these are called.
Supervised learning problems, these are called regression problems, we are finding out cantinas values
in regression problems and in case we are trying to find out any classes like different categories of
values.
So those are called.
Classification problems when we are finding out if something is a gag order, OK?
So in case of supervised learning the ropes, we clearly define the output value.
We provide the input value also and the outcome of the Dellec we provided the data of previous years
or previous months then is and what a claim would be used for there, how much humidity was, then?
All those details would be provided.
And along with that, the expected output would also be provided that the temperature that they were
going to be provided so that it can find note, let us see kind of an equation between the directives,
all the things like the humidity and the temperature and climate and all of those things and the relationship
between the temperature and create an equation out of it.
And similarly, create an equation out of the decisions which we were trying to make about the glasses.
So all these things.
Commander supervised learning via unsupervised learning is to simply finding out different patterns
in the data and to categorize something or to basically group something or to segment something.
So in case of unsupervised learning, we don't provide the output values.
We only provide the input values.
We only talk about us.
We are talking about a particular camp.
So we would say that we are having a certain.
Animal, which has whiskers and it has a bee and it has years and it says male.
And we have another animal, which is Bo, and it has long hair and the long mouth, so all these details
we will have, we will give, but we will not build the machine that this one is a gag and this one
is a dog.
We will only provide the related features about these two glasses, but we will not name those and we
will not expect the machine to also recognize these two.
We will only expect it to segregate these into two different groups.
So that is what unsupervised cloning does, is it does not predict any particular value.
It just groups different, classes different and just segregate the items.
Violent is of reinforcement learning, it is a reward based learning, which is basically like, let's
see, I have a particular reward which is trying to learn to walk.
So what I would tell you is that if you're going towards the wall, if you hit something, don't right
or don't lift or go back.
So each time and I would tell you that this is something wrong, which you have done, I would tell
that it is a negative thing.
And if something it is going towards the wall and it makes a left turn, I would say it is a positive
thing.
You do not hit the wall.
It just it is a reward for you.
So we will provide a reward system so that something so that the robot does not hit the wall again.
So this is what reinforcement learning is, which is completely based on the reward system.
Now, here is the more descriptive of the bill of supervised learning, so in supervised learning,
the new classification and regression in case of regression, we are trying to find out, identify and
predict the continent's values like prices or wheat or height of a person Vilem classification.
We are just trying to classify items into different categories, see cats and dogs.
So in case of supervised learning, we will provide data about different classes and we will also provide
the labels that these are CAT.
So that when different damages would be provided to the machine, it will be able to find out that this
is gack and this is not.
Right, in case of unsupervised learning, they do not provide any labels in case of supervised learning,
we provided these labels get, but in case of unsupervised learning, we are not providing any labels.
We are just providing different features about these animals.
And we are expecting the machine to find out the features and group them in two different groups.
So this has two types of problem, which is clustering, so it will segregate these plus these multiplication
signs and so goes into different different clusters, an anomaly detection where it is trying to find
out some abnormal behavior out of the normal.
Now, what does a machine learning engineer do, the task of the machine learning engineer is to explore
the data to find out actionable insight.
They do model the data using different machine learning algorithms to predict the outcome and then to
report the outcome to the Monch.
This is the entire site where the fetch the data, then we clean the data.
After that, we prepared the data and the brain, a modern.
After training the model to evaluate the model, if the model is not good enough, we again bring the
model.
And keep doing that again and again until we are completely satisfied with Darmody and then we deploy
the model to the production and then monitor the.
Output's, which had given from the model.
We will be discussing about this entire process.
We will be discussing about different machine learning algorithms in future.
So it is fine if you don't really understand much because this is just the beginning.
So if you even understand some very basics about what I have told you, that should be enough, because
we will discuss everything in detail in future.
These are different big chains we're investing in machine learning like Google, Amazon, Invidia,
will, Intel, Microsoft, Salesforce, IBM, Facebook.
And this is the format of the traditional models and the machine learning model.
So in his traditional model, we provided the data and we wrote and handcrafted model, which is a complete
defense, complete model.
We will write by hand and then we will feed it to the computer and get the results.
V, in case of machine learning, there are two phases, one is the learning phase where the machine
will learn from the sample data and from the expected result.
And after that, after the cloning, we will create a modern this cloning phase will result in a modern
and then this model is fed with the new data and then which is used to make different predictions.
So first of all, we will feed the input and output data to the model, to the machine, and we will
train on what we will create a new model.
We will try to learn different patterns from this particular detail, which we get and create a model
out of it.
And then we will use this model to make predictions on the unseen data.
This is what machine learning is.
When we talk about machine learning, when we talk about the supervised learning, to be precise, we
have two types of values.
One is the input value, these input values are different X values, different values, and these are
called features, input or independent values.
These are called independent values because these are not related with each other.
Like I say, we are trying to predict if someone will default on loan or not.
So for this particular problem, we will need the information on a person's complete salary, on the
number of dependents, on the person, if the person has any other loan or not.
Then the age of the person, the education of the person and all the details about the person.
These details are independent of each other.
So these values are called ex.
All these values which I just quoted are known as Xe, these are the values from which we want to find
out certain facts.
And the value which we want to find out is if someone would default on a loan.
Yes or no.
So the answer would be either yes or no.
This, yes or no is called wife.
The label that or the target value, all the all the dependent value.
Because this yes or no is dependent on different X values, on the values of the amount of loan that
person has taken, the salary of the person or the age of the person, the number of dependent on the
person.
All these things, these yes or no, would be dependent on all these things which I just listed.
So we are trying to find out a function of these X values which would be equivalent to these very values.
So why is a function of X plus some added value?
This added value would always be present.
It is not completely reducible.
So there are two types of errors which are present.
One is irreducible error, which can be reduced by improving the model quality, by printing the model
properly, by providing good quality of data.
This error could be reduced and there is another error, which is president, which is called irreducible
error, which would always remain.
The traditional modern modeling methods used to have biased roots while the machine learning models
find a gender relationship that reveals the error by finding the unknown.
So the machine learning models will try to find out this gender relationship.
This function of X, such that it will be able to reduce the error to the maximum value, just possibly.
So this is the process again, so we will provide some input data from this input data.
We will find out different features, different columns which are actually important, which will help
us in finding out if someone will be defaulting on the loan or not.
And then we will be building the model.
We will be training the model.
And once we have the model in time, then we will make predictions.
It is always important to have better features, a good amount of good quality of data in hand in comparison
to the better algorithm, because the better the data that you have, the better the machine learning
model would perform.
If the data quality is not good, if the information which we are trying to find the patterns in is
not correct, then no matter how good the machine learning algorithm is, it will not be able to reduce
the errors much.
So always walk on future engineering, always work on finding out the features, which is very, very,
very important.