In this session, we would work on the rescan, an anomaly detection, so let us, first of all, implement
the scan.
So for that, we will implode the celebrity and scale for scanning the data.
Then we will import the canings.
We will import see one as soon as my Lord unobvious and.
Now we will import the data set, which is more data set.
This is a different type of data set, in which case means is usually not able to perform really well.
So let us have a look at this.
So here we have the data set, which I am reading using read CSFI, and it has this particular data
which is having X values and Y values.
So let me plot these X and Y values using C seven so you can see there are two 1/2 present.
So if we are about to think of it visually, then we can see two types of clusters present.
One is the upper half of the cluster.
That is the first half semicircle and the second question is the second semicircle, which we have deported.
So out of these two clusters, let us see how which clustering algorithm is going to perform.
So the first algorithm which we are applying is the element of clustering.
So here we are already providing the number of clusters to it because we already know that there are
clusters and we are generating the plot for this.
So argumentative clustering is not able to find out the cluster properly.
It is not able to distinguish that this piece of the cluster is actually a part of the lower half moon
and not a part of the upper half.
So this doesn't perform then?
Similarly, we will apply gaming's on top of it, so when we apply gaming's and how we will apply these,
the code we have already discussed in the last few sessions, how we implement key means and how we
implement Aggregative Flustering.
So I will not discuss this again.
So it is just the same algorithm implemented.
Only the data set is different.
So you should be able to understand this.
So we will apply the K means clustering, so this is what he means clustering does now because K means
clustering, always tries to create spherically clusters, so it divides these harpoons into two parts
and creates a cluster one at the right left side and one on the right side and is not able to distinguish
between both of these clusters.
Now, each time I will run this again and again, it will give different clusters while in case of clustering.
Even if I have done this a hundred times, it will still give the same cluster to me.
So now let's run the scan, so for implementing the scan, we will need to provide the values that this
epsilon value and the minimum sample size value.
So here we have the.
Library, which we will be importing, so we will import Escalon Cluster and we will import busgang
from it.
Now we will import, we will run the object of the scan in which we will provide the epsilon value now
based on Absolon value, as we have seen in the visualisations.
So, Max, the larger the size of Epsilon, the faster it will create the neighborhood and the larger
the neighborhood would be.
If we will have a small epsilon value than the neighborhood created would be smaller.
No, that isn't a sudden number.
Next, we advance setting the minimum sample is 30.
So we are seeing that create a cluster.
Only if there are at least 30 points present in that, then we are giving it the distance metric to
be played in and then we putting the data to it.
Now we are creating the cluster.
So after running this, we are putting the data that is clean.
So this has already said to the data which we have provided, and we can basically get the labels from
the divi scan object.
So here are the labels in my column.
So now we will plot the data using some alien plot and we will give X and Y values and the hue will
be the cluster, which is the color would be provided based on the clusters.
Now here you can see that it was easily able to classify, basically cluster those two types of data,
those two clusters of data properly.
And we have we have received good clusters.
Now, you can also see that it has created these blue dots and these blue dots are nothing but the points
which have not been included in any of the clusters which you can identify from the cluster.
Whenever the cluster level comes out to be minus one, it means that these points are not part of any
cluster.
So and the next cluster's is zero and one, so we have got two clusters created clusters as orange cluster
one as the green one, and the rest of the points are the ones which are not included in any of the
clusters.
So next, we will find out the value.
So here you can see that in the fullest cluster there are around nine hundred and ninety nine values
and in those second cluster there are nine hundred ninety five values present.
Now, let us implement anomaly detection with DV scan, so anomaly detection is the practice.
We want to isolate the points which are actually out there for the data.
We don't we want to capture those data points, which are not like the other data point.
So how do we do that?
So we will be able to implement that using the scan.
So it has already been implemented here.
So you can see these blue points, which we have left here, are actually the outliers and these are
actually the anomalies in the data.
That is, it does not like the other data points which we have in these clusters.
Similarly, we have this wholesale customer data set.
So in this dataset, we have data about Milkin grocery and different groceries items.
So we want to check about Milkin Grocery that how these values are associated and who are good customers,
who bad customers and which customers are buying more of the products from from us and which customers
are buying less products for from us.
So we will simply plot this.
So here, using these values, you can see the plot which has been generated, looks something like
this.
Now, from this visualization, we can easily see that the points which are scattered outside this dark
area together are the outliers or are the anomaly points.
So all of these points are actually the anomaly points and they should not be a part of this particular
cluster.
So let us implement the same.
So now we will implement the V scan for this and we will run this for different epsilon values.
So here we are running this four different epsilon values ranging from zero point five to five.
And we are giving the V we scan running the B scan for this and giving the minimum sample values 20.
Then we're fitting the grocery dataset to it.
Now we will see the values for Epsilon and the outlier data.
So here you can see these are the different Absolon values and this is the percentage of data which
is constructed, which is considered in the outliers.
Now, using this particular data, using this particular data, we are calculating these outliers by
rounding out the data that is the top and some of the outliers divided by the labor data.
So this gives the percentage of outliers.
So let us see if I want to find out the top one percent or top two percent of my customers to whom I
want to give a particular discount so that they buy more products from us, then I can do that from
this outlier data.
So what I can do is I can find out that these are the top two percent, which is present at the epsilon
value, one point to thirty one point three to one point forty one.
So based on these Epsilon values or four, five percent of my body that I can use Epsilon values, zero
point seven seven five one.
So I'm all of these Epsilon values, I can actually find out the number, the percentage of the outliers.
And based on the outlier, which I want to see, I want five percent of outlier data or six percent
of our player data.
I can find out those data points and I can isolate those data points and then try to get them for my
next campaign or the or the special or for which I have for them.
So this is what I can do using anomaly detection and similarly, if there are certain transactions which
are actually of which are actually out of the usual transaction, which a person makes, for example,
we are talking about credit card transactions.
So there could be certain transactions which are slightly out of range.
But I want to find out those transactions which are highly different from the actual data.
So maybe I can consider the top five percent of the outliers and then the people and tell them that
these are the actions which are happening in Ukraine.
So is there any kind of problem with your facing or have you lost your credit card?
So these kind of questions can be asked to those people.
It's.
So here's what I found is I have used Absolon values zero point seven seven, which is.
The percentage of outlier being five point six eight.
So for this I'm finding out and creating the disasters here.
So here you can see the cluster which has been created.
So these are the outlier values which are coloured and blue and having the label minus one and the other
cluster, the actual cluster is in color, orange labelled as zero.
So this is what we have gained from the scan and how we can implement anomaly detection and clustering
on top of this.
Next, we will discuss about a very important algorithm, which is the dimensionality reduction.
So I hope you will learn a lot from that.
Thank you.