In this section, we'll dive into preparing our data.
So we have this, um, malaria data set which is made available on the TensorFlow platform.
So here we let's scroll down.
Here we have a description.
The malaria data set contains 27,558 cell images with equal instances of parasitized and uninfected
cells from the teen blood smear slide images of segmented cells.
Um, we have this additional information right here.
And then we can see the size.
And then we have this um, figure which shows us some sample images.
Now we are going to click on explore in Know Your Data.
So let's open this.
And then we could um check out um, different um data samples we have here and overall information about
our data.
Selecting um the cell at random from the right.
You see here we have this uninfected cell.
Um, we have all the different features and the values has faces.
False.
It's portrait.
False labels.
Um, number of faces.
Well, this is, um, a general know your data platform.
So you would have, um, these kinds of features actually.
So although we're not, um, working with feature with faces in this case, um, we would definitely
have this because this is something general.
So we have um, the aspect ratio.
We have the format as PNG, we have the resolution, we have the mode RGB, we have the height and then
we have the width.
So you see we have this um image right here exposure quality sharpness score and so on and so forth.
So that's it.
We see we have the label and then we have our image.
So you could pick out some other one and then check that out.
Anyways um here you notice that we only have this uninfected um cells.
And that's simply because we start with the uninfected.
That's out of this 27,000 items.
The first, um, let's say 13,000 or 13,500 or whatever number is, um, uninfected cells, and then
the rest are parasitized cells.
So if we change the order, um, you see now we have the parasitized cells and clearly you could see,
um, visually the difference between the parasitized and then the, uh, uninfected.
See, notice how you have this little, um, purple patches here and there in the different images.
And then when you click here, you, you find that the image is much clearer or you, you don't have
those kinds of patches we had, um, here, like, you see these patches we have with a parasitized
cells, we no longer have them with the uninfected cells.
You could also decide to group by, um, these different features.
So let's say for example, uh, let's group by aspect ratio.
See we have the different aspect ratios and so on and so forth.
So you could decide, um, on how you want to group, um, the images and visualize them that way.
So that's it for this part.
We'll now dive into loading our data set.
And to load our data set we'll make use of TensorFlow data set um load method.
So we'll head over to the documentation.
That's TensorFlow data sets API docs um Python to be specific TensorFlow data sets.
And then we have this load method right here.
So with this load method we'll be able to simply specify like let's get back here.
We'll be able to simply specify this name and then load um up this data such that we'll have as output.
Um here our TensorFlow data set with obviously it's information.
With that said we'll start with the import.
So we get back to our notebook and then we simply import TensorFlow as tf.
Then we import numpy.
NumPy as np we import our matplotlib for visualizations matplotlib um pyplot as plt.
And then we import TensorFlow.
TensorFlow data sets data sets as tfds.
We'll run that and make sure you have the runtime set to GPU.
Let's now go ahead and load the data.
So we have data set and then the data set um info.
Then we have TensorFlow data set TensorFlow.
That's tfds um load.
And then now all you need to specify is malaria.
So what we have seen already, that's the name which we saw in the documentation is the exact same name
we use here.
So we have that malaria.
And then we have with info with.
Info set to true.
Let's run that and then see what we obtain.
So let's have data set data set.
And then we have data set um info with this downloaded.
You see here we have the data set and we have the data set info.
So we have this data set which is uh Prefetched data set.
You can see the the shape right here.
None by none by three.
So we have an RGB image and the data type is unsigned um int eight.
Uh we have the label which is an integer.
So definitely we will have um zero for let's say parasitic parasitic.
And then we will have one for um uninfected.
Uninfected.
So that's it.
So let's take this off and say we, we select only train and there we go.
You see we have just that data set.
So let's say for um I in train and let's take a single element.
So let's take just that single element and let's print out that I.
Oh that's it.
You see, we have the image.
That's it, 103 by 103 by three.
And then we scroll down and we have the corresponding label which is one.
And then we have data set info or the name, the full name, the description.
And um right up to the citation, we could verify that the length of our data set is 27,558.
By printing this out.
And then we could now dive into splitting up our data set.
So here we have the split.
You see that we specify train train and train.
Because obviously um our data set um initially is made of just the train data.
And so now what we are saying that we want to have all elements between all the first 80%.
So essentially what we're saying is we're going to have um, element zero right up to element um, 0.8
times 27,558.
So let's take this off.
We run that and you see, uh, what this tells us is we're going from zero right up to zero right up
to 20, 2000, um, 46.
So this first 22,000 elements will be used for training.
Then the next 10%, um, of our full data set will be used for validation.
And then the final 10% will be used for testing.
So with that said, let's run this.
And now let's say this set, you will find that we will have this list made of three different, um,
data sets.
So let's say we have data set data set, equal data set um data set zero.
And then we have the same we repeat the same for validation and testing.
So here we have two.
We have um one one.
There we go.
We have one.
And then we could take this off.
We have validation and then we have testing.
So let's run that.
And now we could do the length or we could get the length.
We could get the length of our train data set train data set.
And we could do the same for the validation and the testing test.
And Val run that and see what we get.
So that's it.
That's how we split up our data set.
We can again print out those different lengths for the train validation and testing.
So you see we have 20,046 27,056 for both the validation and the testing.
From here we'll go ahead and shuffle our data set.
If you check out, um, documentation, you see we have shuffle files.
Let's just take this this way.
And we have here shuffle files and we'll set that to true.
And the reason why we need all it's important for us to shuffle is simple.
Um, suppose that we you have some data, you have some data.
And this first part is, um, uninfected.
And then this next part, this next part.
Let's take this off.
This next part is, um, parasitized.
Let's take this from here.
Now, if you take out the first 80, um, percent, then, um, you end up with something like this
being your train.
So this will be your train data.
Let's take that off.
So let's say this this is going to be our train data.
And then this is going to be our validation and testing.
But the problem now is that, uh.
Validation and testing is made of only parasitized cells, whereas if we had shuffled our data, then
we would have a mixture of the parasitized and um, uninfected cells.
So you would have something like this instead.
Let's just, um, take this, uh, randomly.
Well, randomly have that.
And there we go.
So in this case, you see that we're going to have or we're going to go from let's say let's add this
up.
We have that.
We're going to have this first 80% and then we'll break this up again.
And you see now that our validation and the testing data sets have both the uninfected and the parasitized
cells.
So that's why it's very important for us to, um, shuffle our files, run that again.
And there we go.
So we could we, we could, um, run this again and then feel free to check out, um, let's say Val.
Val and that's it.
Well, this should be let's get back.
This should be val data set instead.
So here we have val data set.
And we could take that off.
Run that and see what we get.
You see we have our image and we have its corresponding label.
Another way in which we could carry out the splitting is actually manually.
So instead of relying on this um, split parameter right here, we could actually manually split up
our data set.
And this is important.
And this is useful in cases where we're not dealing with a data set provided by TensorFlow.
So let's say we do not have the split.
Let's take this off.
Let's suppose we do not have the split.
We still have the shuffling.
Um, now we're going to define our split method.
So here we have the split method which will take in our data set.
And then we'll take in our split um ratios.
So we'll define uh train ratio.
Train train ratio um val ratio.
And then we have test ratio.
So right here we will start by defining let's return return none.
So what we'll do is we're going to start by defining this ratios.
So let's say we have train ratio train ratio which will set to 0.80.8.
And then we have val and test ratios.
So test ratio and then val ratio.
And here is 0.1.
And and 0.1.
That's it.
And then now what we'll do is we're going to create a simple data set to illustrate how um our manual
splitting will work.
We have the data set um or let's let's call this let's just say RDS our data set.
And then we have TensorFlow data data data set um range.
And then we put ten.
So we get this ten values.
So let's just print out.
Well let's just let's just um call this um our splits method.
So here we'll have train um RDS we'll have Val RDS.
And then we will have um, test RDS.
So that's what we expect to be returned right here.
Train um data set.
We have um val data set and then we have test data set.
That's it.
And then now we'll call on our method split which will take in the data set or RDS.
And then um we give it we'll pass in it.
Um, the train ratio we pass in the val ratio.
And then we also pass in the test ratio.
So that's it.
Let's get back test ratio.
Let's now comment this out as we've just done.
And then we print out the um our data set as numpy iterator.
And then we have list.
And run that and see what we get.
There we go.
As you could see, we have values ranging from 0 to 9, which is what we expect because that's what
we specified right here.
Now, if we want this first, um, 80%, that's this first eight elements, then it suffices to do or
to, to get right here and say tick or not.
Not this.
Let's say we have or let's say, let's say we just define this values right here and we'll say, um,
dz, take the first eight elements or take the train ratio, the train ratio times, um, the total
or let's say total, um, total total number of elements.
We set that to ten.
We set it to ten.
There we go.
And we replace this with total.
Um that's it.
We have total.
So it's going to be eight like well this should be an int.
So we'll just uh make sure we have an int right there.
Run that again and then we'll see what we get.
See we have our data set.
And now we're making sure we want to get the first eight elements.
Um this should be well let's print let's the initial data set.
Let's now get the val data set.
So because this is a complete data set, now we want to have um the val data set.
You'd see that we get the first eight elements.
You see we have this first eight elements.
Now if we want to get the next eight elements, all you need to do is skip the first eight.
And then once you skip the first eight, you take the next.
So this is 0.1.
So what we'll do is we're going to have our Val data set or well this this is actually train.
So this is actually train data set.
So our train data set.
And then now for our Val data set for our Val data set.
Let's take this off for our while data set we'll specify the val ratio.
So here we have our val ratio.
Um but also this is skip.
So we want to skip we want to skip um the first eight elements.
So let's say we have train.
See that.
So this is the first eight elements that we're going to skip.
And we will take all this to the end.
So here is skip.
And then here is take.
Take simply means you have a data set and you take uh the first um n elements.
So if n is set to eight then we'll take the first eight elements.
Now for train you we or rather for skip you simply um moving or you're, you're going to start from
the n plus one element.
So let's say val data set and we skip this first eight elements.
Um here we have train.
So yeah we should have train.
Train.
Let's copy this.
There we go.
We paste that out.
So now we should have vowel.
So let's take this off and run that.
So we see how um we get the vowel data set.
Well this for now is a combination of the validation and the test data set.
Because we just kept the first eight elements.
You see.
Now we have eight nine.
So this last elements now we'll just change this slightly and call this vowel test vowel test.
Because the combination of the validation and the test data set.
So that's it.
We have our test.
And now we'll define our vowel data set to be simply our vowel test vowel test data set.
And then now we take the top elements.
Or rather we take this first element.
So we have um int.
Let's cancel that.
We have your int and then we have our vowel ratio times um the total.
So this is going to give us one.
So we'll just take the first element.
And then we have um for the test.
Let's take this off.
Piece it out.
We have test and then we have vowel test data set.
But instead now we skip.
So we skip that.
And then we have um 0.1 times ten which is one.
And we get this last element.
So now we could print out let's print out um our vowel and test data sets.
So take this off.
We have vowel and then take this off we have test.
Run that again and see what we get.
You can see that we have now eight for the vowel.
And then we have nine for the test.
Now let's say we had changed this to two, two and six.
Run that again and then see um, that we have for the train the first six elements, and then for the
next we, we're going to have the next two and the last two.
See here we have our train data set.
Here we have our vowel data set.
And then here we have our test data set.
So our method works perfectly.
That's it.
Now let's dive into our split method and then um implement what we just described right here.
So we'll start by getting this total.
To get total you just need to have the data set size.
So let's say data set size which is simply the length of our data set.
And once you have the length of the data set you now do train DDS.
That's exactly what we had seen already here.
And we specify the train ratio times total.
So let's just copy all this or let's cut that out and then paste out here shift this.
There we go.
So we have our train data set val data set and test data set.
Now this is instead data set size data set size.
And here we have data set um data set size.
There we go.
We have um we got we can now run all this.
Then let's get back here.
Let's take all this off.
And now we have our train data set Val data set and test data set.
So let's get back to 0.1 and then 0.8.
Uh we don't need this total anymore.
We don't need this to our data set.
Is this, um data set.
So let's let's take this off.
So we have data set, we run that and then see what we get while that's running.
We could simply update this here to get the length of train data set.
We'll get a length of val data set.
And then we get the length of um test data set on that two.
And make sure that we have exact same lengths for both the first method and our manual splitting method.
So we get in this error.
Um this has no attribute take.
And that's because here we we need to specify that this is train.
So we run that again.
That should be fine.
Now see.
And then we also run this tool.
So we now compare the lengths.
And you could see that they are almost the same.
The only difference we have comes from the fact that um this INT rounds down whereas um with this method
or making use of the load method, um, the load method split, we instead have a round up.
So that's why we go from 275 5 to 2 seven, five six.
Whereas with the manual split we go from 27550. 8 to 2 755.
So that's why we have this little difference.
Um, that said that's it for the section.
We're going to move on to the next section where we'll dive into visualizing our data.