In the section on data preparation, we will start by splitting the data, thus creating a train and
validation data set.
Then we'll preprocesses data, visualize this data.
And finally we are going to make use of voxel 51 for a much better visualization.
Diving into the splitting of our data set, we have this part which we're going to create.
So we have the train path which will simply be the image path.
So here we have image path.
And let's copy the path.
Or let's just let's just copy this.
No need to open that up.
So we copy it and paste it in here.
And then uh we also copy the annotation path.
That's the masks path and paste out here.
So we have annotation path.
There we go.
We copy from here and paste in the cell.
And then we are now going to create our validation path.
So let's just copy this and then um modify it slightly val data set.
And instead of path we have the val image path.
And then the val annotation path okay.
So that should be fine.
This looks fine.
We have images um images.
Uh this should be masks actually.
So let's copy this again.
Copy and paste it in here.
There we go.
And then we also paste this in here.
So this oops.
Um take this off and paste it in here.
So we have val data set val data set there okay.
So this is our different parts.
That's the image and or the train and the validation paths.
With this path defined the next thing we want to do is create this um validation data set directory.
So we have the val data set directory which we're going to create.
And then we also have the PNG images directory and the PNG masks directory.
So it's just going to be similar to what we had already from this original data set.
So now this two year will be for our train.
And then the other two will be for the validation.
Let's run this and there we go.
So let's refresh um and then see what we get.
So here we have our data set which is actually our train data set.
And then here we have the validation data set.
We open this up.
You see we have PNG images and PNG masks.
That's fine.
Now the next step we want to carry out is define or pick out randomly some images from our train data
set and then move them to our validation data set folder.
Now let's select this.
We randomly picked out this um list of images.
If you open up this file or open up this folder.
And here you would find that all the images and even the masks take up this format.
You can see how we have the image underscore the number which is one of this.
And dot png.
So that's uh, the formatting of all our images and even the masks.
So if we open up this mask here, you should be able to have that same format.
Um, here's img underscore number dot png.
And with the mask is set instead of the img.
So that's the only difference.
Let's now use the short tailed tool to move this files or these images from this original folder to
the validation folder.
So we'll start by importing Shutil.
There we go.
Once we import that we could now make use of it.
So we would do for all the different um elements in here.
So for um path or for name in val list.
So for naming val list we are going to do Shutil um move.
And then we specify the initial path and then the final path.
Okay.
So our initial path in a case like this let's suppose we had 0001.
So if we had zero zero 1 or 0 001 as an example, we want to have this path.
That's let's take this that's image path because up to this point is this is the path.
So we have image path.
Then we just simply write here in path.
So we want to have this path.
And then we also want to have IMG underscore.
So take this off plus the name because we're going through this list.
So plus name for a specific name in that list.
And then plus um dot png okay.
So that's it.
And then for the final path final destination, let's copy this and paste out here.
For our final destination, we would have instead vile image path.
And then we'll also have um image underscore name dot png.
So this is going to be and then now we'll just do the same for the annotations.
Remember the annotations we had seq instead of image or instead of img.
So here we have um ano path ano path.
And then here we have an o path.
Okay so let's take this off.
We now have seq.
And then here we have seq.
And let's run this and then see what we get.
We are getting an error.
Let's um look at this.
When we look at this, you see there's no, uh, the arrows showing no such file or directory as this
year.
We see that there is this slash which we omitted.
So let's, um, slide up and then modify here, just add the slash and then we rerun again.
So we have here slash and then here slash run again and move this files.
We still have another error.
And this time around we notice also that we omitted this images folder in our PNG images folder.
So getting back up we just need to make this directory.
So we have images.
And then we also have masks.
So masks okay.
So that's it.
Let's run this again.
Um that's already been done.
And then now let's um move.
We have the files which have been moved as we could see here.
If we click Open Images right here, you could find these files.
Now the next step will be to create a TensorFlow data set.
So we'll have our train data set um which is going to be our TensorFlow data set made of the different
file paths.
So let's go ahead and do that um from tensor slices.
And then the way we will define this is such that we have this image or the path of an image and its
corresponding annotation.
So here we have Im path that's image path.
And then for I in the image list that's the list of all our images.
So we have image path.
And then we'll also have the annotation.
So this is the first slice.
And then our second slice.
Here we have annotation.
Let's just put it below.
So copy this and paste here.
So here we have annotation annotation and annotation.
So for every um element in this TensorFlow data set we have an image path and its corresponding annotation
path.
Let's run this and let's do for I in train data set train data set.
Let's take an element and then we print it out.
Well we have it an error.
So let's get back up here and see why are we getting this error.
Here we have list object has no attribute is identifier.
Now let's get back and try to see where we made an error.
Um looking at this we forgot to make this a tuple.
So this is actually a tuple composed of this input or this image and its corresponding annotation.
So let's run this again and see what we get.
There we go.
You can now see that we have this image or this image path scroll scroll this way.
So we could see that image path.
We have this image path.
And then we also have its corresponding annotation path.
Um, and also looking at this annotation path you will find that it's different actually.
So we should modify this.
Here we have 0987 whereas here we have 0336.
This means that looping through the image path and the annotation path, we are not getting the exact
same order of the different, um, elements.
That's the different images and the annotations.
So that said, instead of looping through the annotation path here, we want to loop through the image
path.
So we'll still loop through the image path.
But since we do not want to have the images, but instead of segmentation, um, what we'll do is we
are going to take off part of this eye right here.
Now let's, let's, let's go step by step so you understand exactly what's going on.
So if you do um, for I in um list directory image path and then you print out I, you print out I brick.
If you print out I, you would.
Find out.
We have this here.
This path, or rather this file name image as img 0336 png.
Okay, now the idea here is to take off this img and replace it with seg.
So let me get back to this.
Easier to open this up since there are fewer files.
So let's scroll down and you see you see here we have seg.
So what we want to do is maintain this path to maintain this path while taking off this first three
letters.
So now what we'll do is we will have I.
So if we want to have I or let's say we take seg, we want to do seg plus or rather seg like this seg
plus I, which goes from because this is zero, one, two, three.
So it goes from three to the end, from three to the end.
Okay, run that again and you see we have 60336.
That's fine.
So getting back here we're going to have an annotation path.
Plus I but instead of having plus I we're going to have plus seg.
There we go.
And then plus I from three to the end.
So let's run this again and then check this out.
We now see here that we have exact same um file.
So we have the image and then its corresponding annotation which is uh which matches exactly what we
expect.
So here we have 0336.
And then here we have 0336.
We could print out let's say three orders.
So we see that all this match because this time around we are taking from the same list.
So that's it
0613070306130703.
So that's fine.
We just replicate this for the validation and we move to the next step.
So here we have validation.
And then we have val image path val image path.
We also have oops we have val image path and val image path.
Then here we have val image path tool okay.
So we have the training and the validation sets ready.
Go run this take this off.
Then we could print out for I in val data set.
Let's take a single element.
Let's print out.
Let's print out I.
There we go.
We have this now.
So here we have image.
Um, 0049 and then here we have 60049.
And this is a valid data set.
Whereas here is our train data set.
Let's obtain the length of our train data set and that of the validation data set.
We have length train data set.
And then Len val data set.
There we go.
So from here we'll see that if we sum this two up we should obtain a thousand different images.
And at this point we are going to dive into the next step which is pre-processing.
So we've done the splitting.
And now we're heading over to preprocess.