Hi and welcome back to the course in this section, will take a look at another point cloud project.
This time we're doing segmentation as opposed to classification.
So let's get started.
So we're doing point cloud segmentation.
So what exactly is going cloud segmentation?
Well, imagine you have that point cloud that say, let's see.
It's a point cloud of a table and chair together.
Now we can individually classify them.
What if we wanted to categorize the legs of the table or chair separately from this seat or the table
top or the backrest?
Those types of things that would involve segmentation.
So that's what would be going.
What what we would be doing in this lesson.
So let's import to a libraries and download our data set.
So the dataset we'll be looking at is the shape that dataset.
Let's take a look at that.
We can see it covers 55 common objects with about fifty one thousand unique models.
So you can see, let's take a look at models.
Well, maybe not sitting.
Still, look at browsing categories here.
So you can see this is what the model consists.
These nice 3D models of different airplanes, different could bird houses.
There's a lot of weight off.
They're going to be loaded now.
So you can see this is a very cool, exhaustive dataset, basically for ships instead of images like
image.
And it was a huge dataset consisting of so much different categories and health care labels ship that
is similar as well.
And they're basically the one we're going to be using.
Here is a 12 object category of the Pasco Treaty plus as part of the ship that core dataset.
So let's download the dataset here.
That takes about two minutes and we load only to sit here as well.
And in this example, we'll be turning point not to the previous model we use to segment parts of an
airplane body.
It's a pretty cool project, isn't it?
So now let's take a look at how the dataset is structured so you can see there's a point clouds here.
This is a list of number five list of objects that contained X, Y and Z coordinates of that point.
Then there's the test point clouds, which is the same format, but doesn't have to label so that our
test data.
And then it's the labels of all the datasets here, as well as the point cloud labels, which again
are the labels for each coordinate and if hot one encoded form corresponding to the point cloud list.
So let's load our datasets here and get on a more pointed out information.
This takes a while to run, maybe just about under five minutes.
So now we can take a look at some samples from the in-memory arrays we just created.
So let's take a look at this here.
You can see body wing. point cloud levels.
What are you doing?
So that's a separate pots tail of the airplane, as well as a point clouds associates associated with
it.
So no, it's actually visualize this.
So let's take a look at that.
Well, it's pretty cool.
Isn't it an actual airplane in 3D space?
You can see this is a body.
This is the wing and believing this is part of the tail.
Yes, it is.
These are the engines they built separately.
Here's another plane as well.
You can see it's been segmented and labeled nicely.
So not that we have already to visualize and we've loaded it.
We can run some processing on it.
So this pre-processing involves some sampling, so we have a fixed number to represent each of those
point clouds, as well as normalization and some to scale invariant to some normalization to make the
data scalar variant.
So let's take a look at what a normalized data looks like, and you can see this again here.
And here's another normalized version of that dataset.
So now we can create a TensorFlow format dataset, so we just create this loaded function here to load
a batch of data.
And we have an augmentation functions here.
So we're just going to add some random jitter.
Believe to this to random noise.
Then we just generate the datasets here with this function and we just create everything here and we
can take a look at what our training and validation data sets look like.
It's it's 29 100 for training, 739 for validation.
And here the fixed point, not model.
So you can see we have our input points going here and by tree input, transform this MLP right there.
Then there's a future transform and the MLP here.
And then this gives us the global features here.
And then there's another MLP that outputs the scores as well.
And then this new segmentation part's here.
Basically, this takes the point features and predicts the scores.
For each point effectively.
So something called permutation and variance.
So given how a point cloud data is quite unstructured and any number of points, you have a number of
permutations.
We do need to find a way to represent, to make it invariant, to input permutations in future.
So what do we do here?
We have to ensure that it's invariant in a way so that when it's encoded into the global feature vector,
it basically can handle some sort of jitter and hautement and variance.
So that's what this part of the network does here provides some global sorry, some variance there.
Two different permutations of the data.
And notice also transformation and variance here.
So we need the transformation and variance to handle issues like scaling and translation as well.
And then also, we have to talk about the point interactions.
So in segmentation, we have to be able to leverage local point features because they're they often
interact and correlate or have some sort of relationship.
So that's what we do here for point that actually takes advantage of that.
I guess you could see correlation or interaction so that we could actually use that information for
our segmentation network.
So now we're going to build the point net network.
So we have all building blocks here.
We have to construct MLP block.
Then we have the orthogonal regularised, which you would have seen in the previous chapter, as well
as, no, we have the transmission that so we create that more layer and this is the transmission block
as well.
And now let's get the ship segmentation model finally, so we can create a model or final model.
So that's now we dysfunction creates that final model for all those building blocks we created previously.
And so now we instantiate the model, just get the number of points and classes from a batch of the
X Y data here.
And then we just create the segmentation model.
SUMMARY Right, dear.
So we have seven million parameters in this network's fairly moderately sized and now we can start training
on that.
We're here.
So let's begin treating it well as this is done in the training group here.
So this is your own experiment loop here that actually executes the training.
And finally, we can begin to train and train for 60 epochs where it takes about 20 seconds.
But epoch, so roughly this would have taken me.
I don't have the time here, but you can just multiply 20 by 60 and given the time it would take just
under 10 minutes.
And now you can visualize what the model is predicting, so we can take a look at the training loss
as well as accuracy and see accuracy is going up over.
There have been some weird spikes in the validation accuracy as well as you can see and validation lost.
Not sure what this would happened in this batch, but anyway, you can see it was getting better close
to how it is and for training at our own mid-eighties, for the validation accuracy.
And now we can finally run inference on some random input data from the validation batch.
So you can see this is an airplane here, and we're predicting fairly accurately to body the wings.
We can see that there and the teal.
Let's take a look again.
So yeah, it's pretty good.
So that's it for this lesson.
That's pretty much very in-depth lesson where we take a deeper look at some point that for point cloud
segmentation, do I do encourage you to check out this link here to learn more about this repository,
and you can use it when your own call up the books on your local machine or some easy to instance out
there of which you are your machine cloud instance is enough to be an Amazon one.
It can be any.
And yeah, that's it for this lesson.
I hope you enjoyed it.
Thank you for watching.