Hi and welcome back to the course intersection.
We'll take a look at replicating or building delineate an Alex that architectures using carrots with
TensorFlow 2.0.
So let's get started.
So open this notebook here, which I've already done.
And first, we'll start with the Linnet architecture.
So you may have remembered the net architecture was made famous in the late 90s, where the researchers
trained it on the amnesty to set that was generated by Eusebius and achieved some remarkable results.
So we're going to try to implement that exact same architecture using carrots.
So this is an overview of the architecture that the researchers used.
You can see the layers described properly here.
You can see it in the diagram as well.
We have Plyo convolutions here now.
One thing to note the input the researchers use in the 90s when this paper was published was 2D two
by 2D two.
However, when we lodo amnesty, they said we're loading to one.
That's a 28 by 28 resolution.
So we lose four pixels.
Not that much, but we lose some information going forward.
So it's not exactly the same dataset that we'll be working on.
However, they're going to use roughly the same architecture.
However, the output shapes will be slightly different because our input starts at a smaller that mentioned.
So just to quickly recap, we have our first layer here with five by five filters, right?
One padding zero.
Then we have a max pool for average pooling to use enough max pooling.
Then we have a continuity layer average pulling again conv with 120 filters, which is quite a bit.
And then we have the fully connected layers here with 220 units, 84 units.
Then finally, outputs to the 10 here.
So let's build this in Keros.
So as I said, we're not building exactly the same thing.
We're building something that's quite similar.
But for now, this should be familiar to you, this part of the code.
Here's where we lowered the amnesty to set.
Here's where we just get our image rows and columns, such as 28 and 28, respectively.
Here's where we add a fourth dimension by reshaping the the data to that specific size that we want,
which is this.
Then we just get the full input image shape.
We convert float to the two.
We normalize our data and then we get one hot and coatings for the labels.
Then get a number of classes as well from that, as well as a number of pixels, which I don't believe
we use here.
But we might use actually maybe this copied and pasted this line unnecessarily, but we'll see if we
use it.
And this is just to go over this.
These are inputs.
This is a optimizer we'll be using at a delta.
We don't actually use batch gnome in this model, so it's taking this out and everything else should
be fine.
Here we are ready to run this block of code.
So go ahead and run this.
It may take a little while because it's the first block of code you're running in this notebook because
what happens when you first run a block?
It has to connect to a cloud instance in the Google Cloud platform to give you basically the machine
that we're running this code on.
So now let's replicate.
Alternately, as if you remember correctly, we had six filters here at five by five kernel sizes where
we didn't.
We use zero padding initially and the architecture.
However, because we have a 28 by 28, which is slightly less input image sized and the 32 by 32, the
researchers would put that, then I'm going to use padding equal to same padding, equal to same basically
as the padding around the input image so that the output feature map is the same size as the input feature
map.
Somebody use that for all the conflicts here.
However, I'm going to use the same number of filters six 16 and one 120 would redo activation.
We'll use max pooling instead of average balloon.
Because I've done this experiment, max pooling does work better.
So I mean, there's no point in replicating the exact same thing, but will it replicate something that's
quite similar?
So we do have a max pool is there, then we flatten.
We're going to use this same one hundred and twenty and 84 units, an output to the ten output nodes.
And Wolf the optimizer.
We're going to use other delta for mid metrics again, accuracy and for loss, categorical cross entropy,
which is appropriate for multiclass classification problem like this?
So let's run this and we get our model output here, which is quite nice, quite convenient.
To have this, we can double check all of these things, all the output sizes.
This is actually quite small, but that should be should work fine.
The total number of parameters is only a hundred and ninety one thousand, which which is tiny.
But let's see how performance train 10 epochs on this model.
Let's see if it gives us these results.
I would say a decent result should be somewhere above 80 percent accuracy after 10 epochs, so let's
see how that goes.
It's going to trend quite quickly, as you can see.
So the accuracy of the first e-book wasn't that great, 11 percent, but then it went up to fifteen
point nine twenty one point eight.
Let's keep going.
Twenty six point four one point three one nine point twenty three seven point four one.
I don't think we're going to get anywhere near 80 now, but as you can see, it's quite fast to train,
so we can easily probably treat this for like 50 bucks, which I will do other than this 50 bucks here
and to see what it gives us.
So we'll move on to the adolescent, though.
In the meantime, just so you know, there's nothing special going on, this lupus as a standard could
be used for treating me as a bad size of one twenty eight.
Just use a model that fit procedure, specify that size of your box, which we define a pill.
We just point to a validation test dataset and the labels.
We always shuffle the output and save the model at the end just so we want to have it saved just in
case we want to reuse it again.
And we just said a few boosts when we start to evaluate the performance for vertical one, just to get
some more information out of it.
And then we print the test Lawson test accuracy at the end there.
So as you can see in 50 bucks is actually starting to do much better, which this highlights an important
lesson right now because initially remember, we started, our accuracy was much worse.
Well, there's two things happening here.
One.
This actually of just thinking about it, this actually may have been a feature implemented in TensorFlow
2.0 that I was not aware of because I've been using PI to it so much lately, but it continue with treating
the model, so it didn't actually start off from the initial scratch layer or initially started.
It was at 11 percent accuracy and when we ended it after 10 epochs, it was at 55 percent accuracy.
That's why when we run this block of code again, it continues the model, so it saves the weights and
continuous training.
Previously, it would have just started from scratch.
So it's actually quite nice to see if we can actually resume training conveniently like this.
So that's how we're able to get so much better accuracy when starting this because we're not starting
from scratch, we're continuing training.
And you can see at the end, maybe close to 50 bucks we will have in the 90s.
Let's see if we get close to 1990, which is where the researchers got with the initial Leonard architecture.
Somehow, I don't want to come close, but I suspect we'll get may be close to 94 percent accuracy,
so I'll leave this.
OK, so you can see we didn't get any food, but we got ninety three point one, three percent accuracy,
which is quite good.
Not disappointed with that after such a short trading time.
We're getting a decent model out of this.
And what's remarkable is that we're getting that performance with such a tiny model of one hundred and
ninety one thousand parameters is tiny, as you can see in it trains quite quickly.
So the code is running here right now.
What it's still doing in the background is doing a model that evaluate metric, which just gets to performance
metrics out of this.
So we're going to get to a test loss and test accuracy at the end of this.
So let's wait for that.
In the meantime, though, let's take a look at the Alex Net Network, which we're going to train.
So we're going to train an Alex net, which is a lot deeper, a lot more complicated network on the
Safari 10 dataset.
So this is an illustration of the Alex Net architecture, and this is an illustration of the Safari
10 dataset.
So you can see it has 10 classes, 10 of these fairly different classes.
I would say, although dog and there could be quite similar sometimes, but the Dems do have these horns,
which make them quite distinctive.
Horses can be similar to dogs as well sometimes, but these images do look a lot different.
Frog, apparently, was an important category this they saw fit to fit in.
But anyway, let's keep going.
So what will do with this?
Lower the 10 dataset here?
So let's run this.
So it's now loading the data set right now, it should take roughly about a minute, maybe less.
In the meantime, we can scroll back up and check our performance metrics at the end, which this is
what we do know on what is giving us ninety three point one per cent accuracy and our test losses point
to four, which is quite good.
It's not amazing.
If you want, I would recommend you train this for maybe 150 bucks and as well.
It's a very good experiment for you to go to the architecture here of the Alex Smith network and added
more filters and went way to disco here.
Added more filters change a filter size.
Maybe add a new, fully connected live increase.
Number of nodes in this fully connected layers maybe adjust destroyed or remove some padding if you
wanted to set padding to be zero.
You can.
You can just put off a commentaire for it to set the padding at zero.
Use.
Valid instead of putting equal seem valid, it basically tells the padding to basically be none, so
it's zero padding.
So let's go back down.
Sorry for the quick scrolling and we've loaded our sofar 10 data sets.
We can see it's 50000 images in the training dataset, 10000 and attested a set dimensions are 32 by
22 and your color.
That's why they have a depth of three.
So here is where we build Alex.
That network you can see we have five convolutional layers in a separate to them nicely.
You have 96 filters in the first one 256 and the second 512 and the third a thousand twenty four and
the fourth and Intels and 24 in the fifth.
And you can see it starts at eleven by eleven and five by five tree by tree tree by Tree Tree Battery,
which is actually against the convention of CNN filters.
I prefer to use smaller filters up in front and larger filters behind.
However, this is what the Alex Net Network it looks like, so we wouldn't change it too much.
We also use an L to regularize the head can see so that the zero.
But it does apply.
Some regularization still.
Actually, I'm not even sure if it does.
Let's set this to zero zero one, actually, just in case, and we're using batch gnome here.
Now, I don't believe that Cinnamon was part of the initial Alex net network.
However, that's added into this.
Like, let's make some improvements on it and we will add drop out as well.
And you can see our fully connected layers are quite bigger.
This one has treated 72 nodes, as well as four thousand ninety six.
Number of classes is 10.
We apply Bartolome again at the end of this, and we were using drop eight point five.
I wouldn't really want to go.
It's so high.
I want to stick to 0.3, but nevertheless, let's try it.
So let's print this.
And lastly, we're using add a delta again as the optimizer so that a delta is quite good.
However, we could use stochastic gradient descent or Adam.
I tend to use Adam a lot, so you can change it to Adam if you want to see what would with the Dakotas
to the
if you wanted to see what the actual way the change this is.
You can dig into the Keros documentation and see what they actually name these different optimizers.
I don't believe this just Adam, but I'm not entirely sure it can be capital e needs in is case sensitive,
so it needs to be set correctly.
So there we go.
So long architectures.
As you can see, it's quite extensive.
Isn't many, many different layers in Alex, but not nearly as much as video resonate or some of those
that are deep CNN's.
But we can see this 78 million parameters, which is quite a lot actually for such a I mean, it's not
a primitive network, but for the network that was introduced maybe ten years ago, and this is quite
heavy.
So will lose about size of 64 will train for 25 e-books will do a regular training method here, see
the model and get the performance scores at the end.
So let's begin training and let's observe.
So sit back and relax to this one is going to take a while to train, so I'm going to let the video
stop recording now and resume when it's finished streamed because this is going to take maybe about
15 15 minutes roughly estimate.
OK.
So that's it for this lesson when I come back after this model has been trained will discuss these results
here.
Thank you.
And using the Alex Net architecture, and you can see, after roughly 20 minutes of training, 25 epochs
have completed and we have gotten 61 percent accuracy now that's OK.
It's not too bad.
I mean, if we could do a lot better and you can see of scroll down and a physical image here, which
is taken from people's would could and they rank basically all of the state of the art.
CNN's and you can see this is their performance on zaffar 10.
And you can see I mean, we've gotten some ridiculously high accuracy rates with the modern day CNN's.
However, these are very, very deep networks.
You can see this is six hundred and thirty two million parameters.
These are actually not that deep at all, and they're quite they're performing quite well on 10.
However, they do use extra training data, so it's it's not cheating.
It's just a different way to put it in a different way.
We can get the best results on previous tests and another test data sets, I should say.
So don't feel bad with 61 percent, then to be honest, you can actually try and improve this and get
mebecause to 70 even 75 percent by a few doing a few tweaks and training for more ebooks.
So we'll stop there for now and then what we'll do will now go in to the some of these pre-trade networks
like Viji and Resonance and a few others, and start loading them.
And I'll show you how to load a pre-treated network and would pay to watch and us and how to just run
some inferences on new images and get the classes back up back out.
So it's quite a fun project we're going to do next.
So I'll see you in the next section.
Thank you.
Bye.