Hi and welcome back.
We're about to take a look at filter and class maximization.
So open notebook 10 here and we'll get started.
So to recap quickly, what we're going to take a look at is filter maximization and class maximization.
So just to basically remind you of what filter activation is, basically we're trying to find the input
that maximizes the output of that particular filter.
This is a way we can inspect our filters and see what they actually look for, as opposed to actually
visualizing them and seeing how they look, which doesn't tell us that much.
That's what we did in the previous lesson.
So to do this, what we have to do, we have to build a lost function that maximizes the value of given
filter so we can specify what filter we want in any convolutional layer.
And next, we will use stochastic gradient descent that technically ascent to adjust the values of the
input image so that it maximizes the filters activation value.
So as I said, we're going to we're going to use a different library for this.
We're using Keros.
However, we're using a library called T.f Dash Kiarostami's.
I call it Crispies for short.
And you can open this link if you want to check it out.
It's a very good library that has been used in the research community quite a bit.
It allows you to perform a number of types of activation maps, which we'll take a look at shortly.
We'll take a look at Cloud Cam and some of these here school cam as well.
And also, it allows you to do all these types of visualizations, so it's quite cool and quite useful.
So we're going to take a very short tour of it here.
So let's install Kerry's physical notebook.
So we just do pip install notice two exclamation mark here.
It's not needed if PIP is the only thing on the slide.
But if you have other like Python code, you do need to put the exclamation mark so that the club interpreter
knows you're trying to run bash here.
And this is what we do and to the Christmas package has been installed.
So now let's load those packages.
We're also loading TensorFlow as TF just we're not using actually cares.
We're using TensorFlow a bit in the back end.
However, it's a bit interchangeable, as you have seen before.
So what we're going to do?
We haven't done this before, but we're going to load a pre-trained network.
What that means is remember when we loaded our previous models like we trained in this model and we
loaded it into another notebook?
Well, that that's basically what we're doing here.
That's a proofreading model.
Free trade means it's been previously trained and TensorFlow actually comes with TensorFlow.
Keros comes with some pre-trained models, some advanced scenes that have been trained on the base computer
vision dataset that exist image net, which has well over a million images.
So it's pretty good.
It's pretty cool that we can actually load these networks and test them, which we'll take a look at
shortly as well.
We have adolescent's Lisa on where we actually do it just right here.
Sorry.
When we do look at some pre-trained the classic scenes like figure and resonated inception and a few
others as well.
So don't worry about it yet.
So what we're doing here, we're just looking this model.
It already knows the architecture because we're specifying the architecture and the model weights that
we want.
We want to image networks and include top means that we're including the full model.
If we if it would include top to be false, what that does is that it just loads everything except the
actually.
And that allows us to use this model for other applications.
But we'll get into that later on.
Don't I don't want to confuse you guys yet.
So let's load or pre-trained model.
It has to download the weights here and there we go.
It's quite fast and we've got this very big model.
You can see it's a hundred and 2.8 million parameters.
That is quite big.
Actually, it's one of the most models, even though these have less parameters, mainly because this
isn't a very well optimized model.
VGA has been known to be a bulky model, but it's a it's a simple model and it does most tasks quite
well.
So it's like a good old, reliable model that you can use.
So we've loaded it here, and you can see there's a number of convolutional layers here.
The 16 and VGA stands for 16 layers, even though you'll have the max pool ever confuse you.
But if you added up its 16 layers with about 12 lives, roughly 18.
So what we're going to do, we're going to take a look.
We're going to extract this layer here and because we want to visualize filters in that last layer,
because those would be the most interesting filters to visualize the room with the final ones, learn
the most advanced patterns and structure.
So.
You visualized the input that maximizes these filters.
You'll you'll see some cool patterns, you'll see what the CNN has been sort of learning to recognize.
So to do that, we just specify the learning here and we create a function called model modifier.
This takes the current model.
That's a model we looked at here.
And what we do, we specified only in name that we wanted to extract.
So we take Leonean here, give it to the input of this function called Cattleya, and we get our target
layer here.
Next, we create a new model.
This new model, we use a carrot stock model function where we specify the inputs to be the current
model, but inputs that the model we loaded here and the outputs to be the target layer, that's the
outcome of that model right there.
So it's a bit confusing so far.
I will admit I do get a model.
Researchers would sort of get confused sometimes when looking at this code because it's not nothing
straightforward and it's nothing that we commonly do as computer vision practitioners.
This is more of a research topic, but it's a quite good one because it helps you.
It reinforces how CNN's actually live.
So I think it's worthwhile to understand and learn it if you want to expand and extend your knowledge.
So lastly, what we do here?
We specify that we want the new model, that's a model we created up here.
All the layers, except the last one to have an activation using linear activations and cameras.
So like I said, this isn't as straightforward, but this is just how we do it.
So let's run this and create that function.
Next, we're going to create the activation maximization.
So to do that, Keros does.
This has a very cool package called both function call activation maximization that simply takes the
model.
The model modify the function, and we just specify we're not going to clone the model otherwise occupied.
Occupy extra space, no memory.
So we do that and then we go, so we have that next.
We're going to specify that we want to look at the seventh filter on the output.
So we have to create a lost function that basically just returns everything for that filter.
No, it's not a real lost function.
The gradient ascent, which is done in the activation maximization function here, that those direct
the rest of it there, and we just use the outputs from this fake lost function.
Now we're ready to visualize.
So I would scroll down and spoil the visualization.
Surprise for you, even though you might see it in your notebook and you can, but you already you already
have seen it, so it spoils.
It could spoil the surprise, but either way, I want to go to the court for you.
So we're just using some.
We're importing some callbacks here that this this callback or print is something that basically callback
is basically a function that operates during your training process.
And we can do things like extract different metrics we can do early, stopping many of other things.
We will talk about callbacks later on, and it's in this chapter.
So I should best mention what it is.
It's just a way.
It is just a way we can exercise functions during training if we want to do some more monitoring or
advanced early stopping or anything else like that.
So we're doing this one, this one called print, which is it just prints at intervals of 50, which
you will see below.
You can see the steps here 50, 100 150.
That's all it does here.
So we have an activation and basically we just can use that activation.
We use that activation here.
We converted to an umpire here so we can visualize it.
Next, we just these a subplot arguments are not going to go into this.
This is just for the rendering and visualization part of it.
We feed those in there and then we just plot basically.
So the maximization is the trading loop.
Effectively, if you want to call the trading loop, that's performing gradient descent so that we can
actually up to maximize that filter so it doesn't take that long to run, as you can see.
And there we go.
So this is this is the input image that would maximize this filter.
What this means, it's zoom in a bit.
This means here that this filter is looking for this pattern in the image.
It sort of looks like a snakeskin pattern, doesn't it?
Interesting.
Or maybe some sort of rock texture, but it's interesting that this filter is looking for that, and
that is the input image that maximizes that filters output.
So that's pretty cool.
Now let's take a look at some random filter numbers.
So let's do tree filters here.
So we just specify tree random numbers.
You can choose different numbers up to five hundred and twelve.
Since there are only 512 filters in this last layer, and I see it only mainly, mainly because there's
actually quite a bit 512 of this.
So we convert, we create our lost functions just between still, but like we did before, but now it's
for these tree filters.
Next, we just we create seed input because for multiple visualization, visualizing multiple convolution
look filters, it requires.
This package requires us to set the seed value seed input value.
So we just creates a random image here.
That's what this dimension run the values using this test at random uniform function, and that creates
our seed input.
Next, we can sort of visualize.
So I'm not going to go into this code in detail.
It's just basically the same code we saw before.
But no, it's a full loop that does the iterations for each.
Filter.
So let's go.
This will take a bit longer to generate, so we'll kick, kick back and relax and wait for it's should
take about a minute maximum.
Maybe less.
There we go.
So that actually was quite fast and we can see these are the inputs that maximized the other filters.
I honestly, I'm not sure what I'm looking at.
This looks like something from like a coral reef or an alien planet.
This one looks like I do not know.
I wouldn't hazard to guess.
Looks like a frog to me, but this one is this weird pattern almost like, I don't know who knows,
but what I thought was actually looking for.
But that's what it is.
So next, we'll take a look at class maximization.
So task maximization is actually quite cool.
It's one of my favorite types of visualizations to do class maximization.
Instead of looking at individual filters that we that input maximizes, we are not looking at what input
image would maximize the network and seeing that it's this belongs to this specific class.
So to take a look at how we do that, we just basically this is starting from scratch again.
So we look at all the packages if you want.
It's already loaded previously, but it's the code is here from the default, so you can just run it
straight from class maximization if you wanted.
So we load the pre-trained model individually.
16.
So let's do the math here and we get our model loaded.
We have to define the same model, modify a function.
We also have to define our activation maximization or create that object.
Next, we have to create a loss function here.
So notice I have no T here.
This this is specific to a class.
So notice when we loaded in the network, we didn't.
We included a top layer here.
True, that means that the soft max output.
So now we can actually specify the output here to what class we want it to be because we have the number
of nodes in the image and data set.
So you can see the image net classes here.
If you want to try different numbers of combinations, there is a thousand classes and it's quite there's
quite a lot of things going on here.
So a lot of animals and a lot of common household objects, although objects as well.
So it's quite extensive, to be fair.
So let's create all those function for that specific class.
And now we can run on maximization.
So it's very similar to what we did before we have our activation maximization function input.
So lost function intervals here, every possible support arguments, we get our activation out of it
and convert it into an image that we can visualize with MATLAB.
And let's run that, and let's take a look at what, by the way, this this task to you was a believe.
That's a boot, if I remember correctly.
Yes, it is a bid, and you can see it's a type of boot.
This is the actual bid that it was supposed, it looks like.
But think about this.
This is the image that maximizes the CNN output for that specific class.
It doesn't actually look like that, but it just looks like variations of a bit on it, on an image.
So this sort of gives you an indication that CNN's are living differently to how over human brains learn
about what's in images so we can do some more classes here.
We're going to do one school for the goldfish to beer and also a assault rifle.
These are the class numbers fits us one, twenty four and 14.
You can double check that here and see what the Class One was goldfish.
That's correct.
So let's run that create our seed input because it's multiple field of multiple visualizations we're
doing, and now we can visualize using that for a loop.
And now we wait shouldn't take too long.
All right, there we go.
So let's take a look at what our CNN, what a big CNN thinks a goldfish looks like.
That doesn't look like a goldfish to me, but maybe it is.
Maybe it's an amalgamation of different goldfish type objects or different goldfish is maybe two different
kinds.
Who knows?
But it does.
I mean, it does look a little more like a wheel to me, but it can kind of see some of the goldfish
features, maybe two fins a bit.
Now this is a beer, and you can see this is what you see in in things the ideal beer looks like, which
is basically I've seen many beers here.
How many beers do you see?
It's quite weird.
And the assault rifle, you can definitely see outlines of a gun, but basically, this is what the
CNN thinks the perfect assault rifle sort of looks like.
So that's it for this lesson.
I hope you enjoyed it because I think it's quite cool to do this and you can easily adapt discovered
that we've seen here, too.
We just learned any model that you've loaded that you've trained before, and you can start visualizing
lives of class maximization from it.
You can look at a filter maximization.
So I think it'd be quite useful, especially in research, especially like in Masters projects of undergrad
projects.
We want to go a bit extra and show that your professor or supervisors, that you've done a little bit
more research and understanding.
I think this would be quite useful for that.
So we'll stop there and will now move on to the grad cam visualization, and we'll also look at Grad
Cam Plus and faster.
What was it called again?
It's actually listed here first, the school camp.
So we'll stop there and I'll see you in the next lesson.
Thank you.