Hi and welcome to Section 36, where we take a look at colorizing black and white photos using a pre-trained
caffey model in open city.
This is a very cool project and I actually have experimented with this with some wolf only black and
white photos, and the results were quite good.
So this is what we can expect.
This picture illustrates what this model does.
You're taking a black and white image and accurately converting it to color, and you can see the results
here.
It looks really, really good.
No, this work was done by this.
What is here?
Let's open the research paper.
This was in 2016 Richard Zang, Philip Isola and Alexi Afroz.
These guys did it from Berkeley University.
Did a really, really good job at creating a colorization algorithm, deep learning algorithm.
So let's take a look at how it works.
So what the authors did, they embraced the underlying uncertainty of the problem, which is black and
white, to color confusion by posing the problem as a classification test.
So if they can understand that firstly, the butterfly, they reduced the amount of colors that it needs
to search.
Basically, that's available for the algorithm to converge on.
So that's already reduced their diversity of colors that for what this this picture could be.
And the system is implemented as a feedforward CNN, which is a standard CNN that streamed over a million
example images.
That's quite a lot of black and white color photos and evaluate the algorithm using the colorization
Turing tests, which basically acts as humans humans to take a look at the AI colorized image, as well
as the ground truth colorized image and let them pick to see which one they believe is the actual color
image as opposed to the air generated one.
And then method successfully fooled 32 percent of people, which is significantly higher than previous
methods.
So that was quite an achievement.
So now let's dive into the code and take a look and see how we implement this algorithm in open TV.
So what do we do?
Let's just dumb little images that we have done that we need to get for the project.
So let's run this block of code here, and it just downloads our colorization model since I've already
done it, and you need to type in way to just copy over a total of these image images.
So there we go.
So that should be done quickly and simply enough.
So what did that file contain that file contain or colorization images, which are the images we're
going to use here, some images of some famous images of some old popular people back in the past.
So we have Audrey Hepburn, Bruce Lee, Elvis, Jimi Hendrix, Marilyn Monroe and Queen Elizabeth.
And you can see these files here.
This is just a colorization notebook that we don't we're not using here with.
Code is in here.
This is a proto text file.
This belongs to the model.
This is the cafe model file, and these are the points in the whole which we which I'm not sure what
it's actually used for, exactly, but it's probably the kernel that is applied to the image.
So we just need to see if this point is probably like pre-processing Typekit.
So we just point to the fact that you get old, black and white images pointed a kernel here and then
well, in the main program here, we used to see v2.10 and read from cafe to lower the coffee model.
So that's the model that we pointed to here.
This is a proto text file and the cafe model.
You don't need to understand what the text file is right now.
That's just a model structure file, essentially.
But you look and you learn these things in the deep learning section of the course.
So don't worry about it.
No.
Read No.
This is more of an implementation cookbook type project that we're going to use that we're going to
do.
So we load a cluster sentence here, so we just load the kernel and then we populate the cluster centres
as a one by one convolutional kernel.
So these things here, these are things that are basically pre-processing steps that may confuse you
to definitely actually confuse me because every research paper has a lot of different things that it
that it what is do put together that is quite difficult for them, for a person who's not familiar with
the research to understand.
So you need to need to read the research paper quite in depth to understand what exactly is going on
sometimes.
So these are some lives that you're extracting here.
You can see two layers right here and then what we do for each image, you know, image about who load
that image, then we just run some pre-processing basically on the image.
So we normalize it and expect it to float to the two.
And then we convert that to the larger format, which is a different color space.
So we got the image as a lab and then we just pull out all channel.
Get to heights and widths here, resize it for the network, because CNN's tend to be trained on specific
sized images, we need to so they can only take inputs of a specific size.
So that's where we do that.
Therefore, then we resize image of the network, which we I was done right here.
We can video tribute to love.
So that's what this is here.
Completing this image to love and then we extract the Ilocano.
Here we subtract 50 four means centering, which again, is a pre processing step that depending on
if it's a mean, the dataset like a normalized dataset, you have to do that for any input, image and
future.
And then we just get the inputs at the inputs.
We just convert the image to a blob in one line here and we feed it into the function right here.
Then we do the forward pass, which you've seen previously with a yellow and SD models that we loaded
in open TV.
And then we get the shape we get to see about oate.
We pre process it to basically converted back into a BGR.
OK, that's the open TV's RGV type image format, and we show the original.
So let's run this code and see how well this performance in our images.
And note these images I pulled off the internet.
They aren't images that this dataset was trained on.
So let's take a look and see how it performs.
So this is Audrey Hepburn.
And you can see it looks quite good, in my opinion.
This is a black and white photo, and this is a color that gets a skin tone here.
Color probably, right?
It looks quite good.
Let's take a look at Elvis, and he looks quite good too as well with the color enhancements.
It looks very accurate.
Let's look at the famous look at the famous bristly.
Again, this looks amazing.
And take a look at Queen Elizabeth, and this one looks very accurate.
However, you can still tell it's not ideal.
It's not like a perfectly natural, color looking image.
And let's take a look at Marilyn Monroe, and this looks really, really well done as well.
And Jimi Hendrix, the if got gets a skin tone.
It's got everything very, very right.
So I'm quite impressed with how this performed on these images, and I hope you are impressed, too,
so you can take a look and experiment with your own images with this.
We're not going to move on to in painting, which is another method which is a method that we're going
to apply to all photos again.
However, this one is basically restoring damaged photos, erasing things like creases or missing spots
in the image.
We're going to use in painting to restore them.
So stay tuned for that lesson.