Hi, guys, and welcome back to the course in this section, we'll take a look at image captioning,
which is a very cool project.
I'll actually show you what it does quite shortly, so let's open this notebook and we can take a look
and get started.
So what is the image captioning?
Well, imagine we have a picture like this and we feed it to an algorithm that tells us what's happening
into this in this picture.
That's what captioning is basically a description of what's happening in this picture.
And you can see in this picture, I would say it's a brown dog jumping over a hurdle in something that
looks like an obstacle course.
And this is what our predicted tag actually says.
The dog jumped over a hurdle, which is quite accurate.
So we're going to create a model that implements image captioning using a CNN and a transformer.
And the reason we need a transformer is because this project is a combination of an old techniques as
well as computer vision techniques.
So let's get started.
So firstly, run this block of code.
This imports the fanciful functions we'll be using and notice here, and then we're importing text factorization.
That's how we include the words.
And speaking of words, how is this set?
How do we use data for this type of task?
Well, we used a Flickr ET key dataset, which, as the name implies, has 8000 images, and each image
has five different levels that human annotators attributed to the image.
So this one image has these five different descriptions.
A child is in a pink dress coming up a set of stairs in an entryway.
A girl going into a wooden building.
Wooden Playhouse Playhouse wooden cabin.
You can see that they're all basically in play.
A girl or little girl.
Some describe her outfit.
Some don't.
But it will describe that she's going into a room or wooden wooden cabinet playhouse or something along
those lines.
So this data set is quite big, so downloaded here.
It may take about a minute roughly and then to set some defaults here.
So we have an image sized vocabulary size that we'll be using the six sequence length, which is a max
of 25 actually fixed length 225.
The embedding dimension for the image embeddings because we're using efficient net network in this case
and some other parameters as well for the training purposes.
So now we need to create some functions here that lower the captions from the dataset.
So that's what this load captions data does.
You can see it returns a caption mapping as well as text data and caption map mapping is actually basically
a dictionary mapping image names to the corresponding captions and then text it that contains all the
viewable captions.
So it's basically a function that does that because we would be using that.
This does text labels as our input into the into creating an algorithm or model, I should say.
So then this is a function that creates the train vowel split as well.
And we just created data sets.
Here you can see training samples 61 14, 15 29 for validation.
That's all fine.
Now we need to vector raised to text, and vector raising text basically means taking the words sort
of in the strings and creating an integer sequence where each integer represents what that word is in
the vocabulary.
So remember, we have 10000 max of 10000 vocabulary words.
So each word, each particular word, I would say the word frog is encoded with a value of C 16.
Every time frog is mentioned in label, it corresponds to number 16.
That's effectively what this factorization is.
I mean, factorization does remove a lot of stop words, a lot of other things.
So you can look it up if you want to learn more about factorization.
This isn't an LP course, but it's always good knowledge.
It's a deep learning practitioner to have some basic A.P LP stands for natural language processing.
Maybe I should have said at the beginning, but that's what it is.
And we also defined image augmentation steps that we would be taking as well.
Consider these are the characters we remove as well, and everything is sent as lowercase.
Now let's start building the data pipeline for loading all of this data, so we have to decode and resize
function.
We have a read and treat image.
Read a validation image.
Make dataset here, which uses Tell us about functions and then everything here.
Well, sorry, this one does that.
And then everything creates the list of images that correspond to the captions for the training dataset,
as well as a validation dataset.
So a lot of data preparation work for this model.
Now let's talk about the model or the models involved.
So firstly, we use a.
CNN to extract the image features next.
We passed whose image features to something called a transformer encoder that takes the extracted features
and passes it to a transformer, which includes a representation of those inputs.
Next, we have the transformer decoder.
That model basically learns to map the inputs to the caption itself.
So that's effectively what we're building here.
So this is just the CNN mall where you can see it's a simple, efficient net model that we're using
to get the features out of it.
So we just build this model quickly here.
Now, the Transformer and A is a bit more involved and you can see it has the self attention modules
with the multi attention header that we create here.
And then there's also the positional embedding, which has typically leads to the sequence.
And then there's the transformer decoder block.
So the transformers are different type of network.
We haven't touched on that much in discourse, except a bit individual transformer section.
But you can look it up if you want to learn more about it.
It's actually a very important part of the planning right now, especially in natural language processing.
So that's it for those networks that we defined and we have to get because, sorry, casual attention
mask as well.
And then we have the class which creates the image captioning model, which basically just creates everything
that we said.
There were two losses that we need to involve.
So that way, we can start trimming the network in this from this class, we you have the training step
here where we step two and we do back propagation.
You know, all of this already is this encoded here as the gradient tape?
So you could actually do it like how you do it and PyTorch effectively and then the test step as well
and then the Matrix as well.
So you can see we get the model here, we get the encoder with candy decoder, we get the caption model,
which generates text labels, and that's basically everything here.
Consider Encoder Decoder and CNN Mobile.
Next, we need to train the model, which is the big part which takes quite a while.
Actually, it might take a movie about a week or two, perhaps maybe longer.
I'm not sure, but you or you're using.
So we just set up language schedule and then we start training the network and see we train for two
epochs, and each book takes about two minutes, and that's roughly an hour it's going to take.
And no, once you're finished, you can actually check some sample predictions.
So let's generate some random samples using this to generate caption function that takes a random choice
from the validation data sets and predicts using the caption model to the image.
Sorry.
Over here you actually get the coded caption here and then then you could print out the decoded caption
at the end so you can see.
Take a look at this one.
Some dogs in the dog hurdle race and you can see a group of dog exercer on a sled team.
There's a guy in a kayak or boat paddling a paddle boat, and it's a dog in some red tube.
A black and white dog is running through a grassy area, so you can see this is looking pretty good.
Let's take a look at a couple of more Trimmel, actually.
So the first one is looks like a group of men talking a man in a white shirt and a man in a blue shirt,
although that's wrong, although this is blue, but I wouldn't say it's that blue is standing in front
of a crowd of people.
Not not.
Yeah, that's right.
It's not the women.
Maybe what a human might say, but it's close.
This one, a man and woman ride bikes on the sidewalk.
That's pretty much self-explanatory.
It's a very good caption that our model is predicting as well.
And this one avoid a skateboard, a boy in a black suit and a boy in a yellow shirt is jumping on a
skateboard, and I only see one boy.
So this one's clearly a mistake.
But that's OK.
Because, as you can see, in the end, when training, we got 41 percent accuracy, which isn't to
create.
But this is a very good start, though, because you can see using deep learning to predict captions
is doing very, very well.
Actually, in my opinion, this is something that maybe 10 years ago and no one would have thought was
possible easily.
So there we have it.
So that's it for the image captioning lesson.
I'll stop there for now, and I'll see you in the next lesson.
Thank you for watching.