Hi there and welcome to this new and exciting session in which we shall be treated, or we shall be
using this transformer network right here to solve problems in computer vision and more specifically
in the task of image classification.
Up until this point, we have seen different convolutional neural networks like the Loonette, the VG,
the rest nets, the mobile nets, the efficient net.
And now we'll be looking at the vision transformers.
This vision transformers were first developed in this paper entitled and Image is what, 16 by 16 Words
Where to Build Transformers for Image Recognition at Scale.
In this section, we'll take a deep dive into how this whole architecture has been constructed and how
it works, and also how and why Transformers perform as well as their convolutional neural network counterparts.
The very first point you want to note here is the usage of Transformers for computer vision tasks has
been developed in very recent times.
You could see here from this date this paper was published.
The authors hear that while the transformer architecture has become the de facto standard for natural
language processing tasks, its application to computer vision remains limited.
In vision, attention is either applied in conjunction with the convolutional networks or used to replace
certain components of convolutional networks while keeping the overall structure in place.
We should add this reliance on the convolutional neural networks is not necessary, and a pure transformer
that's we doubt any convolutional neural networks apply directly to sequences of image patches can perform
very well on image classification text.
They even go ahead to tell us that when pre-trained on large amounts of data and transfer it to multiple
mid sized or small image recognition benchmarks like the image net CI for the white that is a vision
transformer attains excellent results compared to the state of the art conf nets like the efficient
net while requiring substantially fewer computational resources to train.
Now, it's possible that you've never heard of this term transformer or maybe you from an electrical
engineering background and you've only heard of this when it comes to stepping up and stepping down
electric power.
Now we are going to go straight away to explain terms like the transformer or even this attention which
was mentioned to better understand the transformer and the role it has to play in this white architecture
right here.
We would have to get back in time to understand why do it first.
Developed in 2017.
This paper entitled Attention is All You Need was first developed by one year out, and it has turned
out to be one of the most influential papers in the modern deep learning era.
With the development of this transformer architecture right here at the heart of this transformer architecture,
we have this self attention modules and more specifically in this paper, they used the up product attention
that we could see here.
But then, as we said, the whole purpose or the domain in which these kinds of architectures or those
kinds of networks were build was for natural language processing.
But the question is, how does this work in natural language processing to understand how and why the
attention and also the transformers are used in natural language processing will take the following
example, which is that of translation, which we're used to already doing and with Google Translate.
So here we're going to put in I love the weather or could you see the weather today?
It's amazing.
And we translate this to French Luton.
Should we inquire now initially the kinds of deep learning techniques which were used in solving these
kinds of problems, that is taking us from one language to another.
Where the recurrent neural networks, the way the recurrent neural networks work is quite simple.
So we'll start by putting the text here.
Yeah, we've put out our example from Google Translate, and then we've added this extra blocks right
here.
Now, the blocks we see here are recurrent neural network blocks, recurrent neural networks generally
reading.
Ah, and hence you're one of the first deep learning based models used in natural language processing
texts like the Case of Translation we have here.
Now, the way this works is we have our initial text or we have our source.
That's English text.
The weather today is amazing.
And then we have the target, which we want to generate.
So initially we have this input and this output which we're going to train, and later on when we pass
in some random input, we expect to get a reasonable output.
Now, the way we have this or the the way this is structured is such that each and every one of these
is called a token.
So we have this word here, which is a token this, this offers token next token, this token, this
token and this other token then this different tokens have been converted into vectors and then being
passed in this hour and blocks right here.
Now we carry out some simple computations like multiplication and addition, and then some information
is being passed in from one block to another, hence the term the recurrent neural network.
Now, the importance of passing information from one block to another is that this tokens computations
in this block will depend on this other previous tokens that you could see here.
So it depends on this, depends on this, and also depends on this other one.
And then once we're done with convert or passing this information from one block to another up to this,
we are then going to take this year, we're going to have some information.
We're going to be passed onto this other random block here.
So this is our encoded block.
So we encode the information and then, yeah, we decode this information.
So here we have the encoder and the decoder.
And then again, your a similar process is repeated where we have this computations which produce an
output here.
And then we could take this output and feed it in this one to produce this other output and so on and
so forth up to this final output.
But then the problem with this technique or with this method is that first of all, if we have a very
long text, then it may happen that it starts becoming difficult for information to flow from this first
blocks here to this final blocks.
And given that even as humans, we know the importance of taking into consideration some previous context
when trying to carry out a text like, for example, translation, this kind of problem will lead to
very poor results.
Now, another problem here is each time we train, we have to pass this information from one block to
another sequentially.
So here we pass all this information sequentially.
And because this information is passed sequentially, it makes it difficult for us to implement parallel
ization very efficiently.
And so this makes the training of these kinds of neural networks very difficult.
Now, to tackle the issue with long term dependencies, attention networks were developed.
So right here, instead of depending on just this final vector here, we get all this final output we
get from this hidden layer here, which has been passed on your to relay this information from the source
to the target language.
What we'll do is for each and every unit we have here, each and every recurrent neural network block
we have here, we are going to take into consideration inputs from each and every block here.
So this inputs will be taken into consideration.
So each and every block now you see all of this is passed and then we have this attention layer right
here, which then processes this inputs from this different source random blocks such that the layer
that all this attention layer produces an output vector which is now passed as input into this random
block.
And so when we have this source and this target, we pass in the source, then we get or we combine
those inputs from each and every random block right here, pass in this as input into this random block,
get an output.
In this case it's low.
Then we take this output and pass it as an input in your.
But again, once we shift and go and get to this time where all this time frame where we want to get
this second output, what we'll do is we'll have another tension here, which again takes in all these
different inputs.
So we take again all this different inputs year and year and year and year, then carry out some computations
based on the type of attention we are implementing.
And then from here we get an output which is passed together with this right here.
So from here we get this output and then we repeat the same step that's passing in the tone that is
taking this output passing in here, and then also taking in this inputs from those different random
blocks.
And so as you can see here for each and every block we have here, it pays attention to each and every
input.
And for this or from this, we could even come up with an attention map where we would have this text,
this text in English.
The weather today is amazing.
And then this other side we have Luton.
Aujourd'hui enquire.
So now after training this kind of model, we can be able to see how much attention this little piece
to each and every input here.
And it's logical that this layer will pay the most attention to the and then turn to actually years.
Whether so this would pay more attention to our most attention to weather then we will pay most attention
to today which is today a will pay most attention to is and enquire will pay most attention to amazing.
And if we get to this paper entitled Neural Machine Translation by jointly learning to align and translate
that is a famous battle.
Now our paper, you can see some of this attention maps here.
Let's have this.
You see some of this attention maps.
There you see uh la la course LA zone Économique European ET see are not and off and do and you have
an end then the agreement on the European Economic Area was signed in August 1992.
So you see this attention maps here where we see clearly which words attend most to one another.
So yeah we have this image which shows exactly what we're describing previously.
So here you have this inputs and then to get this output y of RT, you will find that we are going to
take in or we are going to attend to each and every input here and then pass this here to obtain our
y dt.
Now at this point we are going to move on from the attention to self attention and to better explain
the self attention will consider a whole different problem, which is that of sentiment analysis.
So you want to we have this model.
We could not take this off.
We don't make use of this, although you should note that we still use center or self attention in the
translation problems, but it will be easier to grasp this concept in the context of sentiment analysis.
So what we are having is we we have the weather to this amazing.
I want to be able to say whether this is a positive or negative statement.
So now we have this model which takes in inputs like this, and then let's draw this model here.
Like this.
We have this model and then outputs or tells us whether the statement we've made is a positive or a
negative statement.
Now you're all for this self attention or layer.
We are not going to need this recurrent neural network hidden states anymore.
In fact, what we could do is we could take all this off actually, because basically we have in this
self attention model, which we'll see in a minute.
How it works.
And then what we're passing in here is some vectors.
So we have this vector, we have this other vector, we have this vector, this one, and finally this
one.
Now, if we combine all this, we'll find out we have a sequence length.
So we have one, two, three, four, five.
Suppose our second line is five.
So we have a sequence length by, let's say, embedding dimension matrix, which we wish we get from
here.
Now let me explain.
Let's suppose that the sequence length is five, as we've all seen, and then the embedding dimension
is let's say three.
So we have this five by three matrix, which we are going to pass into this self attention layer right
here.
Now this embeds all these vectors which we pass into the self attention unit are going to be designed
in a way that words which look alike are going to be close to each other, while words which are opposites
are going to be far away from each other.
Now let's, since we're working in three dimensions, it means we have one, two, three values.
Your one, two, three.
And then finally, here's one, two, three.
So let's.
Let's do something like this.
Three dimensions.
What we'll have is the word happy.
Which in this case can be represented by this vector or this embedding will be or can be plotted out
like this.
And this will be close to a word like smile while a word like sad or like sad will be far away from
this two words because the actually opposites to each other.
So we have sad and we could have angry right here.
Now for this one year or for this text year, we could pick out this two words which are most likely
to be very close to each other.
We could have the right ear and we have is somewhere around here.
And so now getting back to this model, we have this five by three input which is passed into our self
attention layer.
So we could let's, let's have this matrix here.
Five by three would have the, the what the here would have its own embedding.
So we will have some value, some value, some value.
Let's suppose that we're working in three dimensional embedding and then whether we'll have its own
value.
It's on value.
It's on value today, it's on value.
This value, this value could you could take, say, 2.310.5 negative five one whatever value one year.
And then you have this and you have this and you have this.
This is four already and then amazing would have its own.
So you see that each and every one of this year has its own embedding.
So this is this are the different words we have here.
Then at this point, we'll implement a special type of attention known as a dot product attention,
where we'll take this year and multiply it by the transpose of a matrix which has the same shape at
this matrix here.
So we'll take this, we'll call this the query, and then we'll multiply this by the transpose of the
key.
Now this key is going to be three by five since is going to have the same shape as this query.
Now this is our query.
We'll call this a query.
And so here now we have this three by five matrix, and then this product will give us a five by five
matrix.
Now, after this hour, after getting this five by five matrix, we could pass this to a soft max layer.
Now we've looked at the soft max layer in previous sessions, but one thing you should note here is
once we have this five by five matrix, it produces this attention map similar to what we are seeing
before where we have this.
The weather today is amazing to the site and the same again to the site.
And then words which are most similar to each other in a certain context are going to have the highest
values.
And so if we're in the case where you had say, let's replace this weather by happy and then we have
amazing let's let's let's leave Amazon.
So if we have the happy today is amazing it still doesn't make sense.
But let's consider this let's suppose that we have the happy to this amazing then this second row of
foot column because amazing would be around here so we would have this value which is going to be relatively
higher than all the other surrounding values.
And this will be because after training the model, the attention map values would have been modified
such that values or rather words which are similar to one another, take higher values, while words
which are not similar to one another.
Take very small values.
Now from here we have this five by five matrix, which now where multiply by another five by three matrix
will give us a five by three matrix.
Generally we call this matrix, which is multiplied by this attention matrix, the value.
So we have query, we have the key and we have the value.
With this you see that we have this input which got in here, which was five by three, and now we have
a five by three output.
Then this year now we pass through some fully connected layers and then we'll have an output or a fully
connected layer with one neurons output, which will tell us whether an input statement is a positive
statement or a negative statement.
And so as you've seen, we've gotten rid completely of the recurring neural network blocks as now we're
just making use of this self attention blocks to extract information from our input.
Now one of the first papers, if not the first paper, which made use of just the attention and getting
rid of the rans was this attention is all you need paper.
And it happens to be one of the most influential papers in modern day deep learning.
So here in this attention is all you need the paper or the transform our paper to present this new network,
which you could see just right here and then a single block.
Let's take this off a single block, which makes up the transformer model is this multi head attention.
So as you could see right here, we have this single block and then here we have this multi head attention.
So let's look at this multi higher attention.
This is actually the motor head attention here.
So you have this year, which is this whole block.
And then in this multi head attention, you have this killer product attention, which is this self
attention we just talked about.
You see we have the query, the key and the value.
So since it's self attention, you will notice here that we have those inputs and all these come from
the same input.
So we have this input which is split it up into cure K and v query key and value.
Now this resembles or is analogous to data management systems where data is stored in key value pairs,
just like say, Python dictionary.
So you have data in this key value pairs data stored this way.
And then when you want a particular information, you have to pass in a query.
Now, when you pass the query, let's change this color.
When you pass in a query, you have a particular key which is selected.
Once the key is selected, we now obtain the value, which is the data itself.
And it's kind of similar to what we have in here.
And then from you're in this level of the split, not that before the information has been passed into
this skill dot product attention, we actually pass this K and V into some different linear layers.
And so this means that even though we have the same inputs that will end up being projected into three
different inputs.
And so now we have this k v, we are going to carry out Keoki Transpose.
Here we have the mad mol as we saw already cure times K transpose and then we have this scaling which
you can see right here in this formula, this attention formula.
We have DT divided by this d k Then from here we have sof max.
Of all this and then we multiply by V.
So let's get back up to this year.
That's fine.
Now that we have this output, you now see that we have this multi head.
So we, we got this, we have the soft max, we have the mama where we take this self max of this multiplied
by V.
So that's how we get this recall, how we saw that with the example we had previously.
And then we have this multi head attention.
Now this multi head attention here simply means you take this year as you pass in your information like
this, you get this care K and V, and then you again pass the same information into this block.
So let's suppose that this is our skilled put attention.
BLOCK This is called product attention, which is right inside your.
And so once we have this, let's let's make that smaller.
Let's suppose that this is what we have here.
So this year is actually this.
Now to obtain the multi head attention will have this other one here and then we'll have let's suppose
that we have three heads.
If we have three heads that we would have three of this stacked in this way, you have one, two, let's
change the color so it becomes clearer.
We have this one in red, we have this next one in blue here, and then we have this other one in green.
So there we go.
We have this three and then when the information gets in, so you have your cure, you have your K and
you have your V, We pass this to this separate linear layers C for for each of this, we have some
linear layer here.
All of this came from the same inputs as you could see here.
And then now this information is passed.
So we have concave paths into this one, into this block here and then.
This same shockwave also is passed into.
Let's change the color.
We will now have some other year, some in your lyrics.
Let's put it besides this.
We have some other linear layers here.
We'll pass V we will pass, K will pass here, right here.
And then this now will be sent into this self attention block right here.
Then we also finally have this for the red.
So we'll have something like this red.
We have the K, something like this, we have the V, something like this.
So now this V is passed now into this red here and that's it.
And then the outputs here, the outputs will get at the end of this three self attention blocks will
now be concatenated and then pass through a linear layer.
So this linear layer is like our dense layer in TensorFlow.
Now once we have this, you see we have our multi head attention, which is this block, and then now
we'll take this input added on to the output and then go through a layer normalization.
Then from here we pass this to a feedforward network that's like our fully connected network or dense
layer.
And then we will again repeat this addition and normalization looks similar to what we have with the
rest nets.
Now, once we have this now, we could then repeat this end times.
Now you'll notice that this is similar to this, except for the fact that now we have this to multi
head attentions and we also have this mask.
Anyway, we're not going to get into all this details.
What's important for you to understand how this encoder works and now we understand how this works.
We will not get back to our paper.
That is this paper entitled Transformers for Image Recognition at Scale.
And now you should be able to understand this transformer block, which we presented earlier in this
paper with this understanding of how this transformer encoder works.
Let's now get into this unit here where we break this image into this different patches, as we could
see right here, to better understand how and why we make use of patches right here.
Let's not forget that what this transformer encoder takes in is some input sequence.
So we have this input here.
Initially, we had words where each word like this could be represented by this vector or this embedding
vector.
And then this now combined is passed into the transformer here.
Since our input is this image, in order for us to represent this way, we'll have to break this up.
So what we could do or what we could think of at first sight is we have this image.
Let's suppose the image is 256 by 256 by say, three three channels.
Then we could take each and every pixel here.
So let's, let's omit the channel from the channels for now.
So what we could have here is for each and every pixel in this 256 by 256 image we would have a vector
representing that pixel.
And then this other one is vector, this other one is vector, and so on and so forth.
But don't forget that unlike previously where we had only five words, now we have 256 times 256 words.
Because if we have an image like this and we have to get each and every pixel that will have 256 by
256, which is more than 65,000 different
vectors, which we will have to pass here.
And so before where in our attention model we had an attention map which was five by five recall, we
saw that already with the words we had a five or an input sentence with five words we had five, five,
five attention map.
Now we would have a 65,000 by 65,000 attention map.
You see that working with these kinds of matrices and memory isn't very feasible.
And so instead of going pixel by pixel, the authors decide to work patch by patch, let's create this
again.
So you get to see that.
Take this off.
You see, here we go, patch by patch.
So you could see how this image, instead of taking each pixel, we break this up into patches.
So this is now like a pixel.
And then you see this patch, you see this patch, this patch, this other patch, this patch, and
so on and so forth.
Up to this patch right here.
Now, this is what is like the word.
Know your SO with images, we have to break this up like this.
And the authors choose to work with 16 by 16 pixel patches.
So each picture is 16 by 16.
And so given that we have 16 by 16, if we have this patch, for example, then we would have 256 different
pixels for each patch.
Here we have 256, here we have 256 and so on and so forth.
So all the images are at our on line with the words where we had five by three, so we had five words
and each word was represented by a three dimensional vector here, each patches represented by 256 dimensional
vector.
Now, this doesn't mean that in NLP we generally work with this.
This was just done to make it easier for you to understand.
So getting back to computer vision, you see that we have this 256 to 50 6 to 56 and so on and so forth.
Now, when working with the transformer, we may not want to work with this 256 dimensional vectors.
Maybe we want to work with, say, 512 dimensional vectors.
In that case, we would have to do this linear projection of the flattened patches such that we leave
from this.
Say, let's suppose that we have one, two, three, we have nine patches, so the sequence length is
nine.
So we have this input which is nine by two, five, six.
And then after going through this linear projection, we now get to nine by 512 and this will be the
embedded dimension for our transformer.
In the previous example, our embedding dimension was three.
So if this this permits us to be to, to work flexibly as now we could decide on what size we want for
our embedding dimension.
Now that said, we have this output C nine by 512 and then we're ready to pass this into the transformer
encoder.
But just before passing this, we'll add this position embeddings.
You see there we have this input you see in this different this color, you have them getting in and
then we have this position embeddings here is notice zero one, two, three and up to nine.
Now the way this works are let's start by first.
The reason why we even have to do this is because unlike with the conflicts where where the convolutional
or the way the convolutional neural networks work is that for computing the feature maps, it takes
into consideration locality.
So this means that you see these two portions here or when passed with a conf filter will produce a
certain output.
And so this means that pixels which belong to a certain or to a small locality like this one will be
used to produce the output.
And this clearly gives an gives the scenes an upper hand over the transformers as when trying to understand
an image, the positions of particular pixels actually matter.
So this means that CNN's already have an inductive bias due to the way they actually work.
And so to give a help in hand to the transformer network will now need or will need this position embedding,
which gives this transformer encoder an idea of the location of each and every patch which is passed
in.
But again, it should be noted that this will have to be learned automatically by the model.
Now, even though this we have this extra input right here, and the reason why we have this extra input
is simply because we do not want a situation where after going through this encoder, all this transformer
encoder right here, we pick one of this outputs because we will have outputs here.
We don't want we don't want to pick one of these outputs to be used for the MLPs head or to be used
for this fully connected network in this classification unit right here.
So to avoid this sort of bias where we would be picking one of this, the others add this extra learnable
class embedding right here, which will be or whose output will be passed into this MLPs head, and
then we'll be used for classification.
Another important point to note here is the transformer encoder or this or visual vision.
Transformers are some sort of hybrid architecture.
The.
Because we may decide not to pass on this image patches directly, but instead passes image patches
to a convolutional neural network, then get the output embeddings and pass in your directly instead
of this image patches.
It should be noted that the multilayer perceptron contains two fully connected layers with a glue non
linearity yours the general glue nonlinearity.
Compared to the rainbow and the ALU.
So you see we have this realm where all values less than zero or less than zero gives output of zero
and all values greater than zero give the exact same value.
But with the glue, we have this curved function right here.
So that's it.
The type of normalization is the layer normalization, as we mentioned already, and the layer normalization.
Here we could visualize this in this paper by sharing a all entitled power norm, rethinking batch normalization
and transformers, where you see we let's zoom this.
You see, we have layer normalization here and we have the batch normalization put side by side.
So with the layer normalization, that's what we're saying.
If you consider some inputs, let's let's yeah, we have the sequence length or the sequence dimension,
we have the features or the embeddings or like a vector actually.
So we have the different vectors and then we have the batch dimension.
So basically what we're saying is we have this sequence length or we have this different vectors here
which have been passed into some layer.
And then instead of doing or carrying out normalization for, for throughout the batches, as is in
the case of the batch norm here, we carry out this normalization for each and every vector.
And the reason why we do not use the batch norm with the transformers is the fact that the batch statistics
for NLP data have a very large variance throughout training, and this variance exists in the corresponding
ingredients as well.
And so to avoid this kind of situation, it's preferable for us to carry out this normalization on the
features.
Instead.
Before we move on to the experiments, let's look at how the weights are being used in real world.
So actually the weights of are pre-trained on very large data sets and fine tuned to smaller downstream
tasks.
Obviously, when fine tuning we remove this head and replace with a head which now correspond to our
number of classes.
So this means that initially we may have 1000 class head and then we move those to K classes or let's
say three class head.
To better understand why we're going from why we have a D by K output.
Let's get back here.
So after those inputs have been passed in here we have an output sequence length plus one this plus
this one year.
Let's just say we have.
Yeah, we have say from year one by the output if, if we're considering all the sequence length will
be a sequence length by the output.
This here is our embedding dimension which we had fixed from this linear projection right here.
So we have this one by DX and then we pass this through.
Obviously it becomes, it becomes like simply the neurons.
So we have one two, we now have the neurons since it's just one by DX and then we have this output.
Let's say we have 1000 classes, then we'll have this fully connected layer which brings all this year,
this the inputs to this K outputs, or in this case to this 1000 outputs.
Now when we want to fine tune one, we want to fine tune, we are going to take this off, take this
off and replace this now with K outputs.
So we now have K outputs right here and then we initialize this weights of this fully connected layer.
The others also make mention of the fact that doing fine tuning is better to work at higher resolutions.
So this means that the model could be trained at 256 by 256 and then later on fine tuned with 512 by
512 images.
And then since they keep the patch size the same, that results in a larger effective sequence length.
Now let's explain all let's visualize this statement.
So here we have this input, which is say 48 by 48.
Let's say we have your 48 by 48.
And when we divide this or we break this up into three parts, we have 16, 16, 16, 16, six and 16.
So we have 16 by 16 patches.
Now, if we want to fine tune on a higher resolution image, then let's say the higher resolution images
say 96 by 96, so we could have something like this.
So if now we're finding on the 96 by nine six image and that we still maintain the fact that this year
or the patches will be 16 by 16, then this means that instead of three here we're going to have six.
So we now have or one, two, three, four, five and six patches, six patches this way two, three,
four, five, six and so on and so forth.
So now we're going to have 36 different patches instead of nine patches as we have here.
And that's why they make mention of the fact that the sequence length is going to be increased.
And that's so long as they can fit in the memory.
Now due to this modifications, the pre trained that's what we had before the pre trained position embeddings
may no longer be meaningful.
So they therefore perform two D interpolation of the pre trained position and variance according to
the location in the original image.
The experiments here we could see those different models.
They have the V base, the V latch and the V huge number of parameters at 6 million to 632 million.
Then yeah, we have 12 layers.
Recall, let's get back here.
Recall we had this number of layers here.
So basically you're repeating this, you're repeating this here 12 times.
So we get back.
So we have 12 layers for the base as stated here, and then we have 24 for the latch and 32 for with
huge.
Then this hidden size, this D this embedded dimension is 768 for base 1024 for large and 1284 huge.
The MLPs size that's fully connected layers to 3072 4000 960 5120.
Then the number of heads remember the attention heads 12 1616 So this MLPs size here, they're talking
about this MLPs recall.
We have we have this MLP, we have this MLPs right here.
And so this MLP is made of two fully connected layers like this.
Then depending on the size, you have a certain number of neurons.
Their experiments were carried out on this GFT 300 million data set and we see how this 14 by 14 patch
version of the white outperforms this rest net 152.
Now this performs once, although not largely greater than that of the rest.
Nets requires less competition resources to train.
As we see here we have 2500 CPU core days required to train this model as compared to this one, which
requires 9900 CPU car days.
Also from these plots, you see that when you increase the number of pre training samples, the model
which performs the best is this white right here.
So here we have this fit and this outperforms the rest nets, whereas for a reduced number of samples,
the rest nets outperforms the whites.
While here the smaller the patch size like here we have this 1414 by 14 pitch size we have the better
the results.
Now, in order to understand the reason why, as you increase this data set size, the widths start
to outperform the coordinates.
We have to recall that when working with confidence like the rest net, there is some inductive bias
in the sense that the fact that this rest net takes as input, this two dimensional image already gives
this conflict a helping hand when it comes to extracting features from your.
And so even with relatively smaller data sets, this confidence can make sense out of this input image.
Now, with the Transformers, which are some sort of generic neural network, the model doesn't get
to see the image in this, its natural form.
What it sees is some patches which have been converted to some vectors.
And so at the very beginning, or with small data, the transformer model finds it difficult to make
much sense out of this patches.
But as soon as we increase this data set to considerable amounts, this transformer model, now free
of the inductive bias, can even do better than the confidence.
And interestingly enough, you notice that after training a transformer model, this position embeddings
we call the position embeddings, which are added onto the patch embeddings before passing to the transformer.
Actually learn on their own to encode the position of the patches.
You could see from this, uh, plot here where we have the input patch row and the input patch column.
You see that this one one you see the position this is gotten by the model or this is learned automatically
by the model during the training process.
You see to one, it goes a step in the direction and maintains this direction or maintains a row.
And you see three.
One maintains a row that goes three steps.
You see this, you see, you see that.
And then finally, here you go.
You go seven steps to the right and then seven steps downward.
Then year to the left, you could look at this.
You see this, um, this embedding filters right here.
We have this embedding filters which we see here which look much alike to the the conflict filters.
Then to the right, you have this plot which summarizes the reason why the whites end up being more
powerful than the conflicts.
To understand this, let's take this year we will consider a conflict with a given depth.
Now, with the conflicts, the initial layers, let's let's let's have this conflict and we break this
up.
So we break this up.
We have our initial layers and then we have our final layers.
This initial layers parameters extract low level features, while the final layers permit us to extract
high level features.
And so if we have an image like this, like this one, and we have this head and we have this, then
given that we're passing this filters or this conflict filters here, you'll see that this pixel, for
example, attends to this other pixels which are found around its locality.
And then as we go deeper in the network, we would have this pixel here which now tries to attend to
this other pixel here, which is much more far away from it to better picture this.
Remember the example we took for three or rather two three by three filters compared to a single five
by five filter.
Let's let's, let's draw this here.
We compare this with a single five by five filter.
And we saw that although those five by five filter had a larger receptive field compared to a single
three by three filter, which we have here, making or stacking up this three by three filters that
is making our network deeper permitted us to still be able to capture this part of the image.
And so this shows us that with the conflicts in the earlier layers, when the when is not yet deep enough,
we still capturing this local information.
And then as we go deeper, we start capturing much more global information.
And so if we're to have this kind of plot here where this we have mean attention distance, and here
we have the network that would see that for a confident, we will keep increasing this up to a point
where we may we will no longer be able to continue increasing because as this this network depth all
of this, we increase the number of layers.
We are able to capture much more global features.
And so this mean attention distance keeps increasing.
But with the attention or with the transformers, since each patch attends to each and every other patch,
as we have seen already with the self attention, each and every patch will attend to the other right
from the very first attention layer.
We are not going to have this, but instead this plot we have here.
And so this means that if we train our wit with a very large data set right from the very first layers,
we are able to capture both the local and the global features.
And this is what makes the whites more powerful compared to the continents when we work with big data.
Yeah.
We can also visualize what the model sees by looking at this attention maps.
You'll notice that after training the model you see we have this attention here.
See your pixels, which pay much attention to one another here.
These pixels are paying attention or much more attention to one another as compared to the other pixels.
In summary, to understand or to visualize what goes on when training to CNN and VITE model side by
side.
You will see here that with events as you increase this data site, this increased data site, and this
is increased data size here.
So as we increase the the data size, or rather when we start with small data sizes, we have this kind
of accuracy.
While for the CNNs we already have reasonable accuracies, even with small data size.
And then as we keep increasing this data size, as we keep increasing this data size, you see this
accuracy keeps increasing while for the CNN's stat to plateau at some point and this plateauing is simply
comes due to the fact that this CNN here are limited by the inductive biases, whereas this transformers,
which are more generic neural networks, are free to learn even better from this larger data sets.