Hello, everyone, and welcome to this new section in which we shall be treating the screen Transformers.
Up until this point, the kinds of transformer models or specifically image transformer models we've
been working with, I've been designed such that the input shape is the same as the output shape.
So if we have layers, we would have a situation where we could have an input.
Let's let's say we have this input, let's say nine by seven, six, eight, where nine represents the
sequence length, we suppose we're taking all nine input patches.
If we add this, then we'll have ten.
So let's say we have ten by 768.
Where 768 is a embedding dimension or the hidden size.
So we have those kinds of inputs and when we pass this to layer one or we pass this through the encoder,
this ll when L one, when we pass through one of this or the first transformer encoder, we have an
output which has exactly the same shape here, 768 And then passing through this next one, we have
the same and all of this right up to the end.
So yeah, we'll also have seven, six, eight and here we would have seven, ten by seven, six, eight.
Now you have omitted a batch dimension, so you could add the batch dimension here.
Now that said, you see that we have this shape which is maintained throughout the transformer encoder,
but yet with the screen transformer, as we shall see, this shape is modified throughout the network.
So you could see here you see this ship which has been modified as we move from one stream transformer
block to another, coupled with the fact that the transformer is designed in a hierarchical manner.
The authors of this paper also introduced this shift, that window scheme from where the name was Garden
here, swing Transformers.
And in this section we'll go in depth to understand how this hierarchical structure has been designed
and also how the attention is being computed with this shifted window scheme.
Now, unlike with the usual model where we start off with 16 by 16 patches and end up with 16 by 16
patches, as we move to the different image transformer blocks like you could see here.
You see we start out with this and it remains this same right up to the end with this screen transformer.
We starting off with 4x4 patches.
You could see that year.
If you look at this image, you see you have one, two, three, four, one, two, three, four.
So this is one 4x4 patch.
And so the original image is broken up into this 4x4 patches.
And since each 4x4 patch is made of 16 pixels and that this image is RGV.
So we have three channels you see we have 16 times three.
That is 48 channels for each token because we consider this one now as a single token and then the single
token will now be made of 48 different features.
And that's why you see you have this 48.
So do we suppose another heightened with some arbitrary value?
Now if we have height way to be 256, then we would have 256 divided by four, there should be 64,
so this will be 6464 times 64, which is 4900 or 4096 tokens.
Let's take this off.
So as we calculate that 256 times two 5256 is our image height and our image width.
So to 56 whereby 464 to 6 by four, 64, 64, 1064, 4009 to 6 tokens.
So we have 4006 tokens and then each token will be made off of 48 features as we've just seen here.
Now with this we pass this into this linear embedding.
Now we call the role of the linear embedding is to help us get a desired hidden size.
And so here, as we could see this, our desired hidden size is C, so the fixed is to be C, So C could
be any value.
Let's suppose that C is 768.
Now this means that at this point we have 4096 tokens with each 4768
features.
So we have that Now once we get to this, we get to this patch merge.
Notice that this is the first time we meet the we.
Get this patch merging layer here.
And what does patch merging layer does is what you what you find here.
So you'll see that it's going to join this the different patches together, hence the term patch merging.
So it merges you see this, this four patches into this one and then merges this into this one.
It merges this into this, merges this into this.
And so now we have our tokens or a number of tokens which will be divided by four.
And if we divide this by four, we'll have 1024 tokens.
And with or rather in this patch merging layer, the we move from C features to two C features.
So here we have two times 768.
Let's, let's just say to see.
Which is 1536 features.
Now to again better understand how we got this from this year.
Recall we have H and W to be 256.
Then we'll have 256 whereby eight times 256 whereby eight which is 32 times 32, which is 1024.
Then again we go to this exact same process where now this four tokens are being merged into one.
And the number of tokens here again is divided by four.
So now we have 256 tokens, 256, and this number of features is multiplied by two.
And so we end up with this 256 by 3072 and the 64 by 6144.
Then instead of carrying out the attention globally, we instead carry out this attention locally.
And so this means that when the image is played out like this with the different tokens, the attention
here, let's take this off.
The attention here is only in this local region.
So we carry our attention with this local region, attention, this local region, attention, this
local region and so on and so forth.
And it turns out that carrying out the attention locally is computationally less expensive than carrying
out the attention globally.
Now, to understand why the users are working with local attention will be more computationally cheaper
than working with global attention.
Let's consider the following example.
So here we have this eight by eight pixel image, eight by eight pixel image, and then we'll decide
to create patches which are two by two.
So this is one patch, this is another patch.
So these are the different pixels.
So we have all the different patches.
And after this, at the end of this process, we should have 4x4.
That is 16 different patches.
And so when we have this kind of when we have this kind of setup where we have 16 different patches,
it means that our attention map will be 16 by 16 because here you have 16 one, two, three, four,
five, six, seven, eight, nine, ten, 11, 12, 13, 14, 15, 16.
Patches.
We're getting all these 16 patches from all from your and then each and every one of them attends to
one another.
So this one attends to all the other patches and this one attends to all the others and so on and so
forth.
So you have all this attention resulting in a 16 by 16 attention map.
Now, the 16 by 16 attention map comes from the fact that we multiplying 1616 by C times, C by 16,
because the each patch here will be a query and then the key is being transposed to have C by 16.
The C is the hidden size.
And then from here we have 16 by 16.
And so this computation or better still, this matrix multiplication becomes very expensive when we're
dealing with a large number of patches.
Anyway, let's take to 16 and we have 16 C times 16, C ten, C ten by C by 16.
This is not 16 times C, this is a C by 16 matrix.
We saw this already in some previous sections.
So this is C by 16.
Let's have this here and this now gives 16 by 16 OC Now let's suppose that we carried out this local
attention.
So here we are, we carrying out the attention on just this two by two part patches right here.
So the attention here will be we will have one, two, three, four.
So we have one, two, three, four.
This one attends to this three.
This one attends to this.
Attends to this.
Attends back to this.
This one.
Attends to this.
This attends to this.
And this.
Attends to this.
So we end up with four by C times, C by four.
And so now after computing this, we have a 4x4 matrix.
So we have 4x4 output here.
Now, since we'll do this 16 different times for this, we do this locally, this locally, this locally,
this locally and so on and so forth.
It means we take 16 or rather will take this computation and carry it out 16 times.
Now this means that if instead of 16 patches we have 2048 patches will be computing a 2048.
These 2048 by sea.
Times a 2000.
Let's let's write this better.
Let's write a year.
So we'll be doing 2000 of 48 by C times a C by 2048 metrics which would give us a 2048 by 2048 metrics,
which has more than 4 million different positions.
That's in this matrix.
If we if we have two.
If we take 2048 times 2048, you give us more than 4 million different positions.
And then if we're dealing with this, we would have the 16 change to 2048.
We still have our 2048, but here we still have this four by C, ten, C by four.
So we still have this 4x4 matrix.
Which is computed 2048 times.
And so this shows us clearly that the local attention is much computationally cheaper than the global
attention.
But remember that one of the advantages of working with Wits is the fact that we have actually this
global attention.
So how do we take advantage of this local attention, which we've just presented, where attention is
carried out only within the selected patches while still doing some global or while still carrying out
some global attention?
This is done using this shifted window approach, which we saw already from the abstract, where in
a given layer l As we could see here in this layer L we have the usual partitioning scheme and then
attention is carried out.
Only in this local window we see a local window to perform self attention, see with this red box surrounding.
So that's what we have here.
Let's take this off so you could see that clearly.
So you see attention is carried out in this locale and this local patches, this patch, this patch,
this patch and this patch.
And we've seen already the advantage of carrying out this attention locally.
Then to solve the problem of global attention, they modify this partitioning scheme via this shifted
window where original originally we could have something like this and then we go see two steps downward
to steps to the right and we stop here.
We obtain this and then for the next layer we could go again to and this way and see we obtain this
now.
And so this means that unlike previously, let's take this off.
Unlike previously where this only this year, only the pixels in this patch could attend to one another.
Now you see this pixel right here?
This pixel is now found here, and it's now attending to this other pixel, which initially was on a
different patch.
Because now they found at this position together or they now make up this patch.
And so with this now, not only is the local attention helping the screen transformer be more computationally
efficient.
On the other hand, we have this shifted window technique, which still helps us carry out global attention.
And in this table right here, we see how all we can see the results of the screen transformer and a
different version of TSB with image size 224 and B with image 384.
And how it outperforms this d i t and V transformer models while maintaining a reasonable throughput.
For example, here we see we have this of the image size 384 and then 307 million parameters.
When we compare this with the screen B version, see the same image size you see is much smaller number
of parameters.
Then the floating point operations here are smaller and then you see this, the throughput here is higher.
And then also the top one accuracy on image net is your higher two.
Comparing this with the ITE, although here we have slightly higher number of parameters for the number
of floating point operations.
This is fewer floating point operations then the triple here is slightly below the the digit B version,
but the accuracy here is higher than that of the DCT on this table.
Here we have something similar for the image net trained on its 22,000 class version and you could see
that again, the screen outperforms the models and even the rest nets.
Then another strong point of the screen transformer is that due to its shifted window based self attention,
it does also very well on other computer vision tasks like object detection and segment segmentation.