Hello, everyone, and welcome to this new and exciting session in which we shall look at training,
data efficient image transformers and the solution to attention.
In this i t image transformer paper by go to all the show that it is possible to train an image transformer
on a relatively smaller data set like the image net while getting competitive performance with no external
litter.
Now if you're new to this topic on image, Transformers is important for you to check on our previous
section where we treat the image transformers from scratch.
And one thing we know that from that section was that to train this image transformers the weights and
get competitive results.
We needed to do this on a very large data set like say 300 million data point data set.
And then if we want to train this or if we want to work with a smaller data set, then we will now do
fine tuning.
And so as we see with this itis, they are able to produce results without necessarily training on this
kind of large data set.
In this section we'll see how the use this teachers to ten strategy to help the image transformer achieve
competitive results without necessarily being trained on an extremely large data set.
Before getting to the subject matter.
The authors start by an overview of the vision transformer, which we've seen previously.
So here we have the talk about this multi haired self attention layer and then they get to the transformer
block as a whole that's taken into consideration that multi head transformer layer.
And then from here they discuss the class token and how it is used by the image transformer.
And then finally they discuss this modifications made on the positional encoders when carrying out fine
tuning in order to understand this architecture of the Z i.t transformer.
We have to start by recalling that with the usual image transformers, this transformer which doesn't
have any or which has very little inductive biases, finds a difficult learning from very small data
sets.
And so as we increase the data set size, this transformers then start becoming much more performant.
Now, what if in this small data regime, right, here's where the model's performance isn't that great,
we are able to transfer our knowledge from a component which has or which has better performance in
this small data regime such that this image transformer takes advantage of this confidence inductive
bias to update its weights in such a way that it learns from this smaller data sets.
Now, the authors devised this technique known as Hard desolation, where we have this conflict, which
is a teacher and this image transformer, which is actually our usual image transformer, which is a
student.
And then here they add this desolation token just in a similar way.
We added the class token as we saw in the previous section.
So here we have both the class and the dissolution token together with the patch tokens as usual.
Now what goes on here is you have this class desolation patch which had been passed in and then at the
level of the outputs, we are going to ignore this and then we'll focus on the class and this distillation
outputs right here.
And so instead of just minimizing this cross entropy loss, we are going to minimize both this cross
entropy loss and this teacher loss right here, which is given by these two formulas.
You see here we have the model's output.
This is what the model outputs, and this is the true level.
So usually this is what we calculated.
So before we had just this, now we have this and this together.
So now we we've trying to minimize the difference between what the model outputs here and the actual
output and also the difference between what the model outputs and the teacher's output Y of RT.
So this means that we have this conflict right here which has already been pre trained on say, image
net and for a given input we know its output already.
And this calf nets output, which has been pre trained, is not necessarily this y level or the true
value of Y or the true output.
But with this they found that after training to convergence this tokens here, this embeddings we pass
in are not very similar to one another.
But as we go up that as we get deeper into the model right up to the end, we find that we have this
to outputs now which are very similar.
And with this we see how this teacher or this curve net right here, which has been pre trained on the
image net, helps this image transformer to learn from very small or relatively small theater to get
the sense of how small this is compared to the kind of theater we work with previously.
The image net is about 1.2 million data points.
While the dataset we're working with previously was about 300 million data points.
Then the authors your provide these three models, the T, the SE, and the B with your different embedding
dimensions, number of heads, number of layers, number of parameters, the training resolution and
the throughput.
We see here that this DTP version performs as well as the corner teachers.
Then in this table right here, we see that it's preferable to use the d ite with the hard distillation
as compared to the usual distillation technique or when we work with no distillation at all.
So the hard desolation adopted by this authors makes the image transform our perform better.
And then we see that working with both the class and the distillation embeddings gives us better results
as compared to working with only the class or only the distillation embeddings.
In this table.
Right here, we have a summary of the different components, which we have your different state of the
art components, and then state of the art transformers where to carry out this comparison with respect
to a number of parameters, the image size, the throughput now for the throughput is simply the number
of images the model can process per second.
So obviously you want the model or a model to have a high throughput because you want to be able to
process as many images as possible per second.
Then here we have this top one accuracy.
We have this order to one accuracy on this order data set, and then this other one on this other data
set right here.
Now, if you look at this year, you'll see that this de ite model outperforms the usual, right, while
being trained on fewer data and hence closing the gap between the transformer performance and the conflict
performance.
That said, given that this de ite model was trained on smaller or relatively smaller data set compared
to the initial vite, the authors use the data augmentation very much so.
Here you could see the random augment which was used or to augment the mix up, the cut mix and other
regularization strategies to help avoid overfitting.
And that's it for this section where we have seen how to train this data hungry image transformers with
relatively smaller data while achieving even better performance.