So, no, let's take a look at how we use multiple GPUs in PyTorch lightning.
Now, firstly, we can't actually run this on Google Cloud because Google Cloud only has one GPU, so
we can't actually in practice, test this out, unfortunately.
However, you can follow this notebook for guidelines if you have to use install in system.
You can definitely utilize both of them in training and speed up your training time by by double.
So let's take a look and see how we do that.
So firstly, you're going to have to do some code clean up a bit.
So let's take a look at the good habits that you need to establish.
Firstly, delete all your Okuda or the two calls.
So those are the calls that send data or send.
They'll actually reduce sentences to the GPU or to the CPU.
We need to delete these lines of the code next.
What we have to do is synchronize validation and test logging.
So when you're logging your logs, remember yes.
Is this an evaluation step in and test up as well in the training step?
You have this self-taught log.
And then what we need to set here is sync distribution equal to true.
What does does?
This sets the distributed mode to true.
So now we can actually have to put GPU is synchronized properly when logging or otherwise it could.
It could get messy.
So these are the things that play towards the lightning is taking care of kill for us.
So it's quite convenient.
So you can take a look at a document if you want to see more and more of the documentation, here is
a typo right there.
Let me just fix that.
It's actually two Typekit.
This happens in color because there's no spell check.
Anyway, this here is the PyTorch model that we've been using previously.
However, this is the one that supports multiple GPUs, and you can see we have this here.
Actually, we don't have it, you know, training step.
It's only in the validation steps, actually.
So you have the sync distribution.
There should be the equal true sync sync test.
And that's it.
This is ready to go.
All those dot CUDA and the two calls have been deleted from that code.
So you were ready to go.
This code can be used now.
So let's run this.
Keep track of it now.
There isn't.
Everything is commented here because it's not going to really run in CoLab.
But this is an example of how we can use different GPUs so he can list the GPUs, get a range of them
here available, or you can specify CPU 0g if you run can do it this way as well, or you can set this
one to use all GPUs minus one.
So this is the way you can configure using multiple GPUs or different combinations of GPUs in PI to
watch lightning.
So next, we'll take a look at how we actually run that, and it's actually quite simple.
Whatever configuration you want to use, like if you want to use this configuration is what I've commonly
seen used.
You can just place that argument right here with GPUs line.
Replace it here and you can just run this.
So this won't work because we don't have multiple GPUs in the club.
However, this is the way to do it, and it should work on you on multiple CPU configured system.
So next, we'll take a look at the profiler.
So what is a profile of a profiler as a way to?
You can.
You can use this as a detail in software engineering.
You can profile each line of your code and see how long it takes.
This is a very good way to find bottlenecks in your training process, and there are lots of buff lakes
and deep learning training, as you can tell, is a lot of the issues with data loading data transformations.
There's a lot of other stuff that just passing data around from CPU to CPUs, that type of stuff.
So by running this here what it does, it runs it from one e-book, which we specify here, and we can
set it up with simple, which gives us a lot of information still.
You can see this here.
You can see this are all the functions that are running here.
This is a time it takes to run a number of times.
It executes the total time it took and the percentage of time that was attributed to that.
So you can see what functions are quite fast.
These here and the one is quite slow.
It's understandable.
It's running training e-book that takes roughly just over a minute and several seconds.
But you can see things like get training batch that has some delay as well as as well as these things
here.
So you can definitely see what's going on.
And that's quite cool.
It's quite a good way to assess the performance of your modules, and you can see how much clothes as
well here and total time attributed to it.
So that's pretty cool.
Now let's take a look at training on top use.
So what exactly is a TPU?
Well, the TPU is a hardware accelerator that was developed by Google.
It's called Circuit Short for Tensor Processing Unit, and it was.
Designed specifically for deep learning applications.
In fact, you can take a look at this here TPU processing unit has eight cores and each cores optimized
with 128 by 128 matrix multiples.
So that means a single CPU is about as fast as five A100 GPUs, so that's pretty cool.
I'm not sure what TPU unit you get in call up right now.
It changes from time to time, but nevertheless we hopefully we can use the TPU right now.
However, there's an issue right now when I created this notebook and was working.
This is following the instructions on the from the official API to watch the Lightning Library to use
an colab.
However, it's a bit broken at this installs correctly.
However, it's not loading something properly here with the TPU support in PyTorch and because of a
site on a C Python issue, perhaps not entirely sure.
So this this code is broken right now.
Unfortunately, however, hopefully Google does fix it.
Maybe, and we can actually start treating almost all networks using TWRP use here again with Pi to
us lightning.
And it actually trains this network much faster than the GPU, which is pretty cool to observe.
So we'll stop there for now.
And next, we'll take a look at something that's very important.
It's called transfer learning and fine tuning.
That's basically almost everything that deep learning practitioners like myself do.
So we will get started with that.
Stay tuned.