In this lecture we are going to discuss how to work with large data sets in PI talk specifically large
image data sets.
When you're first learning about deep learning it's convenient to have data sets like amnesty CFR 10
and SBA Chen preloaded into a CSP or an umpire.
That way you can focus on the machine learning aspects and not the data processing aspects but in the
real world images don't come in the form of CSF vs instead images or images.
In other words actual image files like JPEG PMG and so forth.
These are files that sit on your computer.
They are not organized in a nice numb pie array for you.
On top of that images are usually much larger than what you see in amnesty or CFR 10 where you have
a 28 by 28 image or a 32 by 32 image image that models like veggie and that are trained on images of
size 2 to 4 by 2 to 4 which is an order of magnitude larger than amnesty
as an exercise.
Think about how much space you would need to store one million images each of size 2 to 4 by 2 to 4.
Please pause the video until you've calculated and answer
all right.
So hopefully you thought about how much space it would take to store one million images of size 2 to
4 by 2 to 4.
Here's how much space it would take.
First we have one million images.
Multiply that by 2 to 4 by 2 to 4 by 3.
The number of bytes per image.
This is the total number of bytes.
Our dataset would take up.
This is about one hundred fifty billion bytes.
This turns out to be about one hundred forty gigabytes.
It should be clear that you cannot fit a dataset of this size into memory on a standard machine.
So what should we do.
As always.
I don't like to give people code without having them first think about how to approach the problem themselves.
So let's first take a moment to think about how to approach this problem.
As an engineer rather than as a simple API user a copying code off the Internet first recognize the
difference between a disk and memory disk is generally slow and memory is generally fast.
The tradeoff is that disk generally has lots of space.
While memory has much less space your model reads data from memory but the images live on disk.
Second recognize that our approach is Batch gradient descent where we only look at one batch of data
at a time rather than the entire dataset.
So technically only that batch of data needs to exist in memory for the time that it's needed.
In fact that's all we need to know.
Let's suppose we have two arrays to represent our dataset.
Alice the file names where the images live and a list of target labels for those images.
Let's say our backside is 32 in order to do a batch gradient descent.
All we need to do is loop through each of the arrays 32 items at a time inside the loop.
We load in those 32 images and store them in an umpire.
Of course thirty two images is a totally feasible amount to store in memory.
One million is not but 32 is just fine.
Then we can call a function like model dot train on batch passing in X and Y which will do one iteration
of gradient descent on this batch of data on the next iteration of the loop.
We throw at the old data and reassign x and y to the next 32 images.
Thus at any point in time only 32 images exist in memory.
Once they are used that they are forgotten until the next epoch
luckily in code although you could do this manually you don't have to.
We're going to make use of a few special functions and classes that make it very easy for us.
First I want to show you all the main ingredients and then I'll describe how to put them all together.
What you should recognize is that this is just like the data set and data loader combinations we've
seen throughout the course previously the image data set objects we use before loaded in images into
single arrays.
These images were very tiny.
So that was OK they could all live in memory at the same time in the same array.
We had specialized data set objects for different datasets such as Amnesty fashion amnesty and CFR.
So your question should be is there a kind of data set object that does not load all the data into memory
All at once that is generic.
So it's not tied to any specific data set and that can be used for actual image files like J pegs and
PND instead of arrays
and of course as you would expect this functionality is included in the torch vision library.
So how does it work well quite appropriately.
It's an object called image folder.
Basically it means your dataset is a set of files that exists in some folder specified by some path
and the arguments are generally what you would expect from a data set object.
The first argument is the path to the Data folder and the second argument is a list of image transformations
as you recall from earlier.
These could be used for data augmentation and for simple scaling like making the pixel values go from
0 to 1.
The last thing I want to mention in this lecture is that using the image folder data set object necessitates
a very specific folder structure.
This folder structure is very reasonable so there's no reason for you to not be able to store your data
this way.
Here's how it goes.
First assume your train and validation data live in different folders.
Let's call them train and validation.
Next within each of these folders we have an individual folder for each class.
The name of the folder should be the class name.
So for example if you want to classify cars trucks and helicopters then we should have three folders
car truck and helicopter next.
Within those nested folders is where we would store the actual images.
So inside the car folder we would have images of all cars inside the truck folder.
We would have images of all our trucks and inside the helicopter folder.
We would have images of helicopters.
Again this is just how towards Vision expects your data to be organized.
So if you want to use those built in functions then you must conform to this format.