Hello, everyone, and welcome to this new and exciting session in which we are going to treat TensorFlow.
Record TensorFlow Records helps us build more efficient data pipelines as they help us store data,
which we are use to train our models more efficiently, and also the help in paralyzing the reading
of the data, hence helping in speeding up the overall training process.
And so in this section, together with the TensorFlow data sets, which we've seen already, we are
going to see how to implement an efficient training pipeline with TensorFlow.
In the previous sections, we've been working with TensorFlow datasets in the session.
We'll see how to convert our TensorFlow dataset into a TensorFlow record and then get this TensorFlow
record and convert it back to TensorFlow datasets to be used for training.
Now, the very first question you ask yourself is, given that we've already carried out this training
process successfully without any problems, why do we need to work with TensorFlow records?
Now?
There are two major problems TensorFlow records come to solve or the two advantages of working with
TensorFlow records.
The very first one is the fact that you can now store your data more efficiently.
Now notice that every time you have to create this data, you're creating this data from this data set
which we've loaded here, which is made up of little files of like, see a few kilobytes, let's open
this up so you could see for yourself.
We have this year let's take this one, open this folder, and then we'll see the size of this.
So you can see the size is about 17.26 kilobytes.
Now, the fact that we have to do with deal with these kinds of files means that we are not going to
always have to load this data very efficiently.
Now, it's true, working with the TensorFlow datasets brings in some efficiency, but what if we start
this data in a very efficient manner?
That is, instead of storing, let's say, for example, 17 kilobytes files like this, what if we just
store, say, ten megabyte files or say 100 megabyte files?
So every time we want to read from a file, we don't have to read from many of these kinds of files,
but from a single file like this one.
Then another good thing with having to work with TensorFlow records, which start in this kind of form,
is that this time around you could carry out the preprocessing before storing the data so you could
augment your data.
So we have augmented data, which is now stored as TensorFlow records.
So instead of having the images and then getting here and each and every time you have to carry out
augmentation or some preprocessing before training, you store the data, which has been augmented already.
So suppose you have your initial data, so you have your initial data year, it passes through some
pre processing, so it passes through some pre processing and then let's change this color.
So yeah, we have the data and then here we have the pre processing and then from the pre processing
we have the augmented data.
So after this the data has been augmented.
What we do is instead of having to go through this each and every time, what we do is we pass through,
we go to this once and we store our data in.
This is augmented form such that the next time we want to train our model, all we need to do is just
make use of this augmented data right here.
Apart from this, it should be noted that sometimes we have models or we have sections of a model.
Let's draw it this way.
We have let's suppose that this is our model here.
So this is our model is made of section one and section two.
The Section one is fixed, meaning that we are now going to train this path, but we're going to train
only this part.
This means that when you pass in data right here, when you take this data and you pass in your the
output, you would have your is going to be the same each and every time.
Given that this year is fixed, the weights here are fixed.
So what we could do is instead of storing the data, we are going to start this outputs which we could
call embeddings.
So instead of storing the data, we store this embeddings here such that we now make use of them.
Bedouins directly and train our model or train this part of the model which is actually trainable on
this embeddings instead of working with the dieter.
So this time around we see that we have data, the preprocessing, the augmentation, and then we have
the embedded or the embeddings from this data.
So here we have the embeddings, so we can now store this as TensorFlow records.
So you see that it gives you or it gives us some kind of flexibility as to what we can store and be
able to retrieve the START data and make use of it as we wish.
The second major advantage of working with TensorFlow records, apart from this efficiency in storing
data, is the fact that they encourage the parallelism of reading data.
Now, this means that if we happen to train our model on several hosts, let's say we have four, four
different hosts or so, we have four different machines on which we train our data like this, and we
have our data set right here.
What we could do is we could create shards of our TensorFlow record data.
So here, for example, let's suppose that this are our data set, our complete data set.
We could break this up into several parts.
So let's say we break this up such that each host takes care of two.
So we have 1 to 1 to 1 to 1.
Let's add another part here.
So each of these takes care of this.
So this this host yet trains on this data, this other one trains and this this other one trains on
this this one trains on this.
Now, as a rule of thumb, is generally advisable to make sure that each host has about ten of this
packs of 10 to 100 mega bytes of our data set.
So that said, if we have a ten gigabyte data set like suppose that all this now is ten gigabyte, let's
take this off.
We have ten gigabyte data set.
That's right this in here.
So we have your ten gigabyte data set.
What we want to do is have your likes each and every host taking care of at least ten packs of our TensorFlow
of our TensorFlow record, which we've created.
So let's take this off.
It's no longer two, we have to create ten packs or ten charts of our TensorFlow records and allocate
that to each in every host we have here.
Now, that said, given that we have four hosts, then we will have four times ten.
Shards to create.
So we'll break this ten gigabyte data set up into 40 different parts.
So we have 40 different power packs or shards of our TensorFlow data set or rather of our TensorFlow
record right here.
And then each and every one of them will be approximately 250 megabytes.
Since ten gigabytes that we convert this to megabytes, we will have 10,000.
Suppose that a thousand megabytes equal a gigabyte.
So we have your 10,000 megabytes divided by 40, which will give us 250 megabytes per shard or per pack
that we've created your per split off our TensorFlow records.
And so this will lead to some reasonable gains because we now can train our model on this paralyzed
data.
And then we could also pre fetch huge chunks of our data set to be precise, 250 megabytes, such that
once the model is ready, the model just feeds on this data, which has already been pre fetched.
Now, you could check out the previous sessions where we talk about pre fetching under TensorFlow data
sets.
That said, what we actually store in this TensorFlow records are the protocol buffers.
And the way TensorFlow manages this is by making use of this TensorFlow example class which is defined
here.
Let's check this out.
Your TensorFlow example, which is defined here as a standard protocol storing data for training and
inference.
And so if you have to convert your data into this, into the proto files, you would have to make use
of this TensorFlow example class and then you would have to understand this representation right here.
And so in our case, where we're dealing with an image and its corresponding label, let's say we have
the image of this person, smile, then we have the label one, so we have this image and the presence
corresponding label would have to convert this data into this format before creating our TensorFlow
records.
Now here we have this dictionary.
As I said here, it contains a key value store example features where each key, which is a string,
maps to a feature.
Now note that here we have this feature with SX and here without SX.
So we have this features and each and every one of them is a feature.
So we could click here, you could click here and you see this feature, you could check out the documentation
and we have this contained.
Let's take this off.
The content list can be one of the three types a byte list.
Generally, this is our information flow list or an inch 64 list.
So you would pick your feature depending on the kind of data you're having.
Now here you have this features, which is like a combination of these different features here.
So you have your the INT list, you have the float feature, the float list, the byte list, and then
here you create the, you create the features from this different features here.
So from the, from each.
And because this is a feature, this, this, this one year, this one year is a feature.
Let's take this off.
This one year is a feature.
This is a feature and this is a feature that has been defined already here.
So you can see NS feature is a type feature there.
Is it float feature the same?
There is.
It bites feature the same, that is it.
Now all this combined forms features which is of type features.
See the difference.
You have this and this without the se obviously the singular and that's plural.
So that said, if we want to create our TensorFlow records, I want to convert our data center TensorFlow
records.
We have to take into consideration this formatting of our data.
So we get back to the code and then we'll add this to in parts here.
Here we have this TensorFlow train.
We import my list float list and in 64 list.
So these are the types of our feature.
And then we have from TensorFlow training, again, we import example, we import features and we import
feature.
So with that, let's run the cell, We get back to our code.
We have your TensorFlow records.
Now what we'll start by doing is on batching our data.
So we'll start by batching our data.
Note that to run this we've taken we've taken off this pre fetching as we will not need this we just
take our data as it is.
So we have our trained data set which has been augmented.
You could carry out any preprocessing you want before starting this data and then the validation data
remains validation.
So we have that valid data set there.
Then from here we're from the cells already from year, we'll go ahead and batch this data.
There we go.
We have training data set, we run this we have this year.
Then we do the same for the validation.
So here we have validation data set and validation is set on batch.
There we go.
We have that you could see for yourself training data set, training data set.
This is going to be unmatched.
So you have your own batch data set.
Notice that the batch dimension is taken off.
Good.
Simply have this for the validation.
There we go.
Validation.
You run that.
See that taken off two.
And then we get back to this documentation.
Recall that before creating the TensorFlow record, we need to put our data in a certain format.
More specifically, we need to create proto files.
And the way we do this is by making use of TensorFlow example and then to create the stance of example.
We have this features which we need to combine to create this TensorFlow examples.
And in this documentation here we have all that is needed to create this.
So here you see for example, this example.
You could copy this out simply and then we paste this right here so you could see the int, the float,
the bytes.
But since we are not going to use the floats here, we could take this off.
We're not going to use this float feature.
What we're interested in is the byte feature because recall we have the image and we have the label,
so we have the image of the person happy and then we have the level one.
So this label will be this end feature and the image will be the bytes feature.
So let's go ahead and get this done.
We have your let's put this first.
So here we have our bytes and then we have our INT.
Now we're going to create an example from your you can see clearly we've imported this already, so
we just have that even here can have this features.
There we go, here we have byte list.
See here we have feature here, we have feature here we have in 64 list feature of type for list and
that should be it.
So we have all this year we need we take this off, we don't need that.
So here we have we change this name.
We put we call this image, we'll call this image our images and then we'll call this labels.
So that's it.
Okay, So we have this year, let's get back.
That's fine.
We have that.
The bytes does images and we have the levels.
Now, once we have this, the next thing to do is to put in the correct values in here.
So instead of having this year, we take this off and then here we're going to create a method which
will call create example, call This method create example is going to take our image and also the label.
So it takes the image and the label and then what it returns is our serialized example.
So we've had this example which we've created and these are serialized examples we pass we call the
serialized to String Method right here.
Now, that said, instead of this year we'll take in our image and then instead of this year we'll take
in our level.
So that should be fine.
We now run this and the next thing we'll do is define the number of charts.
Here.
We have ten charts and then the path.
So you call this TensorFlow records.
Let's create this new folder, your F records, That's OC, t F records.
That's fine.
And then we will have the charts with your specific number.
So here we have the chart or basically we have this name, let's have your file name, and then here
is the extension to F record.
Okay, so that's it.
And then look and run this, and then the next thing we want to do is to get back to the documentation
to f dot i.
O.
And then we get to this TF record rider where we are going to see how the write in a TF record file.
So we have this year years of simple definition arguments.
You specify the path and you also use this example.
So we have here write the records to a file.
So let's copy this simply and then we paste this year.
So we have with this TF record writer, we specify the path you're our path is going to be this path.
We have that path.
And then as file writer, we're going to write our information in that.
Now we want our file to get those different names.
So here we have path, dot format, and then we'll specify a given shard, so a given part of our data.
So let's get back here.
Recall that when we create our TensorFlow record, we could create this as a block like this, and then
we'll later on shard this, break it up into different parts.
So here is actually we have ten shards, so break this up into this ten different parts.
One, two, three, four, five, six, seven, eight, nine, ten.
Okay, So there we go.
We have this ten different charts, and then we want each chart to have a different file name.
And that's why if you look at this year, you see we have this formatting such that we pass in a given
chart number in year.
So we have the chart number.
So that's it.
And what we'll do now is for each shard that's out of the ten charts here for chart number in range,
number of shards, that's number of shards specify Your Honor has ten.
So it's basically for shard number in ten.
So we are going to go through, we're going to loop through this and we are going to create a file for
each shard.
So we're breaking up here.
So for each and every one of this, we're going to create a file, this one, a file, this a file,
and so on and so forth.
So let's take this off.
There we go.
Now, getting back here, once we have had this in place, once we've set this up, now, recall that
we had created this create example method right here.
And if you notice, you find that in this year we actually or this example is kind of like doing exact
same thing with this create example where we have these different features which are created and then
at the end we have this serialized to string method, which is called.
So you could see here there's different features, you see these features and then they all combine
to form the example and then serialization.
So this means that all we need to do here is pass in.
We have here instead of this we have create example and then we'll be passing in the image and the label.
Now we'll take this off here.
We do not need this.
Basically, we don't need that.
And then we'll go for image and label in our data set, in our TensorFlow data set.
Let's get back up the what are we doing with now?
So our training data set.
So for this, in our training data set, that's it.
We're going to write this.
So we're going to have to write this image and this label in our our file will be created.
So in this our TensorFlow record file.
Now, given that we have to write a given chart and not just a full data set, yeah, would we change
this to chart sharded data set?
And then let's have this here we have our sharded data set which is going to be equal.
We have that training data set and then we'll shard so we could get to the definition or I could get
to the TensorFlow data.
Let's scroll this TensorFlow data data set here.
You could have this or you could find the Shard method right here.
Let's scroll down and we'll click here.
You see, we have let's click on Shard Shard.
So you see here we have the definition, takes a number of shards and then it takes the specific index.
So here we have ten shards.
And then for each index we're going to pass in the value here dynamically.
So here you see that this creates different parts or parts of our dataset.
So here we have each pack which is going to be created.
So we specify the number of shots.
There we go.
And then we specify the shot number.
So that's all we need to create a part of our dataset.
So that's it.
Once we have this, let's now run this and see what we get.
Get in this error for sharded number.
Run that again.
What do we get?
This very speeded one of type bytes.
So what we're passing is tensor, but we expected a bite.
Now the next question is how do we convert the sensors to bytes?
So we do a quick Google search here, convert image to bytes in TensorFlow.
There we go.
We have decode image, but in fact, what we'll be using to convert this image into bytes is this encode
jpeg right here.
So we haven't JPEG images so we could use this here using it.
It's quite simple.
We're just simply passing the image and then we'll consider all this to be default values.
So here we have encode JPEG and then we're going to create this encoder method.
Your encode image takes in the image and the label.
And then what we have here is that we're going to start by having getting the image.
You say image is going to be the encoded version.
So we have to do I0 encode JPEG, that's it.
We pass in the image and then we're going to return the image and the label.
So once we have this, let's add this code, so let's run this and then we're going to create a new
encoded dataset data set equal this.
We have training data set.
There we go.
We're going to map, encode, image, That's it.
So that each and every time we want to pass this into this, our file writer, we're going to make sure
that we have this in the form of bytes so that it matches up with this year with this feature, which
is defined in our create example method right here.
So we need to match this up.
So that said, let's get back.
We have our encode encoded data set.
Take this off, we run, we have encoded data set.
We told that this image has type flow 32 that is match expected type of unsigned int eight.
So what we need to do here is we need to convert our image first into this unsigned it.
So we have here now to get a solution.
We'll get into TF image, convert image, the type so that we could convert.
Let's actually convert, let's get to C convert image D type.
So as we're saying, we could actually convert our float 32 to the unsigned int So here all we need
to do, as you could see here, you see you have the tensor and then you have the D type specified.
So that said, let's copy this and we have that anyway, we're going to, we're going to take the default
value for the saturation.
So let's get back here and then at the level of this before the encoding, we're going to convert that.
So here we have image.
Image equals that and then we pass in the image here.
So we have to f dot on sign int eight, int eight and close that.
So that's it then.
Set we can now let's run this and run this again so we could see our encoded data set.
So that works fine now.
So we have our encoded data set and then in here instead of having this, we'll call this encoded data
set.
So we have encoded data set and then we'll run this and see what we get.
Value must be iterable, so we must convert this into an AI terrible data set.
So to do this we just have your as non pi iterator, we have as non power trader.
We run that again.
We are getting here to 55 has type in expected one of bytes of this error is coming from our create
example method.
So let's get back to create example.
Everything looks fine but here we should have a list.
So let's have this here.
There we go and run this again.
And that's it.
The.
Creation of the different files now complete, you could click open your and you see we have all this
ten different charts.
At this point, what we could do is we could save this files and the drive such that we could use it
next time for training.
So let's go ahead and see how we could make use of this for our training process.
Now, what we want to do here is to convert this back to a tensorflow data set.
So we had our TensorFlow data set.
We converted it into a TensorFlow record like we see here, converted this into some different TensorFlow
record files and now want to reconvert this into our TensorFlow data set.
Now, to get this reconstructed data set, we are going to pass the different file names in this TF
record dataset right here.
So what we have here is this list made of all these different file names right here.
So this list, in essence, let's call it l will be our path, which will format and then we'll pass
in this variable P for P in range, the number of shards for P and number of shards.
Let's print out this L run that, and we get this list which is made up of all the different files here.
So that said, what we're going to have here is we're going to copy this, we're going to copy this
here and then simply replace this list with it.
So we have this list now.
Let's have this.
We have this list now, and then we run this and get our reconstructed data set.
But then we need to parse this TF record data set such that we could get our original data, which was
in the form of the image and the label where the image was an array and the label was some integer.
And so with that we have your parse single example method, which we're going to make use of, which
takes an example which is basically what is contained in our reconstructed data set, takes an example
and permits us to split this into the image and the label.
So right here we'll have example and then we'll get the image we have the image that will make use of
the decode JPEG method.
So previously we encoded the chip into JPEG or we convert it into bytes.
Now we're going to convert from bytes back to the unsigned integer.
So here we have decode JPEG and then we're going to pass in the example image.
So we have that image.
We specify a number of channels to be three.
Now we have all the set, while we'll have to do is specify this feature description right here.
Nonetheless, this feature description is basically what we had in this create example.
And so here we have this dictionary.
Let's have this images, labels your images.
We have this dictionary which is made of the images and the labels, and then we have the data types
of the images and the labels respectively.
So we need to pass in this feature description, in this parse single example method.
Now we have that we set to return our output.
So we take example and we have images and then example and we have labels.
So what we're doing here is we take an input example and then we're breaking it up into the images,
into the labels while converting these images from the bytes back to the unsigned end.
So we have that.
Now let's run the cell.
And then here we have our parsed or parsed data set.
There we go.
We have parsed data set and we have recurrence data set.
There we go.
We map this method here.
So we have parse to F records or parse F records.
So we have this method, let's run this.
That's fine.
And then let's see what is contained in our parsed data set.
So let's take our single value and then print this out.
From that.
We get in this error for AI in this, let's run that.
And there we go.
As you could see, we have our input and then our output label right here.
Now the next thing we could do is specify the batch size.
So let's have this batch size here we have our batch configuration and then batch size, that's fine.
And then we could do some pre fetching.
So let's carry out the pre fetching.
Let's just do auto tune.
Auto tune.
And then run this again.
There we go.
Now, we should have 32 elements.
So let's look at our batch, our parsed data set.
It looks fine.
You see, we have this four different dimensions, and then we have our output here.
So that's it.
We now have our parsed data set and we're ready to train.
So as we've said already, it's important for you to save this in some location.
Let's say, for example, in the drive such that the next time when you want to do training, all you
need to do is start from here.
So all you need to do is to reconstruct this.
So you just come and reconstruct this data set, you will parse it, and then you're good to go with
this parsed data set right here so you no longer have to load all these images from memory and all of
that.
Now, before we move on, you should also note that while encoding the image or while encoding the data,
that the image and the label, what we could also do is take the arc max of the label.
So if you have a label, so let's suppose we have an input image like this and then you have an output
label, say 010 instead of taking the constraint the labels to be this, we could take the position
with the highest value.
So this is 012, so it turns to one.
Now if we have 001, then after getting the max in, this case is going to be two because we have zero
position or first position, the composition and this is the one with the highest value.
So this is another way of encoding our labels and that's what we are going to do.
So we will take this and rerun again and create our dataset or create our TensorFlow records and we
are still going to run this cells again.
So let's rerun this.
And we have our past data which you could see here.
See now it's different from what we had before.
So now we have the image or we have the images and then we have the labels.
Then we could go ahead and run the model.
We have the loss function defined.
Now the loss function is a sparse, categorical across entropy.
And this is simply because instead of the one hard notation that is, instead of representing our outputs
like this, for example, we converting them into this single integer.
So like this one, for example, would be two.
If we had 100, then this will be zero.
If we have 010, then this will be one.
So we've seen already that when we have this, these kinds of outputs, then we use this parse categorical,
cross entropy.
And so that's it.
We have this pass accuracy, categorical accuracy.
And now let's go ahead and start with the training.
So there we go.
You can see that train has begun and we train, as we usually do.
So that's it for this section on TensorFlow Records.
See you in the next section.