What's up, everyone?
And welcome to this new section in which we'll build callbacks with TensorFlow.
In the section, we'll look at how to build a callback from scratch by inheriting from the callback
class.
And then we'll build other callbacks made available by TensorFlow like the CSV logger.
L is topping learning rate scheduler model check pointing and finally reduce learning rate on plateau
callback.
Don't forget to subscribe and hit the notification button so you never miss amazing content like this.
Callbacks are methods we call during training evaluating our prediction.
This callbacks can permit us extract useful information from those processes we just listed, or even
carry out changes on those processes.
You're in this TensorFlow documentation we have to TF Keras and then we have the callbacks.
So this is what we have for the documentation.
We could go through each and every one of this, but for the sake of this course we are going to look
at the key ones.
So right here we have this callback.
But before get into this, we are going to look at the history.
The reason why we're looking at history first is actually because we've used it already.
So we use this callback without really knowing that we're using callbacks.
So yeah, we're told this is a callback that records event into a history object.
So remember the times we were having this year?
So after training, that's when when we're training, we start some information in the history.
And then after training, we're able to come up with plots like this because we have this information
stored in this history right here.
So that said, as you could see in this example, which is kind of like similar to what we've been seeing
so far, we have this history and it collects all values we been storing during training and then we
could print out this params history dot params, we get the params and then we could also get the keys.
So that's how this works.
Will send us already.
And then we could look at this callback class right here.
Now, the reason why this is like the most important of all these different callback classes is because
this is kind of like the master class or that abstract base class which can be used in building new
callbacks.
So if you want to build a callback which isn't listed here, you could always get back to this and then
build that callback from scratch.
So here we have this callback class which has attributes, as you could see, params and model, and
then it also has this methods which we'll see how to implement very easily.
If you look at this on batch begin and this on batch and you'll see that the take similar arguments
like here we have the batch and then we have this logs.
Then for the next the epoch begin and the epoch and the actually taken the epoch and the logs that set
will impart the callback.
So here we have from TensorFlow Keras callbacks, we're going to impart callback this class which we're
going to be using in creating our callbacks.
We run that, then we define this callback class, which we'll note as called last callback.
We have this last callback class which inherits from the callback which we've just imported, and then
we will make use of the different methods which have been given to us in the documentation.
Let's start with the on epoch and so on.
Epoch.
And we're taking the epoch.
And then what we're going to do is we are going to print out the last values at the end of an epoch.
So what are we doing is kind of similar to what we have already here.
So let's do that and have print epoch number.
Let's say epoch number this and then has a loss of four epoch number.
This method, the model has a loss of this and then we just format so there we go, we pass in the epoch,
let's have this here and the locks.
So just as in the documentation where we add in the epoch and the logs, now we are going to pass in
the epoch and then the logs.
But since we want to get just the loss, we could get the loss from this.
So we have this dictionary here and we pick out just the loss.
Now let's run this and then we'll see how to include a callback in the training process.
So just right here, we're going to have callbacks.
So the callbacks argument and we have this list.
So in this list we're going to insert this callback we've just created here.
So we have the last callback we should just pass in here and that's fine.
So now let's rerun this.
Let's take this for, say, three epochs and then see what we get.
We get in this error, we click search StackOverflow and then click on this.
You see that we have.
Solution.
Just your and what is said here is what you're going to pass is the object and not the class itself.
So that said, instead of passing the last callback as we just did, we're going to pass on this object.
So we're going to put in the brackets.
Let's modify this print here.
And personal space.
So we're going to have that or two next line and then we run that again.
So let's run that and that's fine.
Then after training for over three epochs is where we get we see that unlike before where we just had
this output it, now we have this message output that is for epoch number zero.
The model has a loss of this for epoch number one, and then for epoch number two.
Now what we could do is we could add this.
So we have this formatted normally, so we don't start from zero, but instead from one.
So we could say plus one right here.
So that's it.
Now another thing we could do is we could have on batch and so on batch, and we have that and then
we have the batch we passing the lox.
This time around, what we want to do is kind of similar to what we've seen already here.
So we just print this out.
That's fine.
Take that off.
So for the batch number, for this batch number, the model has a loss of this.
So there we take in the batch.
So that's it.
And then we log out the loss.
Now, Yeah, let's just pick out this loss.
Totally.
So we have the locks and then that's fine.
So yeah, we've put in the batch like, Yeah, well, passing the epochs, we run this again and run
this, run this.
You'll notice that this time around we have much more information which has been locked out.
Let's take this here and get right to the top.
What we have here is for batch number one, the model has this loss.
So you see that this is after each and every batch.
So here's the first batch.
We have this log, and then the next batch we have this log and so on and so forth.
And since our batch size equal to the simply means, after working on 32 different data points, we're
going to have this locked out.
And then the next 32, we have this locked out right up to the end.
Now for the epochs, you will notice that as we go on.
So we're moving on to let's get right up to 689.
We have that.
And finally here we have 689.
Okay.
So for this last year, last at the end of the epoch, we now have this locked out.
So here we have four epoch, whereas previously we had four batch four for all the different batches.
In case you want to understand how we got this six, eight and nine right here, you could take the
total dataset size and then divide by the current batch size, which is 32.
You should get this 689 which is been given right here.
So from this we can look at the CSV logger.
Now with this here as a logger, what we're actually doing is we are logging out this information in
a CSV file or in some file, which we're going to define.
So what we do is just simply copy this.
Now, this time around, we wouldn't actually have to recreate a class as we just did with the callbacks
because this kind of like a general way of creating this callbacks now with the CSV logger will actually
create the callback more easily.
So here we have this and then we're going to specify the file name.
So here let's have the CSV we call back equal that, let's take this off and we specify our file name.
So yeah, we just have our file or locks dot CSV and that should be fine.
So we take that off, take that off and then up right here we have to impart C as V logger.
That's fine.
We run that everything is okay.
We get back to our callback and then we run this cell right here.
Now, note that this append is as described in the documentation, a boolean which tells us whether
the locks we are currently putting in the CSV file or in the file in general are going to be appended
on previously locked content or not.
So when we have this as false, we're supposing that this is empty and so we are going to like be put
in information in this file for the very first time.
So we've run this and then now all we need to do to take this new callback into consideration is to
have your CSV call back.
So let's make sure, let's take this batch locks out from here so we are not going to take this last
callback.
To conservation.
And the longer we run this, after three epochs, we're going to open this up and then we have this
logs of CSV file which has been created.
So let's have that.
And as you could see, we have this as file, which we could now download and then view it later.
So here we have the accuracy AUC and all the other metrics and lost values which we want to store.
So that's fine.
And next thing we could do is we could get back again here and then I'll select append.
True.
So if we've done training the first time, I want to redo the training process.
We don't want to erase all the values we had previously.
So here we set this to append true around this.
Again, after three epochs we open this log dot C has the file.
And what do you have here?
You see, we have this new information which is just been appended on the previous information from
this CSV logger.
We could now move on to this early stopping callback right here.
To better understand l is talking.
Let's get back to this plots which we had previously.
So let's take all this plot of the models accuracy where we see how the train accuracy keeps increasing
while after a certain point let's see this points here.
Let's take this off after say, this point our models or even this point here.
Our model's validation accuracy doesn't increase any further.
So what we have in here is something like this.
We have this training accuracy which increases and goes towards one, and then the validation accuracy,
which is something like this.
Now, in some other cases you will even have situations where this starts to drop.
Nonetheless, in this case, we have this plot where it just kind of like stabilizes and doesn't increase
any further.
Now, note that this kind of situation is known as overfitting in overfitting.
The model starts to overfeed the training data.
So because the model has been trained on the training data and not on the validation data at certain
points, the model stops or ceases to generalize because the aim of this training process is not to
come up with a model which only performs well on the training data.
We're trying to come up with a model which performs well on any type of data, be it a train, be the
validation or the test data.
So if we're able to have a model which does the same or which has the same performance with a train
and with the validation with the test, then that model is an ideal one.
But in this case we see that as we keep on training, the models parameters have been modified to suit
only the training data.
And this is very dangerous because at a certain point you may feel like because you're having high trained
accuracy, your model is performing well, whereas this isn't the case because when your model will
be showed new data like the validation in this case and the test later on, this model wouldn't perform
as well as it will do or as well as it's doing with the training data.
So to avoid this kind of false measurements, we tend to stop the training once this overfitting starts
to occur.
So this means that if you training and then your validation, let's take let's suppose we stop in at
this year.
So we're training and then your validation data or your validation accuracy seems to be constant, whereas
that of the training seems to kind of increase, then it's better for you to stop training at this point
because after this point the model parameters are just been modified to suit the training data and doesn't
really generalize, which is the case here, because we're trying to extract some information from this
data and make the model intelligent.
So the model doesn't become intelligent by only modifying its weights or parameters based on the data
it's been trained upon is intelligent because after being trained it can perform well on data it has
never, ever seen.
So yeah, we have this l is stopping where after we notice that this the validation accuracy doesn't
seem to increase any further.
We just kind of like stop the training and then use the model parameters from this number of epochs.
So we could say that after, say, 12 epochs, we just stop the training.
Now let's take this off and then do replicate something similar for the loss.
So if we're having a loss, we could have something like this and then that.
So yeah, we have the number of epochs and then we have the loss.
So you could have a situation where you have in your training data which your training loss which keeps
reducing, whereas for the validation you see you would have something like this.
So this is a typical plot for overfitting.
Nonetheless, in our case, or the model of our face, but not that much, in some cases you will have
a situation where this event starts to drop and where the loss starts to increase.
After a certain point there is a validation loss and then there is a training loss.
Obviously, the training loss will always keep reducing because we are training on this training data.
So what we're saying is at this point where the validation stops reducing, it's important to just stop
this training and this is known as early stopping.
Now recall that the aim of callbacks is to be able to modify the training process, the evaluation process,
or the test process as prediction process.
In an automatic manner.
So that said, we shall make use of this early stopping callback right here, which will permit us stop
training automatically once we notice that a given parameter like, say the loss, the validation loss
doesn't drop any longer.
So here we're just going to copy this.
And then we just applied similarly to what we had done with the CSV callback.
So yeah, we add this text and then we add that code, we just paste this out.
Now we define this is callback that's early stop callback.
And and then we'll look at the significance of each and every one of these arguments.
Now, coming back to documentation, we have this monitor quantity to be monitored.
So by default, yeah, we have this vowel loss.
This means that this callback will simply check on this validation loss right here.
And then once it stops reducing, like, see how this points, we're going to stop the training.
Whereas if we change this to say validation, precision or validation accuracy, then what will be monitoring
will be that accuracy of the precision value.
So if you happen like this, we'll see.
We'll stop around this right here to ensure that we don't go and over feed on the training data.
The next argument is this mean delta argument right here.
So with the mean Delta argument, we are defining a minimum change below which any change is considered
as no improvement.
So if we have a loss like this and that our mean delta is say, a value of 0.1, then even if this loss
reduces by value of 0.5, then this callback will consider that there has been no decrease in the loss
because the mean delta is 0.1.
Now, by default, the mean delta is set to zero.
This means that any slight change is considered as a drop.
So if we have even 0.0005, then we consider this as a drop in the loss.
Now, this is important because this has a patients this callbacks make use of this patients with those
patients we are defining the number of epochs above which if we don't have a decrease in the validation
loss, like in this case of the validation loss, we consider that we could stop that training process.
And for the accuracy is the number of epochs above which if we don't have an increase in the validation
accuracy, if we've picked validation accuracy for the monitor then would have to stop the training.
So we define this or we predefined this so that this could run automatically.
Now, for the mode by default, we have the auto mode, but we could specify mean or max.
Note is that when speaking about the loss, we spoke of a value or the number of epochs above which
if the loss doesn't decrease.
So yeah, we're supposing that the loss is meant to be decreasing and in that case we are having a mode
of mean.
Now for the accuracy we spoke of for the patients a number of epochs above which if the accuracy doesn't
increase.
So in this case we have in the max Now what TensorFlow permits us to do is to use an auto and what is
auto TensorFlow automatically infers whether it's dealing with a mean or max.
So this means that if you place in, for example, valid position, this auto should be able to understand
that a position should be increasing.
So it's going to use a max.
Then we move to the baseline where the training stops.
If the model doesn't show improvement over the baseline, then finally we have this restore best weights
with the restore best weights, which by default is false.
We are simply saying that the model is going to take up its final state.
So this means that if we start monitoring the model, say at this point here or at this point where
we have the lowest possible loss, and then the model, let's say we have a patients of five, so we
are going to train for four or five epochs before stopping.
So if after five epochs we are on this, let's suppose that the epoch we are on this.
So we've added from here plus five epochs, we are on this and then we are having this loss is clear
that this model with this loss is less performant than this one.
Now, if this restore best weight is set to false, then we'll just take the model's weights here or
the model weights which give this loss value here.
Whereas if it's set to true, then it means we're going to take the best weights we've had throughout
the training process and which happens to be the weights which provide this loss right here.
That said, here we go.
We have this.
We have, let's say the patience to to re verbosity one or mode auto baseline non restore best is false.
So let's run that and then all we need to do is to just include this year.
So yeah we're going to have is call back now we could take off the as we call back you could always
put all this together, let's just let it so we could see how all that works.
So we just have this list right here and we have this is callback with the CSB callback.
Now we run the training.
We didn't train this for long enough to be able to observe any callback changes.
So let's take this to ten epochs and then we reduce this to one or two.
Let's take this to two.
We run that again, so that's fine.
And then we fit our model.
After training for eight epochs, we see clear how the early stop and callback stops the training process.
Now, let's understand why this is all this training process has been stopped.
If you take take a look at this validation loss right here, you'll find that there was a drop your
drop increase.
But after this increase, there was a drop.
And since the patience is equal to that's the patience we had defined here, equal to we have to get
two successive increases or two successive.
Same loss values before the training process could be stopped.
So since we after this, we have a drop.
The training process continues.
Then we have this increase and then we have this drop.
So it continues.
Then here we have this increase and then here again we have this other increase.
So because now we have had this two successive increases, as we could see in the plot right here,
you see right here.
Well, we have this increase, drop, increase and then increase.
We now have the training process, which is being stopped, as you could see here.
That said, we now move on to the learning rate schedule.