In this lecture I want to discuss two different approaches to transfer learning both of which we will
cover in the subsequent coding lectures.
The main issue I want to cover is this Imagine that the body of our neuron that work is very large.
Let's say hundreds of layers the head as discussed previously can just be a single logistic regression
dense layer.
The issue is even though we are not training any of the weights in the body it still takes time to compute
an output prediction.
Just imagine a really big long nested function at each of these layers.
We have to do a convolution which is faster than matrix multiplication but still can take a non-trivial
amount of time.
Imagine this computation in two parts in order to calculate the output prediction.
We first calculate a feature vector z using the body of the neural network.
Then in order to calculate the output we take Z and pass it through a logistic regression.
My issue is the first part of calculating Z is what takes a long time.
Now consider that our gradient descent loop works as follows.
We loop through each batch of data for each batch.
We calculate the up prediction y.
Then we calculate the greening of the error with respect to the logistic regression parameters W and
B then we update a W and B using those gradients.
The thing we want to think about is this Does the calculation of Z ever actually change since he is
the output of the V G network.
After passing in our input data it doesn't make sense to put this inside our loop because it's going
to be the same every time.
Remember the G weights are not going to be trained.
So here's what I propose.
Remember my rule all data is the same.
The idea is before we even begin training let's kind of hurt our image data into a tabular matrix of
feature vectors z then all we need to do is run logistic regression on Z without ever looking at the
V e.g. network again.
By doing this we can avoid having to pass our data through any pre trained network which can take a
lot of time if it has a lot of parameters.
All right so there is a problem with what I just proposed.
What is it.
Well remember that when we create our image data generator it gives us the option to do data augmentation
with data augmentation.
Every time we loop over the data set the generator modifies the original images just a little bit to
help our neural network generalize better.
Of course if X is different on every iteration of the loop then Z will also be different on every iteration
of the loop.
In this case we cannot pre compute Z before the training loop begins.
The only time it makes sense to do that is if we don't care about doing data augmentation
so these are the two approaches approach no one will be to use the image data generator with data augmentation.
This means we have to put the entire CNN computation inside the training loop approach.
Number two will be to pre compute the feature vector z on the original data X without data augmentation.
This means that after transforming the data all we need to do is train a logistic regression model.
There are pros and cons to each approach.
Let's list them out as discussed earlier.
When we do use data augmentation that means we have to recapture the features for every batch we see.
This is slow especially if you have a large network.
Yes it's faster than training the entire network but still slower than only training a logistic regression
on the other hand.
Data augmentation may help your model generalize better without data augmentation.
It's basically the opposite situation.
You can pre compute the features before training which will save you a lot of time training logistic
regression is extremely fast yet.
This also means that you cannot use data augmentation which may cause your model to perform some optimally.
However it's always worth trying both to see which method gives you the best results for your specific
dataset.