So in this lecture, we'll be doing a very brief introduction to words of EQ, which is what will implement
in the next lecture.
One of the reasons I wants to do this is it allows us to use embeddings within it.
Otherwise we would have to wait until we studied CNN's in art ends, which will take quite some time.
This lecture will look at one specific variants of words of EQ, specifically Zebo, which stands for
continuous bag of words.
The other major variant is called Skip Gram, which is, in my opinion, a little more complex.
So to keep things simple, will be implementing Cibo.
So now that you know about aliens, the design of the Cibo network will be very simple at a high level,
what we are trying to do can be described in a single sentence, given a string of multiple words.
We're going to remove the middle word and use the surrounding words to predict the missing a middle
word.
This should sound very familiar if you've remembered the other things we've studied in this class.
So there are two ways to look at this.
The first way which appears in the word T-VEC paper shows a neural network with multiple inputs all
going through the same embedding layer.
This converts each of the context words into a word vector.
Once we have the vector for all the context words, we take the average, which gives us a single vector
with the same dimension.
Using this vector, we go through a final dense layer to get the predicted word.
So this is like an island with multiple inputs and a single output.
The second way to look at this is it's a sequence of context, words being passed through an embedding
layer.
This gives us a sequence of embeddings.
We then take the average of all the embeddings in that sequence.
We've not yet seen how to do this in TensorFlow.
But note that the average is a pretty simple operation not to be concerned about.
This gives us a single vector, which we can then pass through a final dense layer to get the predicted
word.
So from this perspective, the input is a sequence in the architecture is an and which handles sequences
as input.
So there are two notes to make about this.
Firstly, note how this is just multiclass classification, which you know how to do.
Secondly, note that there are some small details that aren't obvious.
The first detail is that although this is in and there is no activation function in the middle, it's
just to weight matrices one after another.
Therefore, this is effectively a linear network.
But unlike a regular linear model, there is what we call an information bottleneck, which forces data
of large dimension to fit into a vector space of small dimension.
You can think of our neural network as having an hourglass shape.
The input has size V, which is our vocab size.
But the middle only has size D, which is the embedding dimension.
The output once again has size V since we need to be able to predict every possible word.
So the network goes from big to small, back to big.
This is a common theme in deep learning, which usually results in useful representations being learned.
In fact, we saw something similar when we studied LSA when you reduce dimensionality.
The result is useful.
The second detail I want to mention is that both layers of the neural network do not have bias terms.
The embedding layer doesn't have a bias term.
And the final dense layer also does not have a bias term.
But essentially, this is all you need to know to implement Cibo.