-: Hey, welcome, and in this video you're gonna learn,
what are embeddings,
how to use them, and their various use cases.
Embeddings are basically a numerical representation
of a word, or an image, audio, or a video.
You can think of a embedding as a representation
of that piece of data
in a high dimensional vector space.
Vectors are able to capture semantic meaning,
making similar concepts closer together mathematically.
They're a core piece of technology
that enable lots of different AI application.
Embeddings work by breaking text
into tokens, words, or sub words.
You can then also can convert tokens or images
into these high dimensional vectors,
and that gives them a position in space.
So even though they might have a thousand different columns
or dimensions, they have a mathematical space in time.
You can then compare different types of vectors.
So on the left here,
you can see we've got queen being compared with king,
or woman being compared with man.
And we can do mathematical similarity differences
between these different types of vectors.
There's lots of different embedding models
that you might want to use.
OpenAI has some which we'll be covering.
You also have cohere, sentence transformers and Google.
You can also get open sourced embedding,
so you don't have to pay for them
apart from the computation it takes
to generate the embedding.
There's lots of different types of applications,
so one of the main ones is semantic search.
So on the right here, you can see we have a query
and we will then use an embedding model
and we will basically query a bunch of vectors
and find vectors that are similar
to our original user query.
This brings us onto this idea of RAG,
where we can take a query, do an embedding on that query,
then we query a bunch of vectors
that are stored in a database.
We bring back retrieved contexts into the LLM's prompt
or chat messages, and then we return a response.
And the idea behind this
is you do retrieval augmented generation,
so doing a retrieval step
before you actually generate content
using a large language model.
There's also other ways that you can leverage embedding.
So for example, text classification,
where you can have different types of categories,
either positive or negative.
You can generate different types of embeddings,
and you can use those
to power content recommendation systems.
So if we're looking at embeddings for OpenAI,
you just import the package,
and you just do client.embedding.create,
you have your input
and your embedding model that you choose.
Now, once you get that out,
you'll see you get back a structured dictionary
with some data.
And this is the embedding here, this embedding here, right?
And you can see it's just a bunch of numbers.
And basically, that is your numerical representation
of this input text on the left.
There's different ways you can measure similarity
when it comes to embeddings.
The main three are Cosine similarity,
which measures the angles between vectors,
Euclidean distance, which measures straight line distance,
and Dot product,
which is the simple multiplication of vectors.
The main one is Cosine similarity,
so the equation is on the right here,
but basically, Cosine similarity is the main way
that we do embedding similarity metrics,
and it ranges from negative one to one,
one being completely in the same direction,
and negative one being in the opposite direction as well.
You've got a coding example here,
so you can take two NumPy arrays
and calculate the Cosine Similarity between those two.
And you can see that these NumPy arrays are quite similar,
so you end up with a Cosine similarity score,
which is close to one.
There's also lots of advanced techniques
and future directions,
so people are doing things like fine tuning embeddings
for specific language and applications in industries.
You also have multimodal embeddings like clip
and image bind that can create unified vector spaces
for both text, images, and audio.
And you've also got hybrid approaches using embeddings
of traditional methods like BM 25 or term TF-IDF,
which is term frequency inverse document frequency,
so traditional keyword search with embeddings.
And the key challenge is keeping embeddings updated
with new knowledge without completely retraining them.
Okay, great, so you now know about embeddings,
a couple of practical use cases.
The next thing we're gonna do is get our hands on
and go through an embeddings notebook
and see how we can create embeddings
as well as how you can also use Cosine similarity
to compare different embeddings
and how similar those embeddings are.
So the notebook I'd like you to open
is this embeddings notebook in the OpenAI features
and functionality folder.
Open that, and then we're gonna go through this,
and we're gonna run through the code.
So the first thing that we're doing
is we're installing
OpenAI, tiktoken, pandas, scikit-learn, mapplotlib.
And we're then doing all these imports.
We're then gonna go and learn
how to generate an embedding in OpenAI.
So the first thing you wanna do is replace this section
with your OpenAI key.
Next, we're gonna create a function called get embedding.
It has a default parameter
for the model called text embedding-3-small.
We replace all of the new line characters,
and we then run client.embeddings.create
with our input in text queen and the model that we're using.
We're then returning a NumPy array
with the resp.data square bracket zero.embedding.
This gives you your embedding vector.
We can then put a piece of text in.
So for example,
"The quick brown fox jumps over the lazy dog."
We can then run the get embedding function,
and you'll see that we have a embedding
with around 1,536 dimensions.
And we've also printed out the first five dimensions.
Now, I will say that the larger the vector length,
ie, the more dimensions it has, generally speaking,
the more rich it is,
and the more context about that specific word,
or phrase, or paragraph, or document it has.
It does take more memory
to generate those types of embedding,
so they cost more and they also cost more to store as well.
We can also use tiktoken
to count the number of tokens before embedding.
So for example, this specific sample token count
had around 10 tokens.
You can then also generate multiple embeddings.
So we have our sentences here.
We're then using NumPy.vstack to vertically stack these.
And we're just doing a four loop here
in a list comprehension
where we're going over every item
in the censuses Python list.
We're running the get embedding function against it.
And then we are using something called PCA, which allows us
to shrink the embedding space to only two dimensions
whilst retaining a lot of the variance.
When we plot this, you'll actually see,
so we're gonna run this here.
We basically are calculating the embeddings,
and then you can see here,
we have all of these embeddings.
And notice how OpenAI
and machine learning are actually closer together
than the sky and hiking and restaurant is over here.
So you can see that the embeddings actually do have some
sort of spatial awareness of themselves.
Now, there's a quick exercise for you.
I want you to pick five of your own short sentences,
embed them using the get embedding function,
compute pairwise co similarities,
and identify the most two similar sentences.
Now, to do the Cosine similarity one,
what you're gonna use is this from Sklearn.metrics.pairwise,
import the Cosine similarity metric.
And then after you've imported out,
then we're gonna run similarity.
(keyboard keys clacking)
And then we'll just do Cosine similarity of the vectors.
(keyboard keys clacking)
And what you can see here is you end up with a matrix
of all of the different similarities
of each individual vector against every other vector.
Now, notice there's this one going down here
in the diagonal.
That is the metrics
of the vector Cosine similarity against itself.
You can also as well do the similarity
by just comparing two vectors,
so I can do vectors zero
and I can do Cosine similarity.
And then we have to reshape these to be the same
in terms of a negative one and a one.
And I'm just gonna make sure we get the right function here.
And you can see vector zero against vector one
had a Cosine similarity of 0.46.
Cool, so have a go at doing these four steps.
Pick some short sentences,
embed them using the get embedding function,
compute the pairwise co similarities,
and identify the most two similar sentences.
And a bonus as well,
visualize the embeddings
in 2D using the PCA Principle Component Analysis
machine learning model above.