So in this lecture, we will be looking at the code to implement a recommended system using TFI Taf,
as mentioned in the previous exercise prompt, you are strongly encouraged to call this yourself first.
So if you haven't yet done so, this is your last chance to attempt the exercise.
Otherwise, let's move on.
So we'll start by downloading our data set, which is a database of movies.
The next block of code contains our imports.
Note that in this script, we'll need the JSON library since some of the data is stored in JSON format.
Furthermore, note that we've imported cosine similarity and Euclidean distance despite the fact that
these are easy to implement ourselves.
These functions are efficient at doing many computations at once, so it's usually beneficial to use
the built in methods.
The next step is to read on our data using PD that reads GSV.
The next step is to call the overhead to see what's inside our data frame.
OK, so as you can see, the data frame contains quite a few columns, including budget, genres, homepage
keywords and so forth.
Note that some of these columns are stored as JSON strings.
You're encouraged to look for this yourself to see what data you might want to use.
So the next step is to explore our data set format.
We'll begin by retrieving the first row of data which can be done using AI Luke passing in the index
zero.
As you can see, this prints out a panda series with each of the column names along with the corresponding
values.
The next step is to print out the genres.
This is one of the columns will be using to build our text.
OK, so as you can see, this is a JSON formatted string, in particular, it's a list inside the list.
We have individual JSON documents and each of these items has two keys which are ID and name.
So clearly we are interested in the name attribute, which contains the actual names of the genres or,
in other words, what we would simply refer to as the genres.
Note that some genres, like science fiction, contain a two words in order to treat these as a single
token.
We may want to remove the space and treat it as if it were one word.
The next step is to look at the keywords.
This is another one of those columns will be using to build our text.
OK, so as you can see, this is a much more expansive list, but note that it has the same format as
the genres.
It's a list of JSON documents and inside each document we're interested in the name attribute.
OK, so the next step is to convert the JSON string into a format we can actually use in Python, namely
a Python list of Python dictionaries.
We'll do this by calling JSON downloads.
The next step is to write some code to demonstrate how we will convert one of these genes into a single
line of text.
As you recall, this is what is required by the TF IDF vector razor.
So what's going on here?
Well, let's start with the fact that Jay is a list we have a for loop over.
So we are looping through this list.
Each item in the list is represented by the variable double J.
OK, so what are we doing with Double J?
Well, as you recall, this is a tiny dictionary and we want the value stored in the name attribute.
So normally all we would have to do is call double J Square bracket name, but we can only do that.
As you recall, some genres such as science fiction, contain multiple tokens.
We would like to treat them like a single word.
One easy way to do this is to split the string on whitespace and then join it back together using an
empty string.
What this will effectively do is just concatenate each individual token into a single string, removing
any whitespace.
Finally, note that we do this for every genre in our list of genres.
The last step is to join other genres together with a single space in between.
OK, so as you can see, the result is, as expected, all the genres now appear in a single string
separated by a single space.
Note that genres like science fiction, which contained multiple tokens, are now joined into a single
token.
OK, so the next step is to put what we just did into a function.
We're going to do this for both the genres column and the keywords column, and then we're going to
contaminate the results into one string.
So we'll start by defining a function called the genres and keywords to string.
This takes in a single row of our data frame inside this function.
We're going to call JSON downloads for the genres column, as we did above.
The next step is to join all the genres into a single string using the same code as above.
The next step is to do all the same steps to the keywords column.
As you recall, both of these columns have the same format, so the same code will work.
The final step is to join the genres and key words together into a single string.
The next step is to call DFA to apply, which will run the function we just wrote on every row of our
data frame.
One at a time.
Note that will assign this to a new column called String.
The next step is to create an instance of the TF IDF Vector isAre class note that have set the MAX features
to two thousand, which will limit the number of columns in the final matrix.
You're encouraged to read the documentation if you want to learn more, but basically it keeps the most
frequent terms in the corpus.
The next step is to call fit transform on our data set, as you recall, our text is stored in a column
called String.
Note that for this example, we won't have a train and test set since that doesn't reflect how this
would be used in the real world.
In practice, we would have a single database of movies, and our free idea vectors would be trained
based on whatever we had.
The next step is to print out acts just to see what we get.
Note that we did this for count victories there as well, so it would be interesting to see if the result
is still the same.
OK, so as you can see, it is in fact the same as X is still stored as a sparse matrix.
Note that the dataset has about 4800 rows and two thousand columns.
And the reason it has two thousand columns is because we set max features to this value.
In other words, we could have had more columns, but we've decided to throw out less frequent terms.
You're encouraged to experiment with this value and see what impact it has on the results.
Now, the next step might seem a bit strange, but recall that we are now working with a matrix of numbers.
It's not obvious which row of The Matrix corresponds to which movie.
Of course, our data has been processed in order, so the movies in our data frame will correspond to
the vectors in our RDF matrix.
Now, for reasons which may not be clear yet, it would be useful to have a mapping that tells us for
a given movie, which index does it have?
Luckily, the way to do this is relatively simple.
Note that in our data frame, the index of this data frame is already a list of integers starting from
zero and counting up by one.
This, coincidentally, is also the same as how we count the indices in an array, such as our TFI D
of Matrix.
Therefore, all we have to do now is create a panda series.
The values in this series will be D after index that is zero, one or two and so forth.
The index for the series will be the title of the movies, so we're kind of reversing what our data
frame already does.
Our original data frame has integer indices, and this points to the movie.
This new series has the movie title as the index, and it points to the original index as the value.
OK, so as you can see, the titles are stored in the index column and the values in the series are
just zero one two and so forth.
The next step is to see how our movie to ADX mapping will be used and also to use this to make a recommendation.
I've chosen the movie Scream three, but you're welcome to choose any movie you wish.
OK, so notice that this gives us back the index where this movie was stored.
OK, so why do we even want this index?
Luckily, that's the next thing we're going to look at.
As you can see, this index is used to grab the correct arrow inside our ETF IDF matrix, which we've
called X.
This is the TF IDF vector that corresponds to the movie screen.
Three.
OK, so notice that when we print out this query vector, we get back a sparse matrix of size one by
2000, which makes sense.
We've essentially just grabbed a single row of our TFT of Matrix.
Now, suppose that you'd like to see the values that are held within this matrix.
One simple way to do this is to call the two array function.
So as you can see, most of the values are just zero, as you may have expected since this is a sparse
matrix.
The next step is to compute the cosine similarity between our query vector and every vector in X.
Note that this includes the query vector itself because the query vector came from X.
This is not always the case.
For example, if you have a search engine where the user can type in their own data.
However, for the purpose of this lecture?
Well, assume that our system is like Netflix, where we base our recommendations on movies which are
already in our database.
OK, so this gives us back a variable, which we will call scores now just as an exercise, let's think
about how many scores there will be.
Remember that this function does a pairwise similarity.
So if I had a 10 query vectors and 20 vectors to check against, then my result would be 10 by 20.
In this case, I have one query vector and 48 vectors in my database.
Therefore, the result will be approximately one by 4800.
OK, so note that most of the values appear to be zero.
This makes sense since if two movies don't share any common terms, their product will be zero, which
means the cosine similarity will also be zero.
Now, because our array is of shape, one by end, we would like to flatten it to be a one degree.
The next step is to plot out scores just to see what they look like.
OK, so as you can see, it appears very noisy with one big spike.
Of course, this big spike is just the scary movie, since the cosine similarity between the two of
the same vector is just one of the other similarities are much smaller, with a max around zero point
four.
Note that because the movies are in a random order, it's difficult to tell where we have zeros.
Now we know that we would like to do something like sort the scores, but this is not exactly what we
want.
Firstly, we know that sorting by default usually results in the items going in ascending order.
In our case, we want them to go in descending order with the most similar item at the front.
This makes sense since if we want the top five matches, we'll just need to take the first five values.
The second issue is that we don't really want salt because we don't care about the score itself.
We instead want our expert, which tells us which order the movies go in if we sorted by the score.
Again, we don't care about the score values only how they rank amongst one another.
The next step is to plot the scores after sorting them.
Note that we can simply index the scores array by the previous results.
This is because with Nampai arrays, you can index them with other arrays, provided that the index
array contains the appropriate values.
OK, so this result makes much more sense.
Of course, the top scores one, which is just the query, we then have a few hundred movies which are
partial matches with a score less than one but bigger than zero.
Finally, we see that most of the movies are the score of zero because they are completely unrelated
to the query.
OK, so the next step is to actually retrieve our matches.
In fact, we've kind of already done that.
All we really need to do is take the sorted indices and index those from positional one up to position
six.
Recall that we don't want to start at zero because the first movie is just the query itself.
We'll call the result recommended ADX.
Now, currently, these recommendations are just integers.
What would be more meaningful to us is to see the actual titles.
Of course, as you recall, these indices map back to our original data frame.
Thus, we can simply call the iLoad function passing in recommended RDX, since we only care to see
the title.
We can also grab only the column called Title.
OK, so as you can see, these results seem promising for screen three.
We get to Friday the 13th movies Graduation Day, the calling and the glimmer man.
These are all thrillers, which makes sense.
You're welcome to look these up for yourself to confirm that this is the case.
OK, so because we don't want to have to do all that work again, to make new recommendations for other
movies, we're going to write a function to encapsulate all the previous work we just did.
So let's write a function called recommend that we'll take in a single input, which is the title of
a movie.
Well, assume that this title always exists in our database.
The first step in this function is as before to grab the index for the movie title.
Note that one thing we didn't discuss previously is that the Pandas API is a bit inconsistent.
Specifically, if your movie title is the same for multiple rows, the result will not just be a single
index.
In that case, you'll get a panda series, which is a completely different type of object.
So the next step is to check the type of RDX.
If it's a panda series, then we know that there were multiple of the same title.
So inside the CIF statement, we simply grab the first item.
Note that this is arbitrary.
Another option would be to ask the user which one they want to choose or simply combine the results
of all the movies with that title.
Of course, in a real system such as Netflix, you wouldn't necessarily be searching by title, but
instead you would have the idea of movies the user actually watched.
So in that case, there wouldn't be any ambiguity.
Note that the remaining steps in this function are the same as what we've already seen, so I'll just
review them quickly.
The next step is to use RDX to grab the corresponding TF IDF vector from X..
We'll call this our query.
The next step is to apply the cosine similarity function between the query vector and all of X. We'll
call the results scores.
The next step is to flatten the scores so that they become a 1D array.
The next step is to sort the scores in descending order and to grab the top five matches.
Excluding the element at index zero, the final step is to index our original data frame with the recommendations
and to return the titles for those recommendations.
OK, so the next step will be to test our function using Screen three once again to see if the results
will be the same.
OK, so luckily, the results are still the same, which means our function works.
Next, let's check the recommendations for Mortal Kombat.
OK, so these results also look pretty good.
First, we see another Mortal Kombat movie, which makes sense.
We also see dead or alive in the name of the king.
Three.
Street Fighter and alone in the Dark.
Now you probably don't recognize these if you're not into video games, but rest assured.
These are all movies based on video games.
Even more dead or alive and Street Fighter, just like Mortal Kombat, are video games from the fighting
game genre.
In fact, there was a big rivalry between Street Fighter and Mortal Kombat back when arcades were popular
in the 90s.
So these results make a lot of sense.
Finally, let's check the recommendations for Runaway Bride.
OK, so again, these results also look pretty good.
There is House of D, My Big Fat Greek Wedding two.
It happened one night and education in our family wedding again.
You've probably never heard of these movies unless you're into that sort of thing.
But if you look them up, you'll see that these are all movies that fall into the romance, comedy or
drama category.
And of course, there's a lot of overlap between these different categories.
For example, a romance is often also a drama, or you'll have a romance comedy.
At least three of these movies are about weddings, at least from the titles.
The query movie has bride in the title, while my big fat Greek wedding and our family wedding have
the actual word wedding in the title.
So again, these recommendations appear to make sense.