In this lecture we are going to work on a notebook to just do tax free processing so we can explore
a little bit how it works.
This lecture will be useful before we move on to the next script where we will assume that we already
know how it works.
As usual you can look at the title of the notebook to determine what notebook we are currently looking
at.
OK so we're going to start by making some fake data since we eventually want this to be saved as a CSI
V.
It's convenient to create a data frame but in order to do that a convenient way to specify the data
is in a dictionary.
So in this dictionary the key is refer to our eventual columns and the values are lists that correspond
to each row in the data.
So we're going to have three samples in our dataset.
The labels are 0 1 and 1.
The sentences are I like eggs and ham eggs I like in ham and eggs or just ham I've added some punctuation
to this dataset so we can see what happens to it in our token either.
Next we pass this data into the data frame constructor to instantiate a data frame.
Next we do a DFT head so we can look at the data frame.
Hopefully this is what you expected to see
next.
We save the data frame to see as we file by calling the two C as we function we can also use the command
line version of head to see the contents of the file.
Next we're going to create our 2 field objects one for the text and one for the label for the text.
I've set sequential equally to true since our data is a sequence of words I've set batch first decoder
true since I want the data to be n by something rather than something else I've set lower equal to true
so that the data is lowercase I've set token I's equal to spacey.
This might seem odd to you but specie is another an IP library in Python that is somewhat tangentially
related to PI storage.
Basically if you leave this argument unsaid it's just going to use a string that split by default.
I would recommend you try both to see what the differences are.
I wanted to use space here to show you how it deals with punctuation.
Finally we set pad first decode a true so that the padding goes at the beginning of each sentence rather
than at the end.
Next we set the label field.
Here we set sequential equal to False.
Since this column does not contain sequential data we say use vocab to false.
Since this column contains no vocabulary and it's just integers and finally we set is target equal to
true.
Since this is the target column
next we create a tabular data set object where we pass in the C as V file.
We created earlier we said format equal to see as V since it's a C S V and we set skip header equal
to true since Panisse saves the header row by default.
We set the fields argument to a list containing the fields we just created one important note is that
these fields must be specified in the same order as they appear in your file.
So if your file contains the data column to the left of the label column you would put the data fields
first
so if you check the attributes of the data set object you'll see it has one called examples which is
a list of example objects.
What I've done here is I've grabbed the first one so we can see what's in it.
As you can see there is an attribute called data.
And if we look at this attribute we can see that it contains our sentences but not in their regular
form.
In fact our sentence is now tokenization.
You can see that the period has been separated from the word ham which would not be the case if you
just did a plain string that split
if we look at the label it just returns zero as we would expect
next.
I'm going to split the dataset.
This is kind of weird in our case since the data only contains three samples.
But let's just pretend this is OK.
Next we're going to call the bill the vocab function on our text field and pass in the train dataset.
Oddly this is not done automatically when you create the dataset object.
Again it's not a mature library
next we call the vocab attribute and check its type.
We can see that it is a vocab object
next we call that attribute as to AI which returns a dictionary since our dataset is so small which
I did on purpose we can see the entire dictionary as you can see each token in the dataset is assigned
a unique integer.
Also punctuation is considered a token since we use the space tokenization we can see that the unknown
token gets assigned the index 0 and the Pats okay and it gets assigned the index 1.
We can also see that all the words are lowercase
next.
Let's look at the eye to s attribute.
We can see that it contains a list of our tokens.
You should confirm to yourself that the indices of each element in this list corresponds to the values
in the dictionary above.
Next we create a device object as we normally do.
Next we create an iterator from our datasets.
We set the bad size equal to 2.
Since both the train and test set only have a maximum size of 2
next we loop through the train iterator and print the inputs and targets along with their shape.
As you can see the shape of the inputs is 2 by 7.
Next we do the same thing for the test iterator and the shape of the inputs is 1 by 6.
So what do we learn from this.
Well we learned that pi talks already intelligently makes your backsides have the smallest sequence
length possible.
There is no unnecessary padding in the train batch or the test batch.
This is good since for example if our data has a maximum sequence length of 10 but the sentences in
our batch only have length 2 or 3 then that would be a huge waste of computation
finally if we look at these sequences of integers carefully we can probably deduce which sequence belongs
to which original sentence.
The one with all the padding is the easiest because that corresponds to the shortest sentence in our
small dataset.
As an exercise here's what you want to do instead of just deducing which sentence goes with which sequence
make use of the index to token mapping from earlier to programmatically convert these sequences back
into their original sentence form.