-: All right, now we're gonna learn about fine tuning.
And as of recording this video,
it literally just came out yesterday.
So you're getting this fresh.
The API
for
OpenAI's 3.5 model is pretty straightforward actually.
And they have some details on the documentation
and also the blog post where they came out
with fine tuning as well.
I've just followed along with the example here,
but I'm gonna explain to you what's happening at each stage.
The first thing you need to do is get your
data into the right format.
If you are following along here,
you can make a copy of this.
My data from my blog is here and you can open that up
and I'll show you what that looks like.
This is the format, so it's called JSON L,
which is like JSON, except every object is on its own line.
So you can see there's no parent object
and it just is a list of messages.
So the messages are the system message,
I've just kept that default.
But if you are testing fine tuning for yourself,
you already have a prompt that's working in the system
message, you might want to use that prompt here.
I've just gone with the default.
Then I think it takes a lot of experimentation,
but you might get better results if you update the system
message equally, I've just used this really simple prompt,
which is I write the section
and then the section header for the blog posts
and then the blog title.
What I've done is I split, I, you know,
downloaded all 48 of my blog posts.
I split them into sections based on the section headers,
the H one tags, and then I've put the name of that section
into, into here.
And then the, the blog
title here, like the AI, is then responding.
So obviously the AI didn't really respond, this is
what I'm telling you, it should have responded.
So that's how you're training it, saying like,
when the user prompts like this, then gimme this type
of content and my hope is gonna train it to talk like me.
So that's the actual section text there.
So getting into this format is, the first
challenge is actually like the,
you know, probably the hardest thing.
What, what I did actually is I cheated a little bit
and I used the code interpreter.
I'll just show you kind of how I did that.
I uploaded the blog posts
and then I just asked it to prep it into this format
and I gave it one example.
I just said, prep into the format, write this section header
for the blog post article title
and make sure you cover all the blog posts in the CSV.
And the format is like this.
I gave it one example, I just took a section header,
Why must agencies charge so much for one of my posts?
And then I pasted in that text manually. Cool.
It actually did all the work.
And I can include, I'll include the code that it used.
So I asked it to, I actually,
so one thing is initially it had a bunch of HTML in there,
so I asked it to get rid of that,
but other than that it was, it worked really well
and I asked it to package up all the code.
So I'll include this as a .py file.
So hopefully you should be able to just run this
or you could give this to code interpreter
and say, can you do that for, but for, for my data.
Cool. And then once you ask it to generate the output,
you can click and save the the file.
Cool. So that's how you get the data in the right format.
This is how you load my data.
So if you just wanna test, you can load this from this link.
The, it's just extracting the file ID
and then pulling the data and then saving it locally.
And then this loads the data.
So you can see you have 304 observations.
Typically with fine tuning,
what OpenAI says is you probably need at least 50
observations in order to start making,
noticing a difference in the model.
And I've seen in a few papers that it's typically
around like 200.
200 observations that you need in
order to get to the point where
fine tuning actually shows better results than
just general prompt engineering.
So I, I would say if you have 200 samples,
then you're probably in a good
position to start fine tuning.
If you're not in that position, then try
and generate 200 samples manually by getting people
to label them or you labeling them.
Or you could just use prompt engineering for now.
And then once you get to 200 samples,
then you can do this type of work.
The, the code here, basically I just need to get your,
get the OpenAI key, you can paste that in there
and you can get that in
here, view API keys, and you can create a new one.
And then this is the call
to the OpenAI fine tune.
I guess this specifically is like the files section.
So before it can fine tune on your data,
it needs to have your data, right?
So this is just a way of uploading that file to OpenAI
and this basically just splits it
and opens it in a file path and then,
and then you get a response that says, here's the file ID,
you know how, here's how big it is and it was uploaded.
Now one thing I found is that sometimes
with larger files it takes a while to upload.
So if you try and run the rest of the code, it doesn't work.
So I've added this section here, which is just checking on
that file and it just, if you run that then, then it just
finds the latest file that you uploaded
and then once it says processed, then you're pretty happy.
Then you can keep running.
Then, and then you start a fine tuning job.
So again, pretty straightforward,
just call the fine tuning jobs API,
again with your OpenAI key.
And it is recommended that you train on this
GPT-3.5 turbo 0 6 1 3.
The reason is that it doesn't,
they don't support GPT-4 fine tuning yet,
although they said it's coming in the fall.
You can also tune on to other models,
but they're the older models like GPT-2.
So it is not really worth messing with,
or at least, you know, not for most of the tasks
that I've found helpful for.
Cool. So when you call that and then you get a job id,
and you can see that like it's not finished yet
and it says status created.
And again, you can just check the status.
So if you run that a few times,
like it will say like running, if you scroll across
and then eventually it'll say status succeeded.
And once it has succeeded,
then you can see how long it took.
In this case it took 24 minutes.
And then also you can calculate the price as,
so you can calculate this ahead of time,
but the, the cost for fine tuning is as is a 10th
of 8 cents per 1000 tokens.
So it costs me $2.22 to train on these 300 observations.
And then you get the actual fine tuned model itself.
So you can see that here, this is my specific code,
it shows up in my account and then you can just query it
just like you otherwise would.
And so this is just a normal query to, to OpenAI,
except I'm passing this new model name in.
And if it's, I think it's really important
that you keep the system message and the con-
and the prompt style the same,
but I'm getting, I think, pretty good results.
So this actually really does sound like me.
I think if I scroll back to the beginning.
It uses a lot of the same kind of informal language,
but quite direct, not much fluff.
Yeah, like words like under the
hood, I use that quite a lot.
Yeah, here we go. I think I'm using
figures and stuff like that.
Yeah, I think it's,
I'm actually pretty impressed with the results.
And then the other thing I think you should think about is
like, this was asking it to a section
that like it'd already trained on,
but really you want to try
and write a section that it hasn't trained on.
So here's a topic that I have written about memetics.
I actually wrote a whole book about memetics,
but not, there's only like one blog post on memetics on my
personal blog and it doesn't mention criticisms of memetics.
So this is like a tangential topic
and I haven't written specifically this section like,
why isn't memetics a science?
So this is a better test
and what I find is it still sounds like me, it starts
to quote different people I've never read,
but it's, it's like doing,
it's still doing a pretty good job.
So I think that more testing needs to be done.
Ideally what you would do is you would hold back some test
data when you have the 300 or something samples.
If you hold back like 10, 15% of that,
and then you could feed in those prompts, generate
and see how well they did
relative to the real task.
So I, I would say that's good, good, you know,
that's a good idea really with any machine learning.
But, but yeah, that's how you're gonna really tell
whether it's done a good job.
You can also tell programmatically,
so there's like the evals functionality and lang chain
or in OpenAI you can do manual kind of blind,
thumbs up, thumbs down, does this sound like me?
Or not like sentence by sentence.
But you could also do embedding distance
or you could calculate the vector for, for the text,
the reference text, the actual writing that I did,
and then calculate the, the embedding for the, the text,
the, the, the fine tune model generated
and then see what the difference is, see how close it gets.
And you can try different fine tuning techniques.
So you try with a different system message
and see if that improves things and so on.
So it is an iterative experimental process.
The other thing to show you is that these models,
there's not much interface unfortunately with these things,
but these models do show up.
So if I refresh
and then you can see I have my fine tunes that show up here.
If I wanna say the section,
what is my favorite color?
Just a random one, just submit.
See what it comes back with.
Yeah, so it doesn't always work.
I think if it goes off, off piece in terms of the examples,
it doesn't do as good a job.
Cool. The, the other thing to talk about is pricing.
Here we go for fine tuning costs, like 8 cents per thousand
or less, sorry, one 10th of 8 cents
for a thousand tokens to train.
And then it costs like basically a 1 cent for input
and you know, one and a half cent for output
when you're talking to the model.
So if you compare, this is what GPT
3.5 normally costs.
So you can see that we're going from
0.0015
to 0.012.
So it's an order of magnitude different in terms of cost,
but it's still pretty cheap, right?
If you think about a GPT-4, that's 3 cents
and 6 cents per token.
And so you can, you're basically getting a much cheaper,
you know, much cheaper model than GPT-4.
And what a lot of people have been telling me is
that you can get better results
with a fine tuned GPT-3.5 model
for specific tasks than you can with a general GPT-4.
So, you know, depends how you position it.
It's either a really expensive GPT-3.5
or like a really cheap GPT-4 replacement for some tasks.
So yeah, hopefully this is useful for you guys.
Again, this is like really new, so this might change,
but hopefully you understand the, the kind of thinking
behind it and how it works.
And what's really great about this is
it's pretty accessible.
You don't have to be a machine learning engineer to do this.
Like, I'm certainly not.
So hopefully you guys can experiment and,
and find some interesting stuff.