Chain of thought is one of the most common prompting technique.
It's really useful for getting learners to read, and they're still not great at it.
But this makes a really big difference in terms of performance, and it's very easy to implement.
But we're going to walk through this.
There is a paper that you can read on Chain of Thought.
And actually it's used in many scientific papers.
But the one I recommend reading is this one chain of thought prompting with its reasoning in large language
model.
I think this is the one that introduced it, but the general idea of what you have to understand about
chain of thought is that it mimics the way that humans think.
If you think about system one and system two, it's how humans divide up the different tasks they have
to do.
And the vast majority of the operation of your brain is system one.
It's this automatic emotional process where you give an immediate response subconsciously, and that's
how we make most of the decisions that we need to make.
And then when you actually stop and think about something, you're using system two.
So that's a lot more deliberate.
You're actually consciously working through the steps.
And that's really what chain of thought is.
You know, if large language models are really all system one right now are using chain of thought is
how you make it stop and think through the steps and be deliberate against perched.
And this is something that pretty much every mobile vendor in their prompt engineering guide that talks
about life, they say you should give the AI time to think.
That's what OpenAI says.
And you're actually anthropic uses this technique in its tool.
They output like a thinking tag first, and you can actually see that if you ask it to see what the
results are.
So let's dive into the actual example.
First of all, we just have a basic function here which called the OpenAI API passes in a system prompt
and a normal prompt.
So the user prompt.
So the system prompt here solve the following problem and return the answer in the format a answer.
The question is how many hours are there in raspberry.
Now this is a famous failure mode for llms because they can't really count letters that these are all
tokenized, so it does a bad job with them.
And then we also have here a standard prompt which is a three shot prompt.
So we say here's three examples.
We say how many years are there in elephant.
And they convert to how many keys are with three.
And then how many O's are there.
And chocolate becomes two.
So you get a bunch of examples of the types of answers we want.
And then we're given it the question which is how many are there around Around three.
And when we run that, if we just run that now we see hopefully that it came back with two.
But there are actually three R's in raspberry because they have the R beginning and then the two at
the end.
Now the chain of thought prompts got the right result and it did do that.
But this answer three we gave it this thinking in the examples we told it the think step by step.
And then we spell it out.
and then we have Jeff in between the letters.
And it actually, you know, figured out.
So we we actually wrote these right.
And showed it how it should think through the problem.
And then we say, let's look step by step.
And it's doing a much better job.
So there are different ways to implement this of course.
And you know some ways work better than others.
But I think the main thing to think about here is how do you demonstrate to it that it should reason
through the steps within a previous few short examples?
All right.
So that's chain of thought.
And when you're implementing a technique like that you typically want to evaluate the response.
And in this case it might be pretty straightforward because you know the correct answer is.
So I created a evaluate response function.
And it just searches for that final answer.
So we have it's looking for a and then the space and then the number and kind of pull back that number
as the final answer.
It can refer to an integer.
And that's why it was so important by the way for us to provide examples in the first place, because
you'll find that it might say the final answers to and doesn't start with this a colon space.
And that's why few shot prompting to be really helpful because specifying a format to follow.
And then you could check what the right answer is afterwards.
So this just pulls out the right answer and then splits it.
And then the other thing gets checking as well as if there are any steps, any reading steps.
So we run that on this example test response.
And we can see that your answer is correct.
And it does provide the steps that we fasten the correct answers to.
And it is multiple lines here.
If we deleted this then it would say doesn't provide stats.
Or if we said the correct answer three they would say that was all.
Now we know that our evaluation metric is working.
We can run that for lots of different test questions.
Then here we have a bunch of new ones.
How many are in hamburger mdm2, how many ls other umbrella and so on and so forth.
So we run the evaluation.
What we're going to do is give it the prompt type and say if the front type is standard, then we're
going to just give it the normal few shot prompt without any recent steps.
And if the prompt is the kind of thought prompt, and we're going to give it to the other prompt, to
the one with the steps and which have been timely response, we're going to return it and we're going
to check the evaluation.
Is it correct.
And that steps.
And then we're going to figure out the overall accuracy of that prompt the average time.
And then the step percentage how many of these has steps.
So this is all in one function in controlling for the different type.
And then returning the evaluation metrics.
So we can run that on the standard.
And then we can run that on the chain of thought.
And we're going to be able to compare the two results.
So let's see what we get.
Standard prompting.
We get 90% accuracy and it takes less than a second to run.
But none of them have steps and they'll have reasoning.
So you wouldn't be able to debug with it where it went wrong.
Whereas chain of thought prompting we get 100% accuracy.
It does take longer.
So take 1.62 seconds.
And this is something to think about in production.
If you're asking it for more steps than the may cost you more tokens, and it's also going to take longer
to return.
And this is the type of trade off you need to make with Promptitude and chained.
Thought isn't for every system, because sometimes you need to get a result back quickly.
And as speed can be traded off for an accuracy, when you're deciding whether to deploy to your thought
and how you deploy chain, your thought makes a big difference to that trade off to you.