Okay, let's talk about five tricks.
One trap.
Let's get started with trick number one, the illusion of memory.
Something which you've already seen basically before.
But.
But, uh, it's always worth taking a minute to explain this.
So, look, we saw before, when we make a prompt to an LM, like, my name is Ed, and the LM has to
complete this prompt, it comes up with a most likely, uh, text to come after this.
Or as you may know, it's really generating tokens.
There's little fragments of text and it responds, hi, Ed.
And then we give it a separate prompt.
What's my name?
Every time you call an LM, as I explained a million times, it's stateless.
This is GPT, the model, the LM, not ChatGPT the product.
You make a call to GPT, the model, it takes an input sequence.
What's my name?
It predicts the most likely thing to come after that, based on all of the training data examples.
And of course it says I don't know your name or something more charming than that in GPT style.
Okay, so the illusion of memory is this very simple trick, which is that when you prompt an LLM,
you might say, my name is Ed and it might say hi Ed, the next time that you send a prompt to the LLM.
If the user has said, what's my name?
You do not send the LLM the prompt, what's my name?
Do you know what you do?
Send it.
You probably do, but if not, this is the answer.
What you send it is the whole conversation so far.
My name is Ed.
Hi, Ed.
What's my name?
And what it is predicting is what comes next.
And in all of its training data, it was given inputs that have this kind of set of message response
message response.
And so it's expecting something like that.
So when it generates the next tokens, the next few the the text to come at the end of that, it generates
something consistent with the full conversation so far.
And that's why it responds.
Your name is Ed, and that's why every time you talk to ChatGPT through the UI, it's sending the underlying
GPT, LLM the entire conversation so far each time.
And it generates the next text, the tokens based on the full history so far, which is called every
time that you press enter when you're using ChatGPT.
So this trick of sending the full conversation gives us the illusion that this data science model remembers
what we said 30s ago.
It doesn't.
It's just included every time.
In the prompt to the LLM.
Many of you knew that already, but it never hurts to go through it again.
And that that is how trick one works.
And so trick two, which arguably is less to do with agentic AI.
It's just a general trick of thinking reasoning.
This started out like like I guess a year and a half ago or so when people discovered something that
at the time was called chain of thought, which was this idea that you could ask an LLM, a question,
and then you could just add to the end of your prompt.
Uh, please think step by step and you would just get better outcomes.
Just as a result of prompting it that way.
And there seemed to be some, some kind of strange feature that if an LLM is generating text that describes
what it should do and then it generates text to do it, you end up getting better outcomes.
Then nothing, nothing is actually thinking through in that way.
It's just generating likely text.
Uh, but uh, but just as a side effect, by virtue of generating text to describe a thought process,
it ends up coming up with better outcomes.
And so that resulted in this idea of a thinking or a reasoning model.
It means the same thing.
And that is a model which is actually been trained so that when it's given any question before it generates
the text for the final output, it generates some text to describe how it will take step by step your
question to come up with an output.
And that is why you sometimes see with some of these models a sort of what they call a thinking trace
text coming up that describes its thought process before you get the final output.
And so to give you a concrete example, and I'm actually taking this from an example that I do in some
of my technical courses.
You can you can prompt an LLM that is not thinking and reasoning, but a very small LLM like, like
for example, GPT 4.1 nano on a mode where it's not reasoning and you can say, uh, toss two coins,
one of them is heads, what's the chances?
The other one is tails, and it's quite likely to get the wrong answer.
In this case it's responding 50 over 50.5 as the as the probability which which happens to be the wrong
answer.
Because this is one of those sneaky little questions that, uh, has has trickery built into it.
If, however, you prompt it with toss two coins.
One is heads.
What's the chance?
The other one is tails.
First, describe your thought process.
Then give the answer.
Or if you use a reasoning model which has been trained to first output the thought process and then
the answer, you will get some kind of thought process that will say, okay, this this is probably
a trick question or this is a sneaky question.
Now I need to realize that they haven't said I toss two coins.
The one on the left is heads.
What's the chances?
The one on the right is tails, which would of course be 5050.
But the question has been raised differently to that.
Okay, there are four different possibilities here blah blah, blah blah blah.
The answer is two thirds and it tends to get that right.
And you could try these experiments and depending on which models you pick and how you do it, you can
you can quite easily recreate this idea that a small model, when it's not given the chance to think
it through or get the wrong answer.
And when reasoning is on and it's giving tokens, it's outputting text to describe the thought process,
you end up with the right answer.
So this reasoning trick ends up working really well to get more, more sophisticated intelligence out
of using using a model which is just generating the most likely text to follow some input text.
And just to add one more piece here, you may you may know this already, but maybe you don't.
The way that these models actually work, they don't generate the output given an input.
The way I've been describing it.
Exactly.
They're given an input, and the first thing they do is they generate the first few letters.
It's called a token, like a chunk of letters that would come after this input, just the first one,
the most likely next thing to come that is then taken and it's stuffed on the end of the input, and
then that whole thing is passed in again to the model, and it's now challenged to come up with the
next little fragment of letters, the next token that comes after all this input and the first token.
And it does that, that gets shoved on the end and it repeats.
That's the way that models generate text.
That process is known as inference.
And when it's doing that, it's generating tokens that are most likely to come next after all of the
input and the output so far.
And that's why you get this curious side effect that if you ask it to generate tokens to describe a
thought process, it will start by doing that.
And then you can generate tokens for the for the outcome.
And it will be consistent with the thought process and it just results in better outcomes.
It sounds counterintuitive.
It sounds like like that's too good to be true, but it actually works.
It's empirical and it works.
And there are some models, uh, which which are designed for being just in a chat mode.
They have been trained to just respond immediately.
And there are other variants, as we say, other other ways of the same kinds of the same family of
models has been trained in order to be in a reasoning mode where it will generate reasoning stuff first
and then the answer.
And there's also some hybrid models that can can do both.
And it's people.
People tend to always sort of gravitate towards reasoning models, because they tend to be much more
intelligent and do better in any of the benchmarks and tests.
But it turns out often chat models are better for different use cases, particularly in agentic AI when
you're already driving a whole system step by step.
So don't always assume that a reasoning model is going to be better.
Sometimes a chatbot will do better, and you may have the question of okay, how do you tell?
And the answer, as is so often the case when working with Llms, is that you should try both and pick
the one that performs better.
This whole field is super experimental.
There are no right and wrong answers except to experiment.
Have some sort of a measurement of success and use that to make your decision.
And then one final thing I'll say on this, which is that some, some models allow you to specify how
long it should think for.
Sometimes called the reasoning effort, sometimes called the thinking budget.
Then in some models, like the latest GPT 5.1, you can choose a budget of none if you want it to be
in chat mode, or minimal or low or medium or high.
Those are all the settings of the reasoning or thinking budget, and you may wonder how that actually
works, like how the model is just just generating these tokens and how do you you sort of what machinery
have you built around one of these models to tell it, hey, I want you to do more thinking.
And there are a few different techniques that are used, but the one that that is most common is unbelievably
hacky.
It's it's so janky that it's hard to believe this is really how it works.
But, but but let me just, just quickly tell you now, I mentioned that when you're running a model
in inference, as it's called, the model is generating tokens.
You do them token at a time, generates a token that gets shoved back in.
You generate the next token, shove back in.
So during the reasoning mode it's doing this and it's coming up with all of this reasoning stuff as
it thinks things through.
And someone had a bright idea.
And this sounds crazy.
Someone had a bright idea that that you can.
Of course you can choose to add in extra tokens in here.
You don't have to take what it's generated and put that back in.
You could put in something else as well if you wanted.
And that becomes the new input and it generates an output.
It doesn't know that it didn't generate all of this, that there's something extra in there that you've
shoved in there.
It just thinks this is the whole input that it's now generating the next token.
Fine.
So what might you shove in there that might that might help this?
Well, someone had this idea that when it's generating tokens, it generates reasoning a couple of sentences
about how it's thinking things through.
You could wait until it has like a full stop when it's like finished what appears to be a some sort
of a thought process and then just insert in the word wait at the end of that.
So it says something like, okay, this appears to be a trick question.
I should consider the different possibilities.
And then the word wait appears there.
And now it has to generate what comes next.
And the tokens that it now generates need to be coherent with the word weight.
And so it tends to generate something that is kind of taking a step back.
And it's like, wait, I should double check that I've really understood the problem.
Or wait, I hope I'm not jumping to conclusions.
Let me review.
And so it tends to generate tokens that are reflective of the thought process so far.
And it goes back and revisits and challenges itself.
So this trick of adding in things like weight and on the other hand, and stuff like that tends to allow
it to rethink through and often explore new territory.
And if you look in some of the reasoning traces you see when you're chatting with Llms, you'll see
the word weight in there and others, and you'll know that it's this trick at play.
This seemingly really, really simplistic trick does really well, and that is one of the core techniques
that's used to sort of force this thinking budget and make a model think through more the train of thought
before it comes up with the final answer.