-: Hey, you're gonna talk about claim detection
because this is a common issue in creating content
with AI that it makes things up, hallucinates,
and it comes up with quotes that didn't actually happen.
It makes up statistics that are outta date.
It's typically something that is a weak point of AI.
But the interesting thing is that you can actually use AI
to solve AI's own problems. (laughs)
Everything just becomes AI.
What we have here is some article text.
I made this, it's an article based on a real article
here in the New York Times.
I copied and pasted it into ChatGPT,
and I asked it to rewrite the story
but add a few fake things in here.
The fake things that it's added are that it's said
that the court has not, yeah, that the court
has voided his pay package, Elon Musk's pay package.
And in reality they didn't.
We also added in this fake quote from Greg Varallo,
this quote from the lawyer of the disenchanted shareholders.
It's fabricated.
And then we made up a statistic as well.
So saying that we'd, he would own 20% of Tesla,
the actual figure wasn't quoted.
Cool. Those are the fake things.
We'll see if it finds those.
It's the New York Times, so their journalism's pretty good.
You would expect all the other things to be true,
but that's not true of all journalism
either AI generated or human.
So it's useful for us to be able
to detect when claims are being made
and then investigate them after, again, using AI.
You're gonna run that.
Now I have a basic prompt that I put together
here as a diligent copy editor, your role is
detecting claims that need to be fact checked
and I've given some specifics.
A claim could be a statistic, a quote,
an anecdote, an instruction.
So if it tells you this is how
this function works in Excel, right?
That can actually be a problem.
Like it might make that up.
I've seen that happen in the past.
Or say any information that's stated is fact,
which may be out of date incorrect or imagined.
Asking it to return JSON with an unmodified excerpt
from the text, the type of claim being detected,
and some reasoning why the claim might need verification.
So a bit of chain of thought going on here.
I'm using GPT-4o, it makes mistakes,
but it's very fast, very cheap.
So I prefer for this type of task.
And then I'm just passing in the article text here,
and then the temperature I'm putting at 0.2
'cause I don't want it to make up the excerpts.
I've had that as a problem in the past,
but you know, should be good here
if we keep the temperature low,
and then if we run that can see what excerpts we get back.
Okay, so here's an example of an excerpt.
"For months, many Tesla investors are worried about how
engaged, who would be," blah blah blah.
There's an anecdote and then reasoning "the claim that a
judge in Delaware voided Elon Musk's
pay package needs verification."
Yeah, that was one of our fake ones, right?
It says, especially after a judge in Delaware
voided his pay package.
So it has correctly found that.
I also found the statistics,
the shares are worth 48 billion,
so we need to look up the latest value of the shares.
Yeah, you can see that picked up the percentage
of Tesla here, which is pretty good as well.
So I picked up a lot of the claims, you know, both true ones
and also the fake ones that we put in here.
All right, we need to get that into actual JSON.
So in order to do that, we just basically
remove this like beginning part from here.
See it says triple tick JSON, get rid of that,
and get rid of the triple ticks at the end.
And then we have a Python object that we can work with now.
So that's the first claim.
I'm using Tavily to look up these research claims.
Tavily is an AI tool basically to search the web.
It gives web search capabilities to your AI agents.
There are other options here.
Like you could use perplexities API has search
and Google's AI has search as well.
And then you could also use SERP API,
I think is another one that people use for this.
And some of them have free credits, some of them don't.
But Tavily, I've been recommended by a few people
it has a thousand requests for free per month.
It's worth setting it up,
and you don't need a credit card or anything.
Cool, here's Tavily Client, you just import that.
I'm just gonna run this, it's gonna ask me for my API key.
If I go into Tavily here,
I'm just gonna regenerate this,
gonna copy it.
You don't wanna show anyone your API key,
and just paste that in there and hit enter,
and then hopefully that will run.
Here we go. So here's the basic search.
I've just asked it to validate its claim.
I've formatted it slightly so I just put the claim,
and then the reasoning into a string,
and then I put that into here.
It just says validate this claim.
You can see this is the query that I gave it.
And you can see this is the one about
the pay package being voided.
So that's the question I asked.
And then these are the results.
So it's done basically a Google search,
and you can see the content of those searches.
So it has the links.
So if you want to be able to provide citations
and things, this is really great, it's really helpful,
and you've got the content here as well
as the score in terms of how relevant they are, right?
We have the context that's really useful,
but I'm gonna be even lazier here.
Because I don't wanna have to make another call,
and say, "Okay, based on this context,
is this claim correct?"
You know, maybe it depends on
what application you're designing,
but for me I just want to check and get an answer.
So I'm gonna use the Q & A search,
which is basically just another call
in underneath it will do this search,
and then it will give you back the answer.
So it says the claim that a judge in Delaware voided
on Elon Musk's pay package is not valid.
Tesla shareholders recently voted to reaffirm
a $56 billion compensation package.
So this actually just happened a couple days ago.
So it's really impressive that, you know, we're able
to pull out this up-to-date information.
And one of the reasons why AI makes fake claims is
because they don't know what the up-to-date information is.
Really powerful to be able to go
and fact check your sources again with AI, right?
So you know, don't have to have
a human do this fact checking.
When you wanna run this across lots of different ones,
you could do a for loop here,
I'm just looping through every single one,
and then I'll have all my claims checked.
You could also structure it in a different way
so that you would get maybe a yes no answer, for example.
That'd be really helpful because then you could
display an invalid claim in your user interface
instead if you do run into an issue.
There's also error handling as well,
so I've noticed sometimes with Tavily it gets,
I get this like bad request for URL sometimes,
or you might have rate limiting,
so maybe you use multiple solutions
and see which ones work best for you.
But ultimately what you get here is you have the excerpt,
you have the type of claim, so you can see what types
of claims are being made and then you would see the context.
So here we go.
There's no verify information to support the claim
that the judge voided the pay packet.
Here it claims that the shares are worth 48 billion,
because the risk cannot be verified,
does not specify the exact value of the shares.
What you could do now if you were building
an AI application with this is you can have
another cause to LLM for each one of these
and just say, "Okay, based on the context,
do we need to alert the user to check this claim?"
So maybe in user interface you have some visual highlighting
that you say, "Hey, these are the claims you need to look at
and rewrite potentially or make 'em up to date."
You could also use this as a trigger.
So you could say, "Send this back to GPT-4 in order
to rewrite this excerpt if there is some claim
that we have the context to fix."
Yeah, there's lots you can do with this.
This is more of a mental model for you to think about
using for lots of different purposes.
But in particular, I use this for SEO generated content
just to make sure that it is accurate,
and it's not putting more fake information
(laughs) out there on the internet,
because that's one of the biggest
bugbears people have with AI.
And hopefully you can help avoid it with this.